| Message ID | 20181010213102.8373-1-f4bug@amsat.org |
|---|---|
| State | New |
| Headers | show |
| Series | [v3] python: Use io.StringIO | expand |
On 10/10/2018 23:31, Philippe Mathieu-Daudé wrote: > Both Python 2.7 and 3 support the same io.StringIO to > handle unicode strings. > > Use the common form to use indistinctly Python 2.7 or 3. > > http://python-future.org/compatible_idioms.html#stringio > > This fixes running tests on the Fedora Docker image, > which uses Python3 since 356dc290f: > > $ make docker-test-block@fedora > [...] > 045 [failed, exit status 1] - output mismatch (see 045.out.bad) > --- /tmp/qemu-test/src/tests/qemu-iotests/045.out 2018-07-17 16:56:18.000000000 +0000 > +++ /tmp/qemu-test/build/tests/qemu-iotests/045.out.bad 2018-07-17 17:19:22.448409007 +0000 > @@ -1,5 +1,6 @@ > -........... > ----------------------------------------------------------------------- > -Ran 11 tests > - > -OK > +Traceback (most recent call last): > + File "045", line 178, in <module> > + iotests.main(supported_fmts=['raw']) > + File "/tmp/qemu-test/src/tests/qemu-iotests/iotests.py", line 682, in main > + import StringIO > +ModuleNotFoundError: No module named 'StringIO' > 132 [failed, exit status 1] - output mismatch (see 132.out.bad) > 152 [failed, exit status 1] - output mismatch (see 152.out.bad) > > Failures: 045 132 152 > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > --- > tests/qemu-iotests/iotests.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py > index 4e67fbbe96..cd9be1bd9c 100644 > --- a/tests/qemu-iotests/iotests.py > +++ b/tests/qemu-iotests/iotests.py > @@ -679,13 +679,13 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], > > # We need to filter out the time taken from the output so that qemu-iotest > # can reliably diff the results against master output. > - import StringIO > + from io import StringIO > if debug: > output = sys.stdout > verbosity = 2 > sys.argv.remove('-d') > else: > - output = StringIO.StringIO() > + output = StringIO() > > logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN)) > > Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Am 10.10.2018 um 23:31 hat Philippe Mathieu-Daudé geschrieben: > Both Python 2.7 and 3 support the same io.StringIO to > handle unicode strings. > > Use the common form to use indistinctly Python 2.7 or 3. > > http://python-future.org/compatible_idioms.html#stringio > > This fixes running tests on the Fedora Docker image, > which uses Python3 since 356dc290f: > > $ make docker-test-block@fedora > [...] > 045 [failed, exit status 1] - output mismatch (see 045.out.bad) > --- /tmp/qemu-test/src/tests/qemu-iotests/045.out 2018-07-17 16:56:18.000000000 +0000 > +++ /tmp/qemu-test/build/tests/qemu-iotests/045.out.bad 2018-07-17 17:19:22.448409007 +0000 > @@ -1,5 +1,6 @@ > -........... > ----------------------------------------------------------------------- > -Ran 11 tests > - > -OK > +Traceback (most recent call last): > + File "045", line 178, in <module> > + iotests.main(supported_fmts=['raw']) > + File "/tmp/qemu-test/src/tests/qemu-iotests/iotests.py", line 682, in main > + import StringIO > +ModuleNotFoundError: No module named 'StringIO' > 132 [failed, exit status 1] - output mismatch (see 132.out.bad) > 152 [failed, exit status 1] - output mismatch (see 152.out.bad) > > Failures: 045 132 152 > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> This doesn't work for me: 045 1s ... [13:31:47] [13:31:47] [failed, exit status 1] - output mismatch (see 045.out.bad) --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24 14:49:48.000000000 +0100 +++ /home/kwolf/source/qemu/tests/qemu-iotests/045.out.bad 2018-10-11 13:31:47.266876850 +0200 @@ -1,5 +1,26 @@ -........... ----------------------------------------------------------------------- -Ran 11 tests - -OK +Traceback (most recent call last): + File "045", line 178, in <module> + iotests.main(supported_fmts=['raw']) + File "/home/kwolf/source/qemu/tests/qemu-iotests/iotests.py", line 698, in main + unittest.main(testRunner=MyTestRunner) + File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ + self.runTests() + File "/usr/lib64/python2.7/unittest/main.py", line 232, in runTests + self.result = testRunner.run(self.test) + File "/usr/lib64/python2.7/unittest/runner.py", line 151, in run + test(result) + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ + return self.run(*args, **kwds) + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run + test(result) + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ + return self.run(*args, **kwds) + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run + test(result) + File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ + return self.run(*args, **kwds) + File "/usr/lib64/python2.7/unittest/case.py", line 406, in run + result.addSuccess(self) + File "/usr/lib64/python2.7/unittest/runner.py", line 62, in addSuccess + self.stream.write('.') +TypeError: unicode argument expected, got 'str' Failures: 045 Failed 1 of 1 tests $ /usr/bin/env python --version Python 2.7.15 Kevin
On 11/10/2018 13:33, Kevin Wolf wrote: > 045 1s ... [13:31:47] [13:31:47] [failed, exit status 1] - output mismatch (see 045.out.bad) > --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24 14:49:48.000000000 +0100 > +++ /home/kwolf/source/qemu/tests/qemu-iotests/045.out.bad 2018-10-11 13:31:47.266876850 +0200 > @@ -1,5 +1,26 @@ > -........... > ----------------------------------------------------------------------- > -Ran 11 tests > - > -OK > +Traceback (most recent call last): > + File "045", line 178, in <module> > + iotests.main(supported_fmts=['raw']) > + File "/home/kwolf/source/qemu/tests/qemu-iotests/iotests.py", line 698, in main > + unittest.main(testRunner=MyTestRunner) > + File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ > + self.runTests() > + File "/usr/lib64/python2.7/unittest/main.py", line 232, in runTests > + self.result = testRunner.run(self.test) > + File "/usr/lib64/python2.7/unittest/runner.py", line 151, in run > + test(result) > + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ > + return self.run(*args, **kwds) > + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run > + test(result) > + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ > + return self.run(*args, **kwds) > + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run > + test(result) > + File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ > + return self.run(*args, **kwds) > + File "/usr/lib64/python2.7/unittest/case.py", line 406, in run > + result.addSuccess(self) > + File "/usr/lib64/python2.7/unittest/runner.py", line 62, in addSuccess > + self.stream.write('.') > +TypeError: unicode argument expected, got 'str' > Failures: 045 > Failed 1 of 1 tests > > $ /usr/bin/env python --version > Python 2.7.15 Indeed, io.StringIO exists in Python 2.7 but it's different! If Python 2 code is not unicode-friendly (it almost never is) it should use StringIO.StringIO, for example 'six' defines six.StringIO: This is an fake file object for textual data. It’s an alias for StringIO.StringIO in Python 2 and io.StringIO in Python 3. six.BytesIO: This is a fake file object for binary data. In Python 2, it’s an alias for StringIO.StringIO, but in Python 3, it’s an alias for io.BytesIO. So the solution seems to be a try/except (and then, better move the "from X import StringIO" to the top of the file then, rather than keeping it in def main). Paolo
Paolo Bonzini <pbonzini@redhat.com> writes: > On 11/10/2018 13:33, Kevin Wolf wrote: >> 045 1s ... [13:31:47] [13:31:47] [failed, exit status 1] - output mismatch (see 045.out.bad) >> --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24 14:49:48.000000000 +0100 >> +++ /home/kwolf/source/qemu/tests/qemu-iotests/045.out.bad 2018-10-11 13:31:47.266876850 +0200 >> @@ -1,5 +1,26 @@ >> -........... >> ----------------------------------------------------------------------- >> -Ran 11 tests >> - >> -OK >> +Traceback (most recent call last): >> + File "045", line 178, in <module> >> + iotests.main(supported_fmts=['raw']) >> + File "/home/kwolf/source/qemu/tests/qemu-iotests/iotests.py", line 698, in main >> + unittest.main(testRunner=MyTestRunner) >> + File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ >> + self.runTests() >> + File "/usr/lib64/python2.7/unittest/main.py", line 232, in runTests >> + self.result = testRunner.run(self.test) >> + File "/usr/lib64/python2.7/unittest/runner.py", line 151, in run >> + test(result) >> + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ >> + return self.run(*args, **kwds) >> + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run >> + test(result) >> + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ >> + return self.run(*args, **kwds) >> + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run >> + test(result) >> + File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ >> + return self.run(*args, **kwds) >> + File "/usr/lib64/python2.7/unittest/case.py", line 406, in run >> + result.addSuccess(self) >> + File "/usr/lib64/python2.7/unittest/runner.py", line 62, in addSuccess >> + self.stream.write('.') >> +TypeError: unicode argument expected, got 'str' >> Failures: 045 >> Failed 1 of 1 tests >> >> $ /usr/bin/env python --version >> Python 2.7.15 > > Indeed, io.StringIO exists in Python 2.7 but it's different! If Python > 2 code is not unicode-friendly (it almost never is) it should use > StringIO.StringIO, for example 'six' defines > > six.StringIO: > This is an fake file object for textual data. It’s an alias for > StringIO.StringIO in Python 2 and io.StringIO in Python 3. > > six.BytesIO: > This is a fake file object for binary data. In Python 2, it’s an > alias for StringIO.StringIO, but in Python 3, it’s an alias for > io.BytesIO. > > So the solution seems to be a try/except (and then, better move the > "from X import StringIO" to the top of the file then, rather than > keeping it in def main). > > Paolo I think we need to invest more resources in Python 2/3 compatiblity, or else we'll miss our hard deadline of January 1, 2020. https://pythonclock.org/
On 11/10/2018 16:01, Markus Armbruster wrote: > Paolo Bonzini <pbonzini@redhat.com> writes: > >> On 11/10/2018 13:33, Kevin Wolf wrote: >>> 045 1s ... [13:31:47] [13:31:47] [failed, exit status 1] - output mismatch (see 045.out.bad) >>> --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24 14:49:48.000000000 +0100 >>> +++ /home/kwolf/source/qemu/tests/qemu-iotests/045.out.bad 2018-10-11 13:31:47.266876850 +0200 >>> @@ -1,5 +1,26 @@ >>> -........... >>> ----------------------------------------------------------------------- >>> -Ran 11 tests >>> - >>> -OK >>> +Traceback (most recent call last): >>> + File "045", line 178, in <module> >>> + iotests.main(supported_fmts=['raw']) >>> + File "/home/kwolf/source/qemu/tests/qemu-iotests/iotests.py", line 698, in main >>> + unittest.main(testRunner=MyTestRunner) >>> + File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ >>> + self.runTests() >>> + File "/usr/lib64/python2.7/unittest/main.py", line 232, in runTests >>> + self.result = testRunner.run(self.test) >>> + File "/usr/lib64/python2.7/unittest/runner.py", line 151, in run >>> + test(result) >>> + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ >>> + return self.run(*args, **kwds) >>> + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run >>> + test(result) >>> + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ >>> + return self.run(*args, **kwds) >>> + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run >>> + test(result) >>> + File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ >>> + return self.run(*args, **kwds) >>> + File "/usr/lib64/python2.7/unittest/case.py", line 406, in run >>> + result.addSuccess(self) >>> + File "/usr/lib64/python2.7/unittest/runner.py", line 62, in addSuccess >>> + self.stream.write('.') >>> +TypeError: unicode argument expected, got 'str' >>> Failures: 045 >>> Failed 1 of 1 tests >>> >>> $ /usr/bin/env python --version >>> Python 2.7.15 >> >> Indeed, io.StringIO exists in Python 2.7 but it's different! If Python >> 2 code is not unicode-friendly (it almost never is) it should use >> StringIO.StringIO, for example 'six' defines >> >> six.StringIO: >> This is an fake file object for textual data. It’s an alias for >> StringIO.StringIO in Python 2 and io.StringIO in Python 3. >> >> six.BytesIO: >> This is a fake file object for binary data. In Python 2, it’s an >> alias for StringIO.StringIO, but in Python 3, it’s an alias for >> io.BytesIO. o_O Thanks for the clarification. I'll let people who actually do understand Python fix this. >> >> So the solution seems to be a try/except (and then, better move the >> "from X import StringIO" to the top of the file then, rather than >> keeping it in def main). >> >> Paolo > > I think we need to invest more resources in Python 2/3 compatiblity, or > else we'll miss our hard deadline of January 1, 2020. > > https://pythonclock.org/ Some distributions already dropped Python2, so you need to specifically install it to run block tests (other tests work with Python3). See this thread: https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg06461.html https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg06510.html Regards, Phil.
Am 11.10.2018 um 16:01 hat Markus Armbruster geschrieben: > Paolo Bonzini <pbonzini@redhat.com> writes: > > > On 11/10/2018 13:33, Kevin Wolf wrote: > >> 045 1s ... [13:31:47] [13:31:47] [failed, exit status 1] - output mismatch (see 045.out.bad) > >> --- /home/kwolf/source/qemu/tests/qemu-iotests/045.out 2017-01-24 14:49:48.000000000 +0100 > >> +++ /home/kwolf/source/qemu/tests/qemu-iotests/045.out.bad 2018-10-11 13:31:47.266876850 +0200 > >> @@ -1,5 +1,26 @@ > >> -........... > >> ----------------------------------------------------------------------- > >> -Ran 11 tests > >> - > >> -OK > >> +Traceback (most recent call last): > >> + File "045", line 178, in <module> > >> + iotests.main(supported_fmts=['raw']) > >> + File "/home/kwolf/source/qemu/tests/qemu-iotests/iotests.py", line 698, in main > >> + unittest.main(testRunner=MyTestRunner) > >> + File "/usr/lib64/python2.7/unittest/main.py", line 95, in __init__ > >> + self.runTests() > >> + File "/usr/lib64/python2.7/unittest/main.py", line 232, in runTests > >> + self.result = testRunner.run(self.test) > >> + File "/usr/lib64/python2.7/unittest/runner.py", line 151, in run > >> + test(result) > >> + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ > >> + return self.run(*args, **kwds) > >> + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run > >> + test(result) > >> + File "/usr/lib64/python2.7/unittest/suite.py", line 70, in __call__ > >> + return self.run(*args, **kwds) > >> + File "/usr/lib64/python2.7/unittest/suite.py", line 108, in run > >> + test(result) > >> + File "/usr/lib64/python2.7/unittest/case.py", line 431, in __call__ > >> + return self.run(*args, **kwds) > >> + File "/usr/lib64/python2.7/unittest/case.py", line 406, in run > >> + result.addSuccess(self) > >> + File "/usr/lib64/python2.7/unittest/runner.py", line 62, in addSuccess > >> + self.stream.write('.') > >> +TypeError: unicode argument expected, got 'str' > >> Failures: 045 > >> Failed 1 of 1 tests > >> > >> $ /usr/bin/env python --version > >> Python 2.7.15 > > > > Indeed, io.StringIO exists in Python 2.7 but it's different! If Python > > 2 code is not unicode-friendly (it almost never is) it should use > > StringIO.StringIO, for example 'six' defines > > > > six.StringIO: > > This is an fake file object for textual data. It’s an alias for > > StringIO.StringIO in Python 2 and io.StringIO in Python 3. > > > > six.BytesIO: > > This is a fake file object for binary data. In Python 2, it’s an > > alias for StringIO.StringIO, but in Python 3, it’s an alias for > > io.BytesIO. > > > > So the solution seems to be a try/except (and then, better move the > > "from X import StringIO" to the top of the file then, rather than > > keeping it in def main). > > > > Paolo > > I think we need to invest more resources in Python 2/3 compatiblity, or > else we'll miss our hard deadline of January 1, 2020. Did we decide until when we want to support Python-2-only hosts in QEMU? I think I would be okay with just getting rid of Python 2 support and always using Python 3 instead of trying to write code that works with both. Python 3 was installed for me, but the scripts didn't use it. Kevin
On Thu, Oct 11, 2018 at 04:56:04PM +0200, Kevin Wolf wrote: > Am 11.10.2018 um 16:01 hat Markus Armbruster geschrieben: [...] > > I think we need to invest more resources in Python 2/3 compatiblity, or > > else we'll miss our hard deadline of January 1, 2020. > > Did we decide until when we want to support Python-2-only hosts in QEMU? > > I think I would be okay with just getting rid of Python 2 support and > always using Python 3 instead of trying to write code that works with > both. Python 3 was installed for me, but the scripts didn't use it. I'd like to drop Python 2 support on QEMU 3.1.
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 4e67fbbe96..cd9be1bd9c 100644 --- a/tests/qemu-iotests/iotests.py +++ b/tests/qemu-iotests/iotests.py @@ -679,13 +679,13 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[], # We need to filter out the time taken from the output so that qemu-iotest # can reliably diff the results against master output. - import StringIO + from io import StringIO if debug: output = sys.stdout verbosity = 2 sys.argv.remove('-d') else: - output = StringIO.StringIO() + output = StringIO() logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
Both Python 2.7 and 3 support the same io.StringIO to handle unicode strings. Use the common form to use indistinctly Python 2.7 or 3. http://python-future.org/compatible_idioms.html#stringio This fixes running tests on the Fedora Docker image, which uses Python3 since 356dc290f: $ make docker-test-block@fedora [...] 045 [failed, exit status 1] - output mismatch (see 045.out.bad) --- /tmp/qemu-test/src/tests/qemu-iotests/045.out 2018-07-17 16:56:18.000000000 +0000 +++ /tmp/qemu-test/build/tests/qemu-iotests/045.out.bad 2018-07-17 17:19:22.448409007 +0000 @@ -1,5 +1,6 @@ -........... ----------------------------------------------------------------------- -Ran 11 tests - -OK +Traceback (most recent call last): + File "045", line 178, in <module> + iotests.main(supported_fmts=['raw']) + File "/tmp/qemu-test/src/tests/qemu-iotests/iotests.py", line 682, in main + import StringIO +ModuleNotFoundError: No module named 'StringIO' 132 [failed, exit status 1] - output mismatch (see 132.out.bad) 152 [failed, exit status 1] - output mismatch (see 152.out.bad) Failures: 045 132 152 Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> --- tests/qemu-iotests/iotests.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)