Message ID | 20190503004130.8285-20-ehabkost@redhat.com |
---|---|
State | New |
Headers | show |
Series | [PULL,01/19] tests/acceptance: show avocado test execution by default | expand |
On 03/05/2019 02.41, Eduardo Habkost wrote: > From: Daniel P. Berrangé <berrange@redhat.com> > > Unless overridden via an env var or configure arg, QEMU will only look > for the 'python' binary in $PATH. This is unhelpful on distros which > are only shipping Python 3.x (eg Fedora) in their default install as, > if they comply with PEP 394, the bare 'python' binary won't exist. > > This changes configure so that by default it will search for all three > common python binaries, preferring to find Python 3.x versions. > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > Message-Id: <20190327170701.23798-1-berrange@redhat.com> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> > --- > configure | 18 +++++++++++++++--- > 1 file changed, 15 insertions(+), 3 deletions(-) I haven't bisected it, but I think this patch here broke the gitlab-ci tests: https://gitlab.com/huth/qemu/-/jobs/206806257 Seems like the test is now failing when you don't have an UTF-8 locale: LANG=C make check-qapi-schema [...] TEST tests/qapi-schema/union-base-empty.out --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err 2019-05-03 15:21:39.000000000 +0000 +++ - 2019-05-03 15:42:01.561762978 +0000 @@ -1 +1 @@ -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9' /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed make: *** [check-tests/qapi-schema/unicode-str.json] Error 1 Any ideas how to fix this? Thomas
On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote: > On 03/05/2019 02.41, Eduardo Habkost wrote: > > From: Daniel P. Berrangé <berrange@redhat.com> > > > > Unless overridden via an env var or configure arg, QEMU will only look > > for the 'python' binary in $PATH. This is unhelpful on distros which > > are only shipping Python 3.x (eg Fedora) in their default install as, > > if they comply with PEP 394, the bare 'python' binary won't exist. > > > > This changes configure so that by default it will search for all three > > common python binaries, preferring to find Python 3.x versions. > > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > > Message-Id: <20190327170701.23798-1-berrange@redhat.com> > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> > > --- > > configure | 18 +++++++++++++++--- > > 1 file changed, 15 insertions(+), 3 deletions(-) > > I haven't bisected it, but I think this patch here broke the gitlab-ci tests: > > https://gitlab.com/huth/qemu/-/jobs/206806257 > > Seems like the test is now failing when you don't have an UTF-8 locale: > > LANG=C make check-qapi-schema > [...] > TEST tests/qapi-schema/union-base-empty.out > --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err 2019-05-03 15:21:39.000000000 +0000 > +++ - 2019-05-03 15:42:01.561762978 +0000 > @@ -1 +1 @@ > -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' > +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9' > /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed > make: *** [check-tests/qapi-schema/unicode-str.json] Error 1 > > Any ideas how to fix this? python3 is basically doomed if you use the C locale for LC_CTYPE, as it is not 8-bit clean. If a python3 program is liable to see UTF-8 input data, the following env should generally be set when running python: LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 Regards, Daniel
On Fri, May 03, 2019 at 05:54:35PM +0100, Daniel P. Berrangé wrote: > On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote: > > On 03/05/2019 02.41, Eduardo Habkost wrote: > > > From: Daniel P. Berrangé <berrange@redhat.com> > > > > > > Unless overridden via an env var or configure arg, QEMU will only look > > > for the 'python' binary in $PATH. This is unhelpful on distros which > > > are only shipping Python 3.x (eg Fedora) in their default install as, > > > if they comply with PEP 394, the bare 'python' binary won't exist. > > > > > > This changes configure so that by default it will search for all three > > > common python binaries, preferring to find Python 3.x versions. > > > > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > > > Message-Id: <20190327170701.23798-1-berrange@redhat.com> > > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> > > > --- > > > configure | 18 +++++++++++++++--- > > > 1 file changed, 15 insertions(+), 3 deletions(-) > > > > I haven't bisected it, but I think this patch here broke the gitlab-ci tests: > > > > https://gitlab.com/huth/qemu/-/jobs/206806257 > > > > Seems like the test is now failing when you don't have an UTF-8 locale: > > > > LANG=C make check-qapi-schema > > [...] > > TEST tests/qapi-schema/union-base-empty.out > > --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err 2019-05-03 15:21:39.000000000 +0000 > > +++ - 2019-05-03 15:42:01.561762978 +0000 > > @@ -1 +1 @@ > > -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' > > +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9' > > /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed > > make: *** [check-tests/qapi-schema/unicode-str.json] Error 1 > > > > Any ideas how to fix this? > > python3 is basically doomed if you use the C locale for LC_CTYPE, as > it is not 8-bit clean. > > If a python3 program is liable to see UTF-8 input data, the following > env should generally be set when running python: > > LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 Oh, actually I forgot we did that and then changed approach in QEMU, see these: commit 0d6b93deeeb3cc190692d629f5927befdc8b1fb8 Author: Matthias Maier <tamiko@43-1.org> Date: Mon Jun 18 19:59:58 2018 +0200 Revert commit d4e5ec877ca This commit removes the PYTHON_UTF8 workaround. The problem with setting LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 is that the en_US.UTF-8 locale might not be available. In this case setting above locales results in build errors even though another UTF-8 locale was originally set [1]. The only stable way of fixing the encoding problem is by specifying the encoding in Python, like the previous commit does. [1] https://bugs.gentoo.org/657766 Signed-off-by: Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com> Signed-off-by: Matthias Maier <tamiko@43-1.org> Message-Id: <20180618175958.29073-3-armbru@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> [Commit message tweaked] Signed-off-by: Markus Armbruster <armbru@redhat.com> commit de685ae5e9a4b523513033bd6cadc8187a227170 Author: Markus Armbruster <armbru@redhat.com> Date: Mon Jun 18 19:59:57 2018 +0200 qapi: Open files with encoding='utf-8' Python 2 happily reads UTF-8 files in text mode, but Python 3 requires either UTF-8 locale or an explicit encoding passed to open(). Commit d4e5ec877ca fixed this by setting the en_US.UTF-8 locale. Falls apart when the locale isn't be available. Matthias Maier and Arfrever Frehtes Taifersar Arahesis proposed to use binary mode instead, with manual conversion from bytes to str. Works, but opening with an explicit encoding is simpler, so do that. Since Python 2's open() doesn't support the encoding parameter, we need to suppress it with a version check. Reported-by: Arfrever Frehtes Taifersar Arahesis <arfrever.fta@gmail.com> Reported-by: Matthias Maier <tamiko@43-1.org> Signed-off-by: Markus Armbruster <armbru@redhat.com> Message-Id: <20180618175958.29073-2-armbru@redhat.com> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com> Reviewed-by: Eric Blake <eblake@redhat.com> Regards, Daniel
On 5/3/19 6:41 PM, Thomas Huth wrote: > On 03/05/2019 02.41, Eduardo Habkost wrote: >> From: Daniel P. Berrangé <berrange@redhat.com> >> >> Unless overridden via an env var or configure arg, QEMU will only look >> for the 'python' binary in $PATH. This is unhelpful on distros which >> are only shipping Python 3.x (eg Fedora) in their default install as, >> if they comply with PEP 394, the bare 'python' binary won't exist. >> >> This changes configure so that by default it will search for all three >> common python binaries, preferring to find Python 3.x versions. >> >> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> >> Message-Id: <20190327170701.23798-1-berrange@redhat.com> >> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> >> --- >> configure | 18 +++++++++++++++--- >> 1 file changed, 15 insertions(+), 3 deletions(-) > > I haven't bisected it, but I think this patch here broke the gitlab-ci tests: > > https://gitlab.com/huth/qemu/-/jobs/206806257 What's the easier way to notice that automatically? The quicker fix I have is Peter setup'ing a GitLab account mirroring his repo:staging branch, and warn him, but that won't scale much. > Seems like the test is now failing when you don't have an UTF-8 locale: > > LANG=C make check-qapi-schema > [...] > TEST tests/qapi-schema/union-base-empty.out > --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err 2019-05-03 15:21:39.000000000 +0000 > +++ - 2019-05-03 15:42:01.561762978 +0000 > @@ -1 +1 @@ > -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' > +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9' > /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed > make: *** [check-tests/qapi-schema/unicode-str.json] Error 1 > > Any ideas how to fix this? > > Thomas >
On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote: > On 03/05/2019 02.41, Eduardo Habkost wrote: > > From: Daniel P. Berrangé <berrange@redhat.com> > > > > Unless overridden via an env var or configure arg, QEMU will only look > > for the 'python' binary in $PATH. This is unhelpful on distros which > > are only shipping Python 3.x (eg Fedora) in their default install as, > > if they comply with PEP 394, the bare 'python' binary won't exist. > > > > This changes configure so that by default it will search for all three > > common python binaries, preferring to find Python 3.x versions. > > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > > Message-Id: <20190327170701.23798-1-berrange@redhat.com> > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> > > --- > > configure | 18 +++++++++++++++--- > > 1 file changed, 15 insertions(+), 3 deletions(-) > > I haven't bisected it, but I think this patch here broke the gitlab-ci tests: > > https://gitlab.com/huth/qemu/-/jobs/206806257 > > Seems like the test is now failing when you don't have an UTF-8 locale: > > LANG=C make check-qapi-schema I couldn't reproduce it this way, probably because I'm running Python 3.7 which implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale"). But I can force it to break using: PYTHONIOENCODING=ascii make check-qapi-schema > [...] > TEST tests/qapi-schema/union-base-empty.out > --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err 2019-05-03 15:21:39.000000000 +0000 > +++ - 2019-05-03 15:42:01.561762978 +0000 > @@ -1 +1 @@ > -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' > +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9' > /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed > make: *** [check-tests/qapi-schema/unicode-str.json] Error 1 > > Any ideas how to fix this? Probably we just need to specify an explicit encoding at the statement that prints the error message to stderr. I will give it a try.
On Fri, May 03, 2019 at 06:00:11PM -0300, Eduardo Habkost wrote: > On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote: > > On 03/05/2019 02.41, Eduardo Habkost wrote: > > > From: Daniel P. Berrangé <berrange@redhat.com> > > > > > > Unless overridden via an env var or configure arg, QEMU will only look > > > for the 'python' binary in $PATH. This is unhelpful on distros which > > > are only shipping Python 3.x (eg Fedora) in their default install as, > > > if they comply with PEP 394, the bare 'python' binary won't exist. > > > > > > This changes configure so that by default it will search for all three > > > common python binaries, preferring to find Python 3.x versions. > > > > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > > > Message-Id: <20190327170701.23798-1-berrange@redhat.com> > > > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> > > > --- > > > configure | 18 +++++++++++++++--- > > > 1 file changed, 15 insertions(+), 3 deletions(-) > > > > I haven't bisected it, but I think this patch here broke the gitlab-ci tests: > > > > https://gitlab.com/huth/qemu/-/jobs/206806257 > > > > Seems like the test is now failing when you don't have an UTF-8 locale: > > > > LANG=C make check-qapi-schema > > I couldn't reproduce it this way, probably because I'm running Python 3.7 which > implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale"). > > But I can force it to break using: > > PYTHONIOENCODING=ascii make check-qapi-schema > > > [...] > > TEST tests/qapi-schema/union-base-empty.out > > --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err 2019-05-03 15:21:39.000000000 +0000 > > +++ - 2019-05-03 15:42:01.561762978 +0000 > > @@ -1 +1 @@ > > -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' > > +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9' > > /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed > > make: *** [check-tests/qapi-schema/unicode-str.json] Error 1 > > > > Any ideas how to fix this? > > Probably we just need to specify an explicit encoding at the statement that > prints the error message to stderr. I will give it a try. Forcing a specific encoding inside test-qapi.py would very easy on Python 3.7+ (sys.stderr.reconfigure(...)), but tricky on older versions. I believe this is the simplest way to fix the problem on Python 3.5 and 3.6. Can somebody confirm this really fixes the problem on gitlab-ci? --- diff --git a/tests/Makefile.include b/tests/Makefile.include index 7c8b9c84b2..af88ab6f8b 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json $(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \ - $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ + PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ $^ >$*.test.out 2>$*.test.err; \ echo $$? >$*.test.exit, \ "TEST","$*.out")
On 03/05/2019 19.04, Philippe Mathieu-Daudé wrote: > On 5/3/19 6:41 PM, Thomas Huth wrote: >> On 03/05/2019 02.41, Eduardo Habkost wrote: >>> From: Daniel P. Berrangé <berrange@redhat.com> >>> >>> Unless overridden via an env var or configure arg, QEMU will only look >>> for the 'python' binary in $PATH. This is unhelpful on distros which >>> are only shipping Python 3.x (eg Fedora) in their default install as, >>> if they comply with PEP 394, the bare 'python' binary won't exist. >>> >>> This changes configure so that by default it will search for all three >>> common python binaries, preferring to find Python 3.x versions. >>> >>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> >>> Message-Id: <20190327170701.23798-1-berrange@redhat.com> >>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> >>> --- >>> configure | 18 +++++++++++++++--- >>> 1 file changed, 15 insertions(+), 3 deletions(-) >> >> I haven't bisected it, but I think this patch here broke the gitlab-ci tests: >> >> https://gitlab.com/huth/qemu/-/jobs/206806257 > > What's the easier way to notice that automatically? Maybe Peter could tweak one of his acceptance test environments to use a non-UTF-8 locale? Thomas
On 03/05/2019 23.34, Eduardo Habkost wrote: > On Fri, May 03, 2019 at 06:00:11PM -0300, Eduardo Habkost wrote: >> On Fri, May 03, 2019 at 06:41:43PM +0200, Thomas Huth wrote: >>> On 03/05/2019 02.41, Eduardo Habkost wrote: >>>> From: Daniel P. Berrangé <berrange@redhat.com> >>>> >>>> Unless overridden via an env var or configure arg, QEMU will only look >>>> for the 'python' binary in $PATH. This is unhelpful on distros which >>>> are only shipping Python 3.x (eg Fedora) in their default install as, >>>> if they comply with PEP 394, the bare 'python' binary won't exist. >>>> >>>> This changes configure so that by default it will search for all three >>>> common python binaries, preferring to find Python 3.x versions. >>>> >>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> >>>> Message-Id: <20190327170701.23798-1-berrange@redhat.com> >>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> >>>> --- >>>> configure | 18 +++++++++++++++--- >>>> 1 file changed, 15 insertions(+), 3 deletions(-) >>> >>> I haven't bisected it, but I think this patch here broke the gitlab-ci tests: >>> >>> https://gitlab.com/huth/qemu/-/jobs/206806257 >>> >>> Seems like the test is now failing when you don't have an UTF-8 locale: >>> >>> LANG=C make check-qapi-schema >> >> I couldn't reproduce it this way, probably because I'm running Python 3.7 which >> implements PEP 538 ("Coercing the legacy C locale to a UTF-8 based locale"). >> >> But I can force it to break using: >> >> PYTHONIOENCODING=ascii make check-qapi-schema >> >>> [...] >>> TEST tests/qapi-schema/union-base-empty.out >>> --- /builds/huth/qemu/tests/qapi-schema/unicode-str.err 2019-05-03 15:21:39.000000000 +0000 >>> +++ - 2019-05-03 15:42:01.561762978 +0000 >>> @@ -1 +1 @@ >>> -tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' >>> +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name '\xe9' >>> /builds/huth/qemu/tests/Makefile.include:1105: recipe for target 'check-tests/qapi-schema/unicode-str.json' failed >>> make: *** [check-tests/qapi-schema/unicode-str.json] Error 1 >>> >>> Any ideas how to fix this? >> >> Probably we just need to specify an explicit encoding at the statement that >> prints the error message to stderr. I will give it a try. > > Forcing a specific encoding inside test-qapi.py would very easy > on Python 3.7+ (sys.stderr.reconfigure(...)), but tricky on older > versions. I believe this is the simplest way to fix the problem > on Python 3.5 and 3.6. > > Can somebody confirm this really fixes the problem on gitlab-ci? > > --- > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 7c8b9c84b2..af88ab6f8b 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) > .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y)) > $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json > $(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \ > - $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ > + PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \ > $^ >$*.test.out 2>$*.test.err; \ > echo $$? >$*.test.exit, \ > "TEST","$*.out") Yes, thanks, this seems to fix this issue: https://gitlab.com/huth/qemu/-/jobs/207041381 It also works on my laptop now (which uses Python 3.6). Thomas
diff --git a/configure b/configure index 60719ddcc5..f88011b94d 100755 --- a/configure +++ b/configure @@ -899,7 +899,18 @@ fi : ${make=${MAKE-make}} : ${install=${INSTALL-install}} -: ${python=${PYTHON-python}} +# We prefer python 3.x. A bare 'python' is traditionally +# python 2.x, but some distros have it as python 3.x, so +# we check that before python2 +python= +for binary in "${PYTHON-python3}" python python2 +do + if has "$binary" + then + python="$binary" + break + fi +done : ${smbd=${SMBD-/usr/sbin/smbd}} # Default objcc to clang if available, otherwise use CC @@ -1818,8 +1829,9 @@ EOF exit 0 fi -if ! has $python; then - error_exit "Python not found. Use --python=/path/to/python" +if test -z "$python" +then + error_exit "Python not found. Use --python=/path/to/python" fi # Note that if the Python conditional here evaluates True we will exit