diff mbox

python modules search in non english locale (zlib, ...)

Message ID 5515E2AC.3010100@armadeus.com
State Not Applicable
Headers show

Commit Message

Sébastien Royen March 27, 2015, 11:07 p.m. UTC
setup.py: do not add crosscompile header locations if language is not
english

With buildroot toolchain, gcc can be multi language.
Python package setup.py need gcc english output to work fine. (find zlib
for example)
We force language to en_US for the need of output parsing.

Signed-off-by: Sebastien Royen <sebastien.royen@armadeus.com>
         in_incdirs = False
         inc_dirs = []

Comments

Thomas Petazzoni March 29, 2015, 12:46 p.m. UTC | #1
Dear Sébastien Royen,

On Sat, 28 Mar 2015 00:07:24 +0100, Sébastien Royen wrote:
> setup.py: do not add crosscompile header locations if language is not
> english
> 
> With buildroot toolchain, gcc can be multi language.
> Python package setup.py need gcc english output to work fine. (find zlib
> for example)
> We force language to en_US for the need of output parsing.
> 
> Signed-off-by: Sebastien Royen <sebastien.royen@armadeus.com>

What you posted is not a Buildroot patch, so we cannot apply it. I
guess it's a python or python3 patch. Can you rework this to make it
actually usable by Buildroot, and also make sure whether the issue is
applicable to python, python3, or both?

Thanks,

Thomas
Sébastien Royen March 29, 2015, 7:08 p.m. UTC | #2
Sorry,

I should not submit patch of patch too late in the evening.
I hope the version below is better.
Not tested with python3, and not sure toolchain output language shouln't
be fix in another way to avoid same problem on other packages.

Le 29/03/2015 14:46, Thomas Petazzoni a écrit :
> Dear Sébastien Royen,
>
> On Sat, 28 Mar 2015 00:07:24 +0100, Sébastien Royen wrote:
>> setup.py: do not add crosscompile header locations if language is not
>> english
>>
>> With buildroot toolchain, gcc can be multi language.
>> Python package setup.py need gcc english output to work fine. (find zlib
>> for example)
>> We force language to en_US for the need of output parsing.
>>
>> Signed-off-by: Sebastien Royen <sebastien.royen@armadeus.com>
> What you posted is not a Buildroot patch, so we cannot apply it. I
> guess it's a python or python3 patch. Can you rework this to make it
> actually usable by Buildroot, and also make sure whether the issue is
> applicable to python, python3, or both?
>
> Thanks,
>
> Thomas
--- a/package/python/116-enforce-cross-compile-headers-search.patch  
 1970-01-01 01:00:00.000000000 +0100
+++ b/package/python/116-enforce-cross-compile-headers-search.patch  
 2015-03-27 23:58:21.478991279 +0100
@@ -0,0 +1,20 @@
+setup.py: do not add crosscompile header locations if language is not
english
+
+With buildroot toolchain, gcc is multi language.
+setup.py need english output to work fine. (find zlib for example)
+We force language to en_US for the need of output parsing.
+
+Signed-off-by: Sebastien Royen <sebastien.royen@armadeus.com>
+Index: b/setup.py
+===================================================================
+--- a/setup.py    2015-03-27 23:55:53.738987211 +0100
++++ b/setup.py    2015-03-27 23:44:48.482968892 +0100
+@@ -414,7 +414,7 @@
+         tmpfile = os.path.join(self.build_temp, 'gccpaths')
+         if not os.path.exists(self.build_temp):
+             os.makedirs(self.build_temp)
+-        ret = os.system('%s -E -v - </dev/null 2>%s 1>/dev/null' %
(gcc, tmpfile))
++        ret = os.system('LANGUAGE=en_US %s -E -v - </dev/null 2>%s
1>/dev/null' % (gcc, tmpfile))
+         is_gcc = False
+         in_incdirs = False
+         inc_dirs = []
Thomas Petazzoni March 29, 2015, 8:40 p.m. UTC | #3
Dear Sébastien Royen,

On Sun, 29 Mar 2015 21:08:16 +0200, Sébastien Royen wrote:

> I should not submit patch of patch too late in the evening.
> I hope the version below is better.

Yes and no. It's better because it's a Buildroot patch, but it's still
not good since it's not a Git formatted patch.

> Not tested with python3, and not sure toolchain output language shouln't
> be fix in another way to avoid same problem on other packages.

Maybe we should export LC_ALL globally, or something like that.

Thomas
Yegor Yefremov March 30, 2015, 11:05 a.m. UTC | #4
On Sun, Mar 29, 2015 at 10:40 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Sébastien Royen,
>
> On Sun, 29 Mar 2015 21:08:16 +0200, Sébastien Royen wrote:
>
>> I should not submit patch of patch too late in the evening.
>> I hope the version below is better.
>
> Yes and no. It's better because it's a Buildroot patch, but it's still
> not good since it's not a Git formatted patch.
>
>> Not tested with python3, and not sure toolchain output language shouln't
>> be fix in another way to avoid same problem on other packages.
>
> Maybe we should export LC_ALL globally, or something like that.

Can confirm, that I'm having the same issue with my German openSUSE
12.1 machine. Exporting LANGUAGE=en_US helped. haven't tried LC_ALL
yet. Python 2.7.9.

The whole stuff got screwed only lately. I've been using python-dpkt,
that requires PYTHON_ZLIB for ages and had no problems on this
machine.

I think this bug (https://bugs.busybox.net/show_bug.cgi?id=7971) is
also locale related.

Yegor
Thomas Petazzoni March 30, 2015, 11:49 a.m. UTC | #5
Dear Yegor Yefremov,

On Mon, 30 Mar 2015 13:05:09 +0200, Yegor Yefremov wrote:

> Can confirm, that I'm having the same issue with my German openSUSE
> 12.1 machine. Exporting LANGUAGE=en_US helped. haven't tried LC_ALL
> yet. Python 2.7.9.

Ok.

> The whole stuff got screwed only lately. I've been using python-dpkt,
> that requires PYTHON_ZLIB for ages and had no problems on this
> machine.

Seems weird. Do we have an idea what broke things ? Can you try 2015.02,
2014.11 and see if the issue also happens (or not) ?

> I think this bug (https://bugs.busybox.net/show_bug.cgi?id=7971) is
> also locale related.

Ok.

Thomas
Yegor Yefremov March 30, 2015, 1:37 p.m. UTC | #6
On Mon, Mar 30, 2015 at 1:49 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Yegor Yefremov,
>
> On Mon, 30 Mar 2015 13:05:09 +0200, Yegor Yefremov wrote:
>
>> Can confirm, that I'm having the same issue with my German openSUSE
>> 12.1 machine. Exporting LANGUAGE=en_US helped. haven't tried LC_ALL
>> yet. Python 2.7.9.
>
> Ok.
>
>> The whole stuff got screwed only lately. I've been using python-dpkt,
>> that requires PYTHON_ZLIB for ages and had no problems on this
>> machine.
>
> Seems weird. Do we have an idea what broke things ? Can you try 2015.02,
> 2014.11 and see if the issue also happens (or not) ?

Hm. Strange. 2014.11 seem to have the same issue. Can it be the change
from Python 2.7.8 to 2.7.9?

# cat /etc/os-release
NAME=Buildroot
VERSION=2014.11
ID=buildroot
VERSION_ID=2014.11
PRETTY_NAME="Buildroot 2014.11"

I'm also working with en englich Xubuntu VM. I can imagine, that I
haven't made updates/cleaning on my suse machine. Most upstream work
was made with Xubuntu.

>> I think this bug (https://bugs.busybox.net/show_bug.cgi?id=7971) is
>> also locale related.
>
> Ok.
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com
Yegor Yefremov March 30, 2015, 2:05 p.m. UTC | #7
On Mon, Mar 30, 2015 at 3:37 PM, Yegor Yefremov
<yegorslists@googlemail.com> wrote:
> On Mon, Mar 30, 2015 at 1:49 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Dear Yegor Yefremov,
>>
>> On Mon, 30 Mar 2015 13:05:09 +0200, Yegor Yefremov wrote:
>>
>>> Can confirm, that I'm having the same issue with my German openSUSE
>>> 12.1 machine. Exporting LANGUAGE=en_US helped. haven't tried LC_ALL
>>> yet. Python 2.7.9.
>>
>> Ok.
>>
>>> The whole stuff got screwed only lately. I've been using python-dpkt,
>>> that requires PYTHON_ZLIB for ages and had no problems on this
>>> machine.
>>
>> Seems weird. Do we have an idea what broke things ? Can you try 2015.02,
>> 2014.11 and see if the issue also happens (or not) ?
>
> Hm. Strange. 2014.11 seem to have the same issue. Can it be the change
> from Python 2.7.8 to 2.7.9?
>
> # cat /etc/os-release
> NAME=Buildroot
> VERSION=2014.11
> ID=buildroot
> VERSION_ID=2014.11
> PRETTY_NAME="Buildroot 2014.11"
>
> I'm also working with en englich Xubuntu VM. I can imagine, that I
> haven't made updates/cleaning on my suse machine. Most upstream work
> was made with Xubuntu.

Python 3.4.2 seems to have this issue too.

Yegor
Yegor Yefremov March 31, 2015, 9:05 a.m. UTC | #8
On Mon, Mar 30, 2015 at 4:05 PM, Yegor Yefremov
<yegorslists@googlemail.com> wrote:
> On Mon, Mar 30, 2015 at 3:37 PM, Yegor Yefremov
> <yegorslists@googlemail.com> wrote:
>> On Mon, Mar 30, 2015 at 1:49 PM, Thomas Petazzoni
>> <thomas.petazzoni@free-electrons.com> wrote:
>>> Dear Yegor Yefremov,
>>>
>>> On Mon, 30 Mar 2015 13:05:09 +0200, Yegor Yefremov wrote:
>>>
>>>> Can confirm, that I'm having the same issue with my German openSUSE
>>>> 12.1 machine. Exporting LANGUAGE=en_US helped. haven't tried LC_ALL
>>>> yet. Python 2.7.9.
>>>
>>> Ok.
>>>
>>>> The whole stuff got screwed only lately. I've been using python-dpkt,
>>>> that requires PYTHON_ZLIB for ages and had no problems on this
>>>> machine.
>>>
>>> Seems weird. Do we have an idea what broke things ? Can you try 2015.02,
>>> 2014.11 and see if the issue also happens (or not) ?
>>
>> Hm. Strange. 2014.11 seem to have the same issue. Can it be the change
>> from Python 2.7.8 to 2.7.9?
>>
>> # cat /etc/os-release
>> NAME=Buildroot
>> VERSION=2014.11
>> ID=buildroot
>> VERSION_ID=2014.11
>> PRETTY_NAME="Buildroot 2014.11"
>>
>> I'm also working with en englich Xubuntu VM. I can imagine, that I
>> haven't made updates/cleaning on my suse machine. Most upstream work
>> was made with Xubuntu.
>
> Python 3.4.2 seems to have this issue too.

Tried to build image via Sourcery CodeBench ARM 2014.05. Python 2.7.9
shows no problems. But when I'm doing the same with BR's toolchain,
zlib module won't be found at runtime.

BR2_arm=y
BR2_cortex_a8=y
BR2_CCACHE=y
BR2_TOOLCHAIN_BUILDROOT_LARGEFILE=y
BR2_TOOLCHAIN_BUILDROOT_INET_IPV6=y
BR2_TOOLCHAIN_BUILDROOT_INET_RPC=y
BR2_TOOLCHAIN_BUILDROOT_WCHAR=y
BR2_BINUTILS_VERSION_2_25=y
BR2_GCC_VERSION_4_9_X=y
BR2_PACKAGE_PYTHON=y
BR2_PACKAGE_PYTHON_ZLIB=y
Sébastien Royen March 31, 2015, 9:32 a.m. UTC | #9
Le 31/03/2015 11:05, Yegor Yefremov a écrit :
> On Mon, Mar 30, 2015 at 4:05 PM, Yegor Yefremov
> <yegorslists@googlemail.com> wrote:
>> On Mon, Mar 30, 2015 at 3:37 PM, Yegor Yefremov
>> <yegorslists@googlemail.com> wrote:
>>> On Mon, Mar 30, 2015 at 1:49 PM, Thomas Petazzoni
>>> <thomas.petazzoni@free-electrons.com> wrote:
>>>> Dear Yegor Yefremov,
>>>>
>>>> On Mon, 30 Mar 2015 13:05:09 +0200, Yegor Yefremov wrote:
>>>>
>>>>> Can confirm, that I'm having the same issue with my German openSUSE
>>>>> 12.1 machine. Exporting LANGUAGE=en_US helped. haven't tried LC_ALL
>>>>> yet. Python 2.7.9.
>>>> Ok.
>>>>
>>>>> The whole stuff got screwed only lately. I've been using python-dpkt,
>>>>> that requires PYTHON_ZLIB for ages and had no problems on this
>>>>> machine.
>>>> Seems weird. Do we have an idea what broke things ? Can you try 2015.02,
>>>> 2014.11 and see if the issue also happens (or not) ?
>>> Hm. Strange. 2014.11 seem to have the same issue. Can it be the change
>>> from Python 2.7.8 to 2.7.9?
>>>
>>> # cat /etc/os-release
>>> NAME=Buildroot
>>> VERSION=2014.11
>>> ID=buildroot
>>> VERSION_ID=2014.11
>>> PRETTY_NAME="Buildroot 2014.11"
>>>
>>> I'm also working with en englich Xubuntu VM. I can imagine, that I
>>> haven't made updates/cleaning on my suse machine. Most upstream work
>>> was made with Xubuntu.
>> Python 3.4.2 seems to have this issue too.
> Tried to build image via Sourcery CodeBench ARM 2014.05. Python 2.7.9
> shows no problems. But when I'm doing the same with BR's toolchain,
> zlib module won't be found at runtime.
>
> BR2_arm=y
> BR2_cortex_a8=y
> BR2_CCACHE=y
> BR2_TOOLCHAIN_BUILDROOT_LARGEFILE=y
> BR2_TOOLCHAIN_BUILDROOT_INET_IPV6=y
> BR2_TOOLCHAIN_BUILDROOT_INET_RPC=y
> BR2_TOOLCHAIN_BUILDROOT_WCHAR=y
> BR2_BINUTILS_VERSION_2_25=y
> BR2_GCC_VERSION_4_9_X=y
> BR2_PACKAGE_PYTHON=y
> BR2_PACKAGE_PYTHON_ZLIB=y
>
Hi,

Problem is on parsing output of : gcc -E -v - </dev/null 1>/dev/null
Maybe it would be better to fix setup.py in order to be generic.
Output with en_US is like :
...
#include <...> search starts here:
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/include
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/include-fixed
 /my_path/bsp/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/../../../../arm-buildroot-linux-gnueabihf/include
 /my_path/buildroot/output/host/usr/arm-buildroot-linux-gnueabihf/sysroot/usr/include
End of search list.
...

Output with de is like :
...
#include <...> - Suche beginnt hier:
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/include
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/include-fixed
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/../../../../arm-buildroot-linux-gnueabihf/include
 /my_path/buildroot/output/host/usr/arm-buildroot-linux-gnueabihf/sysroot/usr/include
Ende der Suchliste.
...

Output with fr version is :
la recherche pour #include <...> débute ici:
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/include
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/include-fixed
 /my_path/buildroot/output/host/usr/lib/gcc/arm-buildroot-linux-gnueabihf/4.8.4/../../../../arm-buildroot-linux-gnueabihf/include
 /my_path/buildroot/output/host/usr/arm-buildroot-linux-gnueabihf/sysroot/usr/include
Fin de la liste de recherche.
...

Or maybe we should keep only C language in toolchain build ? ( something
to do with BR2_ENABLE_LOCALE_WHITELIST ? )
diff mbox

Patch

Index: b/setup.py
===================================================================
--- a/setup.py    2015-03-27 23:55:53.738987211 +0100
+++ b/setup.py    2015-03-27 23:44:48.482968892 +0100
@@ -414,7 +414,7 @@ 
         tmpfile = os.path.join(self.build_temp, 'gccpaths')
         if not os.path.exists(self.build_temp):
             os.makedirs(self.build_temp)
-        ret = os.system('%s -E -v - </dev/null 2>%s 1>/dev/null' %
(gcc, tmpfile))
+        ret = os.system('LANGUAGE=en_US %s -E -v - </dev/null 2>%s
1>/dev/null' % (gcc, tmpfile))
         is_gcc = False