| Message ID | 20190801210920.31338-4-romain.naour@smile.fr |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [1/5] package/python-numpy: fix occasional build and run-time failure with lapack | expand |
Hi Romain, all, > -----Original Message----- > From: Romain Naour <romain.naour@smile.fr> > Sent: Friday, August 2, 2019 12:09 AM > To: buildroot@buildroot.org > Cc: Damien DUVAL <damien.duval@smile.fr>; Alexandre PAYEN <alexandre.payen@smile.fr>; Alexey Brodkin > <abrodkin@synopsys.com>; Romain Naour <romain.naour@smile.fr>; Thomas Petazzoni > <thomas.petazzoni@free-electrons.com>; Alexey Brodkin <abrodkin@synopsys.com>; Fabrice Fontaine > <fontaine.fabrice@gmail.com>; Evgeniy Didin <didin@synopsys.com> > Subject: [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by libc > > From: Damien DUVAL <damien.duval@smile.fr> > > With a c library which does not provide fenv.h, it won't work at runtime: > Crash after an "import numpy" on python. > > Since numpy v1.16.0: > "Alpine Linux (and other musl c library distros) support > We now default to use fenv.h for floating point status error reporting. > Previously we had a broken default that sometimes would not report > underflow, overflow, and invalid floating point operations. Now we can > support non-glibc distrubutions like Alpine Linux as long as they ship > fenv.h." > Also there is a lack of define in glibc for ARC architecture and python > numpy cannot build. > The removed patches were just fixes for the build for uclibc and glibc > for ARC Well we used to lack fenv.h in ARC's port back in the day but not any longer, check [1] as what we have in arc-2019.03 release so I guess since [2] commit be0aaaaecda5 ("toolchain: bump ARC tools to arc-2019.03 release") numpy should build and work for ARC with glibc as good as for other arches. > diff --git a/package/python-numpy/Config.in b/package/python-numpy/Config.in > index 6d0b373413..bcd51e6000 100644 > --- a/package/python-numpy/Config.in > +++ b/package/python-numpy/Config.in > @@ -16,6 +16,10 @@ config BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS > config BR2_PACKAGE_PYTHON_NUMPY > bool "python-numpy" > depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS > + # python-numpy needs fenv.h which is not provided by uclibc > + depends on BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL > + # python-numpy is not compatible with glibc for ARC architecture. > + depends on !(BR2_TOOLCHAIN_USES_GLIBC && BR2_arc) > select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK Given my comment above can we try it on ARC? I.e. what exactly is a test-case so that we may try it and confirm? [1] https://github.com/foss-for-synopsys-dwc-arc-processors/glibc/blob/arc-2019.03-release/sysdeps/arc/bits/fenv.h [2] https://git.buildroot.org/buildroot/commit/?id=be0aaaaecda56d16acf99a38319293913fd1e0b5 -Alexey
On 01/08/2019 23:09, Romain Naour wrote: > From: Damien DUVAL <damien.duval@smile.fr> > > With a c library which does not provide fenv.h, it won't work at runtime: > Crash after an "import numpy" on python. > > Since numpy v1.16.0: > "Alpine Linux (and other musl c library distros) support > We now default to use fenv.h for floating point status error reporting. > Previously we had a broken default that sometimes would not report > underflow, overflow, and invalid floating point operations. Now we can > support non-glibc distrubutions like Alpine Linux as long as they ship > fenv.h." > Also there is a lack of define in glibc for ARC architecture and python > numpy cannot build. > The removed patches were just fixes for the build for uclibc and glibc > for ARC Since Alexey indicated that this is not needed any longer, I removed that part and applied to master, thanks. Alexey, could you do a runtime test (based on the one submitted by Romain) for ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm asking you instead of doing it myself :-) Regards, Arnout > > Configurations which does not provide fenv.h (like uclibc) won't be able > to select the python-numpy package to avoid errors at runtime. > > Signed-off-by: Damien DUVAL <damien.duval@smile.fr> > Signed-off-by: Alexandre PAYEN <alexandre.payen@smile.fr> > Cc: Alexey Brodkin <Alexey.Brodkin@synopsys.com> > Signed-off-by: Romain Naour <romain.naour@smile.fr>
Hi Arnout, > Since Alexey indicated that this is not needed any longer, I removed that part > and applied to master, thanks. > > Alexey, could you do a runtime test (based on the one submitted by Romain) for > ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm > asking you instead of doing it myself :-) Any hint on why QEMU is in the question? I.e. is there any framework or test-suite that require QEMU? What about real boards, proprietary simulators? -Alexey
On 03/08/2019 12:48, Alexey Brodkin wrote: > Hi Arnout, > >> Since Alexey indicated that this is not needed any longer, I removed that part >> and applied to master, thanks. >> >> Alexey, could you do a runtime test (based on the one submitted by Romain) for >> ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm >> asking you instead of doing it myself :-) > > Any hint on why QEMU is in the question? > I.e. is there any framework or test-suite that require QEMU? Yes, the Buildroot runtime tests in support/testing use qemu. Regards, Arnout > What about real boards, proprietary simulators? > > -Alexey >
Hi Arnout, > -----Original Message----- > From: Arnout Vandecappelle <arnout@mind.be> > Sent: Saturday, August 3, 2019 3:25 PM > To: Alexey Brodkin <abrodkin@synopsys.com> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>; Alexandre PAYEN > <alexandre.payen@smile.fr>; Damien DUVAL <damien.duval@smile.fr>; Evgeniy Didin <didin@synopsys.com>; > Fabrice Fontaine <fontaine.fabrice@gmail.com>; Romain Naour <romain.naour@smile.fr>; > buildroot@buildroot.org > Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by > libc > > > > On 03/08/2019 12:48, Alexey Brodkin wrote: > > Hi Arnout, > > > >> Since Alexey indicated that this is not needed any longer, I removed that part > >> and applied to master, thanks. > >> > >> Alexey, could you do a runtime test (based on the one submitted by Romain) for > >> ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm > >> asking you instead of doing it myself :-) > > > > Any hint on why QEMU is in the question? > > I.e. is there any framework or test-suite that require QEMU? > > Yes, the Buildroot runtime tests in support/testing use qemu. That's cool, though I haven't tried that yet so may I ask what do I need to run for testing numpy in particular? -Alexey P.S. Pls pardon my noob's questions - if there's an answer in form of one-liner that would be super cool, otherwise docs will work.
On 03/08/2019 14:49, Alexey Brodkin wrote: > Hi Arnout, > >> -----Original Message----- >> From: Arnout Vandecappelle <arnout@mind.be> >> Sent: Saturday, August 3, 2019 3:25 PM >> To: Alexey Brodkin <abrodkin@synopsys.com> >> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>; Alexandre PAYEN >> <alexandre.payen@smile.fr>; Damien DUVAL <damien.duval@smile.fr>; Evgeniy Didin <didin@synopsys.com>; >> Fabrice Fontaine <fontaine.fabrice@gmail.com>; Romain Naour <romain.naour@smile.fr>; >> buildroot@buildroot.org >> Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by >> libc >> >> >> >> On 03/08/2019 12:48, Alexey Brodkin wrote: >>> Hi Arnout, >>> >>>> Since Alexey indicated that this is not needed any longer, I removed that part >>>> and applied to master, thanks. >>>> >>>> Alexey, could you do a runtime test (based on the one submitted by Romain) for >>>> ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm >>>> asking you instead of doing it myself :-) >>> >>> Any hint on why QEMU is in the question? >>> I.e. is there any framework or test-suite that require QEMU? >> >> Yes, the Buildroot runtime tests in support/testing use qemu. > > That's cool, though I haven't tried that yet so may I ask what do I > need to run for testing numpy in particular? Specifically for numpy, there is no runtime test yet - Romain submitted a patch [1] but it needs a new iteration. But I realise now that in your case the runtime testing infra isn't that useful since you anyway need to change the test completely to get the right arch and kernel. So yes, by all means, test it on the board :-) You just need to add this to the config: BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_NUMPY=y And run this in Python: import numpy as np list = [1,2,3,4] arr = np.array(list) print (arr) Regards, Arnout [1] http://patchwork.ozlabs.org/patch/1140773/ > > -Alexey > > P.S. Pls pardon my noob's questions - if there's an answer in > form of one-liner that would be super cool, otherwise docs will work. >
Hi Arnout, all, > -----Original Message----- > From: Arnout Vandecappelle <arnout@mind.be> > Sent: Saturday, August 3, 2019 5:12 PM > To: Alexey Brodkin <abrodkin@synopsys.com> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>; Alexandre PAYEN > <alexandre.payen@smile.fr>; Damien DUVAL <damien.duval@smile.fr>; Evgeniy Didin <didin@synopsys.com>; > Fabrice Fontaine <fontaine.fabrice@gmail.com>; Romain Naour <romain.naour@smile.fr>; > buildroot@buildroot.org > Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided by > libc > > > > On 03/08/2019 14:49, Alexey Brodkin wrote: > > Hi Arnout, > > > >> -----Original Message----- > >> From: Arnout Vandecappelle <arnout@mind.be> > >> Sent: Saturday, August 3, 2019 3:25 PM > >> To: Alexey Brodkin <abrodkin@synopsys.com> > >> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>; Alexandre PAYEN > >> <alexandre.payen@smile.fr>; Damien DUVAL <damien.duval@smile.fr>; Evgeniy Didin > <didin@synopsys.com>; > >> Fabrice Fontaine <fontaine.fabrice@gmail.com>; Romain Naour <romain.naour@smile.fr>; > >> buildroot@buildroot.org > >> Subject: Re: [Buildroot] [PATCH 4/5] package/python-numpy: disable numpy if fenv.h is not provided > by > >> libc > >> > >> > >> > >> On 03/08/2019 12:48, Alexey Brodkin wrote: > >>> Hi Arnout, > >>> > >>>> Since Alexey indicated that this is not needed any longer, I removed that part > >>>> and applied to master, thanks. > >>>> > >>>> Alexey, could you do a runtime test (based on the one submitted by Romain) for > >>>> ARC to be sure? You'll need a qemu for ARC for that, obviously, which is why I'm > >>>> asking you instead of doing it myself :-) > >>> > >>> Any hint on why QEMU is in the question? > >>> I.e. is there any framework or test-suite that require QEMU? > >> > >> Yes, the Buildroot runtime tests in support/testing use qemu. > > > > That's cool, though I haven't tried that yet so may I ask what do I > > need to run for testing numpy in particular? > > Specifically for numpy, there is no runtime test yet - Romain submitted a patch > [1] but it needs a new iteration. > > But I realise now that in your case the runtime testing infra isn't that useful > since you anyway need to change the test completely to get the right arch and > kernel. So yes, by all means, test it on the board :-) > > You just need to add this to the config: > > BR2_PACKAGE_PYTHON3=y > BR2_PACKAGE_PYTHON_NUMPY=y > > And run this in Python: > > import numpy as np > list = [1,2,3,4] > arr = np.array(list) > print (arr) It works perfectly fine: ------------------------------>8--------------------------------- # python Python 3.7.4 (default, Aug 6 2019, 12:07:44) [GCC 8.3.1 20190225] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> list = [1,2,3,4] >>> >>> arr = np.array(list) >>> print (arr) [1 2 3 4] >>> # uname -a Linux buildroot 5.2.5 #2 PREEMPT Tue Aug 6 12:10:30 MSK 2019 arc GNU/Linux ------------------------------>8--------------------------------- -Alexey
diff --git a/package/python-numpy/0001-Don-t-use-fenv.h-on-uClibc.patch b/package/python-numpy/0001-Don-t-use-fenv.h-on-uClibc.patch deleted file mode 100644 index 8b3937a1ba..0000000000 --- a/package/python-numpy/0001-Don-t-use-fenv.h-on-uClibc.patch +++ /dev/null @@ -1,30 +0,0 @@ -From 52b47439d17463304e5bd7974dec17ced0b0f24a Mon Sep 17 00:00:00 2001 -From: Fabrice Fontaine <fontaine.fabrice@gmail.com> -Date: Sat, 16 Mar 2019 10:38:27 +0100 -Subject: [PATCH] Don't use <fenv.h> on uClibc - -Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> -Signed-off-by: Alexey Brodkin <abrodkin@synopsys.com> -Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com> -[Upstream status: https://github.com/numpy/numpy/pull/13137] ---- - numpy/core/src/npymath/ieee754.c.src | 3 ++- - 1 file changed, 2 insertions(+), 1 deletion(-) - -diff --git a/numpy/core/src/npymath/ieee754.c.src b/numpy/core/src/npymath/ieee754.c.src -index d960838c8..f3f15f841 100644 ---- a/numpy/core/src/npymath/ieee754.c.src -+++ b/numpy/core/src/npymath/ieee754.c.src -@@ -681,7 +681,8 @@ void npy_set_floatstatus_invalid(void) - fp_raise_xcp(FP_INVALID); - } - --#elif defined(_MSC_VER) || (defined(__osf__) && defined(__alpha)) -+#elif defined(_MSC_VER) || (defined(__osf__) && defined(__alpha)) || \ -+ defined (__UCLIBC__) - - /* - * By using a volatile floating point value, --- -2.14.1 - diff --git a/package/python-numpy/0002-FIX-no-FPU-exceptions-bits-on-ARC-glibc.patch b/package/python-numpy/0002-FIX-no-FPU-exceptions-bits-on-ARC-glibc.patch deleted file mode 100644 index ece52118d7..0000000000 --- a/package/python-numpy/0002-FIX-no-FPU-exceptions-bits-on-ARC-glibc.patch +++ /dev/null @@ -1,31 +0,0 @@ -From 1125f1ee33324bc91b4e8dd9da49163af572d04a Mon Sep 17 00:00:00 2001 -From: Fabrice Fontaine <fontaine.fabrice@gmail.com> -Date: Sat, 16 Mar 2019 10:48:25 +0100 -Subject: [PATCH] FIX: no FPU exceptions bits on ARC glibc - -The FPU exceptions bits are missing in fenv.h in glibc for ARC -architecture. - -Signed-off-by: Evgeniy Didin <Evgeniy.Didin@synopsys.com> -Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com> -[Upstream status: https://github.com/numpy/numpy/pull/13137] ---- - numpy/core/src/npymath/ieee754.c.src | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -diff --git a/numpy/core/src/npymath/ieee754.c.src b/numpy/core/src/npymath/ieee754.c.src -index f3f15f841..3f66b24a4 100644 ---- a/numpy/core/src/npymath/ieee754.c.src -+++ b/numpy/core/src/npymath/ieee754.c.src -@@ -682,7 +682,7 @@ void npy_set_floatstatus_invalid(void) - } - - #elif defined(_MSC_VER) || (defined(__osf__) && defined(__alpha)) || \ -- defined (__UCLIBC__) -+ defined (__UCLIBC__) || (defined(__arc__) && defined(__GLIBC__)) - - /* - * By using a volatile floating point value, --- -2.14.1 - diff --git a/package/python-numpy/Config.in b/package/python-numpy/Config.in index 6d0b373413..bcd51e6000 100644 --- a/package/python-numpy/Config.in +++ b/package/python-numpy/Config.in @@ -16,6 +16,10 @@ config BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS config BR2_PACKAGE_PYTHON_NUMPY bool "python-numpy" depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS + # python-numpy needs fenv.h which is not provided by uclibc + depends on BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL + # python-numpy is not compatible with glibc for ARC architecture. + depends on !(BR2_TOOLCHAIN_USES_GLIBC && BR2_arc) select BR2_PACKAGE_LAPACK_COMPLEX if BR2_PACKAGE_LAPACK help NumPy is the fundamental package for scientific computing @@ -25,3 +29,9 @@ config BR2_PACKAGE_PYTHON_NUMPY C library. http://www.numpy.org/ + +comment "python-numpy needs glibc or musl" + depends on !(BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL) + +comment "python-numpy is not compatible with glibc for ARC architecture" + depends on BR2_TOOLCHAIN_USES_GLIBC && BR2_arc