From patchwork Mon Sep 30 08:43:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Esben Haabendal X-Patchwork-Id: 1169181 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=geanix.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=geanix.com header.i=@geanix.com header.b="IyBz/m/d"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46hbXT048vz9sDB for ; Mon, 30 Sep 2019 18:43:36 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id EE60085631; Mon, 30 Sep 2019 08:43:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MfuFvPijmUWa; Mon, 30 Sep 2019 08:43:33 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id E5E5885683; Mon, 30 Sep 2019 08:43:32 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id C61C81BF228 for ; Mon, 30 Sep 2019 08:43:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C3122203E7 for ; Mon, 30 Sep 2019 08:43:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GxuAlIcpph-b for ; Mon, 30 Sep 2019 08:43:30 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from first.geanix.com (first.geanix.com [116.203.34.67]) by silver.osuosl.org (Postfix) with ESMTPS id 6DBAD20033 for ; Mon, 30 Sep 2019 08:43:30 +0000 (UTC) Received: from localhost (unknown [193.163.1.7]) by first.geanix.com (Postfix) with ESMTPSA id B242584AB8; Mon, 30 Sep 2019 08:42:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=geanix.com; s=first; t=1569832957; bh=U8/h7ui4WanbF9ZSMBgPIgekghLM/l3cmb3zCeApQEM=; h=From:To:Cc:Subject:Date; b=IyBz/m/dZbRe/Tq0BE+hqH17FCRu+HfPN/ulMZLDTEoDUu5j8cSKKfrjnVgKIP9e3 S2tQ4nbul8RbxVxl4KbxY+8RAC/Z4l7/xmTRASebgVd/8FggeicWpMk5NC2NNU/Z7G +QIpiZ+tpbenXMRfr1oClWeL20xdulXPAThP63W/CKrFf2J4KXCBgIO0wJKYlPjyEz Q2Q5y0t5/tbkgtgBxuMFrF0CPNTKmmK/vAkcbNMzzmpb8qEJz6DusEudOpjW/rnEcq s5+eQ/pla05Bcdy9LktCc+jbOZv7sJeMYyjqm6khFdmnJ2T6zqR8L0kQgrzEQjeuT4 AAATTv/0jcrqQ== From: Esben Haabendal To: buildroot@buildroot.org Date: Mon, 30 Sep 2019 10:43:25 +0200 Message-Id: <20190930084326.4414-1-esben@geanix.com> X-Mailer: git-send-email 2.23.0 MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 1/2] package/python-numpy: fixup numpy distutils for cross compilation X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexandre PAYEN , Asaf Kahlon Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Fix problems using the numpy distutils extension for cross compilation, so that linking with npymath library will use target library when building target packages. Signed-off-by: Esben Haabendal --- .../0001-NPY_PKG_CONFIG_PATH.patch | 85 +++++++++++++++++++ package/python-numpy/python-numpy.mk | 21 +++++ 2 files changed, 106 insertions(+) create mode 100644 package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch diff --git a/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch b/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch new file mode 100644 index 000000000000..3e8620a37cb8 --- /dev/null +++ b/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch @@ -0,0 +1,85 @@ +commit 153fc148eec60e5cbec0e80617f75a3a5dd2a3f8 +Author: Esben Haabendal +Date: Thu Sep 12 21:59:58 2019 +0200 + + ENH: Allow NPY_PKG_CONFIG_PATH environment variable override + + Allow overriding npy-pkg-config directory using the NPY_PKG_CONFIG_PATH + environment variable, making it easier to use numpy in cross-compilation + setups. + +Upstream-Status: Accepted (scheduled for 1.18.0) +Signed-off-by: Esben Haabendal + +diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py +index 1e10e92fdadf..0eaaeb7364d4 100644 +--- a/numpy/distutils/misc_util.py ++++ b/numpy/distutils/misc_util.py +@@ -1687,6 +1687,43 @@ class Configuration(object): + + and will be installed as foo.ini in the 'lib' subpath. + ++ Cross-compilation ++ ----------------- ++ When cross-compiling with numpy distutils, it might be necessary to ++ use modified npy-pkg-config files. Using the default/generated files ++ will link with the host libraries (i.e. libnpymath.a). For ++ cross-compilation you of-course need to link with target libraries, ++ while using the host Python installation. ++ ++ You can copy out the numpy/core/lib/npy-pkg-config directory, add a ++ pkgdir value to the .ini files and set NPY_PKG_CONFIG_PATH environment ++ variable to point to the directory with the modified npy-pkg-config ++ files. ++ ++ Example npymath.ini modified for cross-compilation:: ++ ++ [meta] ++ Name=npymath ++ Description=Portable, core math library implementing C99 standard ++ Version=0.1 ++ ++ [variables] ++ pkgname=numpy.core ++ pkgdir=/build/arm-linux-gnueabi/sysroot/usr/lib/python3.7/site-packages/numpy/core ++ prefix=${pkgdir} ++ libdir=${prefix}/lib ++ includedir=${prefix}/include ++ ++ [default] ++ Libs=-L${libdir} -lnpymath ++ Cflags=-I${includedir} ++ Requires=mlib ++ ++ [msvc] ++ Libs=/LIBPATH:${libdir} npymath.lib ++ Cflags=/INCLUDE:${includedir} ++ Requires=mlib ++ + """ + if subst_dict is None: + subst_dict = {} +@@ -2092,9 +2129,22 @@ def get_numpy_include_dirs(): + return include_dirs + + def get_npy_pkg_dir(): +- """Return the path where to find the npy-pkg-config directory.""" ++ """Return the path where to find the npy-pkg-config directory. ++ ++ If the NPY_PKG_CONFIG_PATH environment variable is set, the value of that ++ is returned. Otherwise, a path inside the location of the numpy module is ++ returned. ++ ++ The NPY_PKG_CONFIG_PATH can be useful when cross-compiling, maintaining ++ customized npy-pkg-config .ini files for the cross-compilation ++ environment, and using them when cross-compiling. ++ ++ """ + # XXX: import here for bootstrapping reasons + import numpy ++ d = os.environ.get('NPY_PKG_CONFIG_PATH') ++ if d is not None: ++ return d + d = os.path.join(os.path.dirname(numpy.__file__), + 'core', 'lib', 'npy-pkg-config') + return d diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk index 3b474efa6e6c..1c7dd17adad0 100644 --- a/package/python-numpy/python-numpy.mk +++ b/package/python-numpy/python-numpy.mk @@ -30,6 +30,27 @@ define PYTHON_NUMPY_CONFIGURE_CMDS echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg endef +# The numpy distutils extensions are not very cross friendly. It comes with +# it's own pkg-config look-alike, which we are patching to allow overriding +# where it locates the configuration files. This allows us to use fixed up +# target configuration files, which we make sure includes full path to the +# target staging area, so that when building for target, we actually use the +# target libraries. Without this, target builds using numpy distutils +# extensions (such as fx. python-scipy) will use the host libraries, which +# obviously will not work. +ifeq ($(BR2_PACKAGE_PYTHON3),y) +PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/numpy +else +PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/numpy +endif +PYTHON_NUMPY_NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_STAGING_DIR)/core/lib/npy-pkg-config +define PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES + sed -e '/^pkgdir=/d' \ + -e '/^prefix=/i pkgdir=$(PYTHON_NUMPY_STAGING_DIR)/core' \ + -i $(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)/npymath.ini +endef +PYTHON_NUMPY_POST_INSTALL_STAGING_HOOKS += PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES + # Some package may include few headers from NumPy, so let's install it # in the staging area. PYTHON_NUMPY_INSTALL_STAGING = YES From patchwork Mon Sep 30 08:43:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Esben Haabendal X-Patchwork-Id: 1169182 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=geanix.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=geanix.com header.i=@geanix.com header.b="kfKvH3Cc"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46hbXb0rn5z9s7T for ; Mon, 30 Sep 2019 18:43:42 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 5FA0883F66; Mon, 30 Sep 2019 08:43:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pdv3iMMfRgWv; Mon, 30 Sep 2019 08:43:38 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id A4D718410C; Mon, 30 Sep 2019 08:43:37 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 1331C1BF228 for ; Mon, 30 Sep 2019 08:43:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0008A856E9 for ; Mon, 30 Sep 2019 08:43:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sdh1e05jbKTh for ; Mon, 30 Sep 2019 08:43:32 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from first.geanix.com (first.geanix.com [116.203.34.67]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 2C9C184480 for ; Mon, 30 Sep 2019 08:43:32 +0000 (UTC) Received: from localhost (unknown [193.163.1.7]) by first.geanix.com (Postfix) with ESMTPSA id 563D384B8F; Mon, 30 Sep 2019 08:42:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=geanix.com; s=first; t=1569832959; bh=30IPetNJSBSOpvWnKxnQPo460WTHzVoHYp3+8ZyO1MM=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=kfKvH3Cc8nsafVgpHbfcDG9gRWuESemcJPddN/7ZUtA6Ctu/EhE+py+yFDAtRcPiX 5Z3HMWGr6Xm+KRdd6x4xKmjRTLoRXJSF5rJNY8xU5evGq+X0L27MYFCW+syGFGb+UV lfiIbHq+0SpOSMO8qBsfnLrwoHLI0jaQqTOU8ind1u1Sdhv+QcMwC6SZ4V3SDKkbaT O2zxK4NJeTXeOtlucxqjnW6hlMGH+6yZ+nya5jd1fwwtHdnyVExi49OGY50xOqYUs5 ivdDi0lsDbDvw1NDG03E1dha75wsjUZcoiATHoW0IytTAKwurqFhC3fdBJYtr7+zyx ePPXSKhFhZGKg== From: Esben Haabendal To: buildroot@buildroot.org Date: Mon, 30 Sep 2019 10:43:26 +0200 Message-Id: <20190930084326.4414-2-esben@geanix.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20190930084326.4414-1-esben@geanix.com> References: <20190930084326.4414-1-esben@geanix.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 2/2] package/python-scipy: new package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alexandre PAYEN , Asaf Kahlon Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" See http://scipy.github.io/devdocs/building/linux.html for SciPy build requirements: * BLAS and LAPACK libraries * C and Fortran compilers * Python header files * Cython compiler For now, clapack package is used to provide both BLAS and LAPACK libraries, as that is what is supported by the current python-numpy recipe which python-scipy depends on. With additional work, it should be possible to improve both python-numpy and python-scipy to support other BLAS and LAPACK library implementations, but it is probably best to keep such changes aligned. A site.cfg file is used to configure numpy distutils extension to find include files and libraries in stagging. Note, host-python-numpy is needed in order to use numpy distutils extension from it. Signed-off-by: Esben Haabendal Tested-by: Jagan Teki --- DEVELOPERS | 1 + package/Config.in | 1 + package/python-scipy/Config.in | 23 +++++++++++++++ package/python-scipy/python-scipy.hash | 9 ++++++ package/python-scipy/python-scipy.mk | 40 ++++++++++++++++++++++++++ 5 files changed, 74 insertions(+) create mode 100644 package/python-scipy/Config.in create mode 100644 package/python-scipy/python-scipy.hash create mode 100644 package/python-scipy/python-scipy.mk diff --git a/DEVELOPERS b/DEVELOPERS index 8cbf55fd1013..499fa5954ffe 100644 --- a/DEVELOPERS +++ b/DEVELOPERS @@ -727,6 +727,7 @@ F: package/szip/ N: Esben Haabendal F: boot/gummiboot/ F: package/python-kiwisolver/ +F: package/python-scipy/ N: Etienne Carriere F: boot/optee-os/ diff --git a/package/Config.in b/package/Config.in index 713ef79edb9e..1a98ad9b2bcb 100644 --- a/package/Config.in +++ b/package/Config.in @@ -1059,6 +1059,7 @@ menu "External python modules" source "package/python-scapy/Config.in" source "package/python-scapy3k/Config.in" source "package/python-schedule/Config.in" + source "package/python-scipy/Config.in" source "package/python-sdnotify/Config.in" source "package/python-secretstorage/Config.in" source "package/python-see/Config.in" diff --git a/package/python-scipy/Config.in b/package/python-scipy/Config.in new file mode 100644 index 000000000000..071ceb2b70a6 --- /dev/null +++ b/package/python-scipy/Config.in @@ -0,0 +1,23 @@ +config BR2_PACKAGE_PYTHON_SCIPY + bool "python-scipy" + depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS + depends on !BR2_powerpc || BR2_TOOLCHAIN_USES_GLIBC # clapack + depends on !BR2_m68k_cf # clapack + depends on BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL # python-numpy + depends on BR2_TOOLCHAIN_HAS_FORTRAN + depends on BR2_PACKAGE_PYTHON3 + select BR2_PACKAGE_HOST_PYTHON_CYTHON + select BR2_PACKAGE_PYTHON_NUMPY + select BR2_PACKAGE_CLAPACK + help + The SciPy library is one of the core packages that make up the SciPy + stack. It provides many user-friendly and efficient numerical + routines such as routines for numerical integration, interpolation, + optimization, linear algebra and statistics. + + https://www.scipy.org/scipylib/ + +comment "python-scipy needs toolchain w/ fortran and glibc or musl" + depends on BR2_PACKAGE_PYTHON3 + depends on !BR2_TOOLCHAIN_HAS_FORTRAN || \ + (!BR2_TOOLCHAIN_USES_GLIBC && !BR2_TOOLCHAIN_USES_MUSL) diff --git a/package/python-scipy/python-scipy.hash b/package/python-scipy/python-scipy.hash new file mode 100644 index 000000000000..6b0c3605a3b8 --- /dev/null +++ b/package/python-scipy/python-scipy.hash @@ -0,0 +1,9 @@ +# Locally generated +sha256 4ea68de2840cf7d35c58464412d21f6f154cab7fba610c11002603ee0b9e2372 python-scipy-1.3.1.tar.gz +sha256 904942f7df1dc0f4a48af8d72405854fd96497bd1fe9ddb2c69d1797c22dfcd7 LICENSE.txt +sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35 scipy/linalg/src/lapack_deprecations/LICENSE +sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609 scipy/ndimage/LICENSE.txt +sha256 3df9207af2fdb861af0ae3b22026f163b9bcfab4e525dc4943afe2ffa3f77624 scipy/optimize/tnc/LICENSE +sha256 f0cedf52503b2d42b83411a0a16e6fefac346dfad8fddc66f50050150123470c scipy/sparse/linalg/dsolve/SuperLU/License.txt +sha256 0926566f9f406178d1214f8cc796e166b1213dd7c05e0c5b461a8b8ac9e50bbe scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING +sha256 51a5a08d537b34febb851c761496ead461f90b6c3c46a5248780d63870124e9c scipy/spatial/qhull_src/COPYING.txt diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/python-scipy.mk new file mode 100644 index 000000000000..bad392121da6 --- /dev/null +++ b/package/python-scipy/python-scipy.mk @@ -0,0 +1,40 @@ +################################################################################ +# +# python-scipy +# +################################################################################ + +PYTHON_SCIPY_VERSION = 1.3.1 +PYTHON_SCIPY_SITE = $(call github,scipy,scipy,v$(PYTHON_SCIPY_VERSION)) +PYTHON_SCIPY_LICENSE = BSD-3-Clause, BSD-2-Clause, BSD, BSD-Style, \ + Apache-2.0, MIT +PYTHON_SCIPY_LICENSE_FILES = \ + LICENSE.txt \ + scipy/linalg/src/lapack_deprecations/LICENSE \ + scipy/ndimage/LICENSE.txt \ + scipy/optimize/tnc/LICENSE \ + scipy/sparse/linalg/dsolve/SuperLU/License.txt \ + scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING \ + scipy/spatial/qhull_src/COPYING.txt +PYTHON_SCIPY_SETUP_TYPE = setuptools +PYTHON_SCIPY_DEPENDENCIES += host-python-cython host-python-numpy python-numpy \ + clapack + +PYTHON_SCIPY_BUILD_OPTS = config_fc --fcompiler=gnu95 +PYTHON_SCIPY_ENV += F90=$(TARGET_CROSS)gfortran F77=$(TARGET_CROSS)gfortran + +# Provide system configuration options to numpy distutils extenions, telling +# to find all include files and libraries in staging directory. +define PYTHON_SCIPY_CONFIGURE_CMDS + -rm -f $(@D)/site.cfg + echo "[DEFAULT]" >> $(@D)/site.cfg + echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg + echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg +endef + +# Use the target numpy pkg-config configuration files modified for target +# cross-compilation. Without this, numpy distutils will cause the linker to +# link with host libnpymath.a. +PYTHON_SCIPY_ENV += NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH) + +$(eval $(python-package))