From patchwork Mon Feb 17 23:50:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1239695 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48M13C2lZ7z9sPK for ; Tue, 18 Feb 2020 10:51:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id E1A5F84EAD; Mon, 17 Feb 2020 23:51:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YqyQ-FglbyCu; Mon, 17 Feb 2020 23:51:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 4FA4384E10; Mon, 17 Feb 2020 23:51:15 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 3EFFC1BF360 for ; Mon, 17 Feb 2020 23:51:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3C2A484D11 for ; Mon, 17 Feb 2020 23:51:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IM5XgLCytXgd for ; Mon, 17 Feb 2020 23:51:00 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 386A984D0F for ; Mon, 17 Feb 2020 23:51:00 +0000 (UTC) X-Originating-IP: 173.221.118.3 Received: from localhost (173.221.118.3.nw.nuvox.net [173.221.118.3]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id EE0E61C0003; Mon, 17 Feb 2020 23:50:56 +0000 (UTC) From: Thomas Petazzoni To: buildroot@buildroot.org Date: Tue, 18 Feb 2020 00:50:47 +0100 Message-Id: <20200217235048.107343-1-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Subject: [Buildroot] [PATCH] package/pkg-python: fix for per-package directories X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Asaf Kahlon , Adam Duskett , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" With per-package directory support, Python external modules are causing a problem: the _sysconfigdata.py module installed by the Python interpreter contains a number of paths that are relative to the current package per-package directory, i.e python or python3. For example: 'BLDSHARED': '/home/thomas/projets/buildroot/output/per-package/python/host/bin/arm-linux-gcc -shared', 'CC': '/home/thomas/projets/buildroot/output/per-package/python/host/bin/arm-linux-gcc', 'CXX': '/home/thomas/projets/buildroot/output/per-package/python/host/bin/arm-linux-g++', etc. These paths are problematic, because it means that the wrong compiler gets used when building external Python modules: instead of using the compiler from the external Python module per-package host directory, it uses the one from the 'python' or 'python3' per-package host directory. Due to this, any native dependency needed by the external Python module is not found, even though it is properly present in the current package per-package directory. Of course, the problem occurs with both target Python modules and host Python modules. To fix this, we simply rewrite those paths in _sysconfigdata.py before building a Python package. Interestingly, until now, the _sysconfidata.py that was used during the build was the one from $(TARGET_DIR), which is a bit unusual: it is more common to use files from $(STAGING_DIR) during the build process. So this commit changes the PYTHON_PATH and PYTHON3_PATH variables so that they point to $(STAGING_DIR), which makes the _sysconfigdata.py fixup in $(STAGING_DIR) effective. Fixes: http://autobuild.buildroot.net/results/a24b0555fd4261b50dc3986635c30717d9cbe764/ (python-psycopg2) http://autobuild.buildroot.net/results/080fa893e1b0e7a8c8a31ac1c98eb8871b97264d/ (python-alsaaudio) http://autobuild.buildroot.net/results/79bc070f98d6d9d8ef78df12b248cdc7d0e405c3/ (python-lxml) and many more Python packages that use native code with a native library Signed-off-by: Thomas Petazzoni --- package/pkg-python.mk | 10 ++++++++++ package/python/python.mk | 2 +- package/python3/python3.mk | 2 +- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/package/pkg-python.mk b/package/pkg-python.mk index 4ded4fde83..4bf762e662 100644 --- a/package/pkg-python.mk +++ b/package/pkg-python.mk @@ -90,6 +90,14 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ --root=/ \ --single-version-externally-managed +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define PKG_PYTHON_FIXUP_SYSCONFIGDATA + find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ + -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" +endef +endif + ################################################################################ # inner-python-package -- defines how the configuration, compilation # and installation of a Python package should be done, implements a @@ -234,6 +242,8 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON) endif endif +$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA + # # Build step. Only define it if not already defined by the package .mk # file. diff --git a/package/python/python.mk b/package/python/python.mk index 41a981e3d3..1c393b255d 100644 --- a/package/python/python.mk +++ b/package/python/python.mk @@ -240,7 +240,7 @@ HOST_PYTHON_POST_INSTALL_HOOKS += HOST_PYTHON_INSTALL_PYTHON_SYMLINK endif # Provided to other packages -PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ +PYTHON_PATH = $(STAGING_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/ $(eval $(autotools-package)) $(eval $(host-autotools-package)) diff --git a/package/python3/python3.mk b/package/python3/python3.mk index 9432f1c59e..6a12c1f0e8 100644 --- a/package/python3/python3.mk +++ b/package/python3/python3.mk @@ -253,7 +253,7 @@ HOST_PYTHON3_POST_INSTALL_HOOKS += HOST_PYTHON3_INSTALL_SYMLINK endif # Provided to other packages -PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/ +PYTHON3_PATH = $(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/ # Support for socket.AF_BLUETOOTH ifeq ($(BR2_PACKAGE_BLUEZ5_UTILS_HEADERS),y)