From patchwork Wed Nov 14 10:55:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 997616 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 42w1dL5n7fz9s5c for ; Wed, 14 Nov 2018 21:56:22 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 527E5883F1; Wed, 14 Nov 2018 10:56:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AC+WEXZYytNF; Wed, 14 Nov 2018 10:56:20 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id A22DA87E14; Wed, 14 Nov 2018 10:56:20 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 7B3111C22DE for ; Wed, 14 Nov 2018 10:56:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 78DD487E3D for ; Wed, 14 Nov 2018 10:56:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7C8mdNLlE8vw for ; Wed, 14 Nov 2018 10:56:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.bootlin.com (mail.bootlin.com [62.4.15.54]) by hemlock.osuosl.org (Postfix) with ESMTP id 57B33883F1 for ; Wed, 14 Nov 2018 10:56:04 +0000 (UTC) Received: by mail.bootlin.com (Postfix, from userid 110) id 827D120731; Wed, 14 Nov 2018 11:56:03 +0100 (CET) Received: from localhost (aaubervilliers-681-1-30-49.w90-88.abo.wanadoo.fr [90.88.15.49]) by mail.bootlin.com (Postfix) with ESMTPSA id 4A362206A1; Wed, 14 Nov 2018 11:56:03 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , "Arnout Vandecappelle (Essensium/Mind)" , "Yann E. MORIN" , Peter Korsgaard , Andreas Naumann Date: Wed, 14 Nov 2018 11:55:51 +0100 Message-Id: <20181114105557.12599-1-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.19.1 MIME-Version: 1.0 Subject: [Buildroot] [PATCH next v4 0/6] Per-package host/target directory support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Hello, Here is a fourth iteration of the per-package SDK and target directory implementation. If you're interested in testing it, you can find the patch series at: http://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v4 Changes since v4 ================ - Dropped patches that have been merged upstream, mainly the "extract dependency" patches, but also a few other preparation patches. - Dropped the change of the RPATH handling. As discussed during previous Buildroot Developers meeting, it is not a big problem if during the build, the RPATH of a binary points to the library folder of another package per-package folder. Therefore, instead of fixing up RPATH at the end of each package installation, we keep what Buildroot does today: fix them at the very end of the build. - Made the support for per-package folders optional. Indeed, we realized there was no way to make sure it would be perfect from day 1. Even if the principle works, there are lots of package-specific details to solve, and solving all of them before merging the base per-package folder support is not reasonable. So for now, an option BR2_PER_PACKAGE_FOLDERS is introduced, which defaults to off. When this option is enabled, the .NOTPARALLEL statement of the main Makefile goes away, which allows to do top-level parallel build. - Addition of a commit that reworks how the top-level directories are created. - Introduction of a prepare-per-package-folder function in pkg-generic.mk to encapsulate the logic to create the per-package folders from the dependencies of a package. - Rebased on top of the latest master. Changes since v3 ================ - Dropped patches that have been merged upstream: pkgconf: use relative path to STAGING_DIR instead of absolute path toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN - For now, removed the most controversial/complicated patches (changing the rpath strategy, and switching to per-package host/target directories). The goal for now is to merge only the preparation patches. Changes since v2 ================ - Solved the problem of all DEPENDENCIES_HOST_PREREQ targets: host-ccache, host-tar, host-lzip, host-xz, host-fakedate. To achieve this, introduced _EXTRACT_DEPENDENCIES and used that to handle the dependencies on host-tar, host-lzip and host-xz. Used regular dependencies for host-ccache and host-fakedate. See below for more details. Changes since v1 ================ - Rebased on the latest Buildroot next branch - The pkg-config changes are reworked according to Arnout's comments: only @STAGING_SUBDIR@ is replaced in pkg-config.in to generate pkg-config, the rest of the logic is internal to the script. - New patch to move the "host skeleton" logic into a proper host-skeleton package. - New patch to have the CMake related files generated as part of the toolchain package. - New patch to add a new "install" step that depends on the different install steps (target, staging, images, host). This is needed to later add some logic that is executed once all install steps of a package have been done. - Fix the approach to solve the RPATH logic: instead of using LD_LIBRARY_PATH, we actually fix with patchelf the RPATH of host binaries right after their installation. - Misc other improvements in the per-package SDK implementation. What is this all about ? ======================== See the cover letter of v3 for details: http://lists.busybox.net/pipermail/buildroot/2017-December/208384.html What remains to be done? ======================== The main problem that remains is fixing packages that don't work well with per-package folders. I have already sent a few fixes (such as adding missing host-pkgconf dependencies, or fixing the freetype package), but I have setup a separate autobuilder instance to test this, and it uncovered a number of other problematic packages: - apr-util doesn't build, because some hardcoded paths from the "apr" per-package folders leak into the apr-util build. - Meson packages currently do not build, because pkg-meson.mk doesn't use $$(HOST_DIR) to reference cross-compilation.conf, but more importantly because cross-compilation.conf uses hardcoded paths. - The qt5 stack is entirely broken, because qmake hardcodes the installation path to be $(STAGING_DIR), so all qt5 packages install to the per-package $(STAGING_DIR) of qt5base. I started working on this topic, and will be sending a separate e-mail about this. Best regards, Thomas Thomas Petazzoni (6): Makefile: evaluate CCACHE and HOST{CC,CXX} at time of use support/scripts/check-host-rpath: split condition on two statements Makefile: rework main directory creation logic Makefile: move .NOTPARALLEL statement after including .config file Makefile: define TARGET_DIR_WARNING_FILE relative to TARGET_DIR core: implement per-package SDK and target Config.in | 18 ++++++ Makefile | 94 ++++++++++++++++++++------------ fs/common.mk | 3 +- package/pkg-generic.mk | 27 ++++++++- support/scripts/check-host-rpath | 8 ++- 5 files changed, 110 insertions(+), 40 deletions(-)