From patchwork Fri Nov 3 16:06:24 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 833968 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yT6KC39h9z9ryQ for ; Sat, 4 Nov 2017 03:06:55 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 770DB2CE53; Fri, 3 Nov 2017 16:06:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hl7mjlaYeugI; Fri, 3 Nov 2017 16:06:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id C1AAD2FA77; Fri, 3 Nov 2017 16:06:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id C40551C0916 for ; Fri, 3 Nov 2017 16:06:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id BE3FA2FA77 for ; Fri, 3 Nov 2017 16:06:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CsUJ4aXhoza8 for ; Fri, 3 Nov 2017 16:06:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by silver.osuosl.org (Postfix) with ESMTP id 5DEFD2CE53 for ; Fri, 3 Nov 2017 16:06:48 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id E5DE221D8E; Fri, 3 Nov 2017 17:06:46 +0100 (CET) Received: from localhost (132.230.147.77.rev.sfr.net [77.147.230.132]) by mail.free-electrons.com (Postfix) with ESMTPSA id 992E2212A2; Fri, 3 Nov 2017 17:06:36 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Peter Korsgaard Date: Fri, 3 Nov 2017 17:06:24 +0100 Message-Id: <20171103160627.6468-2-thomas.petazzoni@free-electrons.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> References: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> Subject: [Buildroot] [RFCv1 1/4] pkgconf: use relative path to STAGING_DIR instead of absolute path X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The pkg-config wrapper script is currently generated with absolute paths to $(STAGING_DIR). However, this will not work properly with per-package SDK, and each package will be built with a different STAGING_DIR value. In order to fix this, we adjust how the pkg-config wrapper script is generated, so that it uses a relative path to itself: the sysroot (i.e STAGING_DIR) is always located in $(path of pkg-config)/../$(STAGING_SUBDIR). This change is independent from the per-package SDK work, and could be applied independently from it. Signed-off-by: Thomas Petazzoni Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- package/pkgconf/pkgconf.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk index cc190d26da..d11ce485f7 100644 --- a/package/pkgconf/pkgconf.mk +++ b/package/pkgconf/pkgconf.mk @@ -19,8 +19,8 @@ endef define HOST_PKGCONF_INSTALL_WRAPPER $(INSTALL) -m 0755 -D package/pkgconf/pkg-config.in \ $(HOST_DIR)/bin/pkg-config - $(SED) 's,@PKG_CONFIG_LIBDIR@,$(STAGING_DIR)/usr/lib/pkgconfig:$(STAGING_DIR)/usr/share/pkgconfig,' \ - -e 's,@STAGING_DIR@,$(STAGING_DIR),' \ + $(SED) 's,@PKG_CONFIG_LIBDIR@,$$(dirname $$0)/../$(STAGING_SUBDIR)/usr/lib/pkgconfig:$$(dirname $$0)/../$(STAGING_SUBDIR)/usr/share/pkgconfig,' \ + -e 's,@STAGING_DIR@,$$(dirname $$0)/../$(STAGING_SUBDIR),' \ $(HOST_DIR)/bin/pkg-config endef From patchwork Fri Nov 3 16:06:25 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 833971 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yT6KP2v28z9ryQ for ; Sat, 4 Nov 2017 03:07:05 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id CEA552FD2D; Fri, 3 Nov 2017 16:07:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SeG59hhQQz6v; Fri, 3 Nov 2017 16:06:58 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 998942FD76; Fri, 3 Nov 2017 16:06:58 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 4C56E1C0916 for ; Fri, 3 Nov 2017 16:06:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4775A87F12 for ; Fri, 3 Nov 2017 16:06:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T99HhBc8kmLq for ; Fri, 3 Nov 2017 16:06:49 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by hemlock.osuosl.org (Postfix) with ESMTP id 8FDC687EBF for ; Fri, 3 Nov 2017 16:06:49 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 1170D212A2; Fri, 3 Nov 2017 17:06:48 +0100 (CET) Received: from localhost (132.230.147.77.rev.sfr.net [77.147.230.132]) by mail.free-electrons.com (Postfix) with ESMTPSA id 7770F21D47; Fri, 3 Nov 2017 17:06:37 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Peter Korsgaard Date: Fri, 3 Nov 2017 17:06:25 +0100 Message-Id: <20171103160627.6468-3-thomas.petazzoni@free-electrons.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> References: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> Subject: [Buildroot] [RFCv1 2/4] core: change host RPATH handling X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Currently, our strategy for the RPATH of host binaries is as follows: - During the build, we force an absolute RPATH to be encoded into binaries, by passing -Wl,-rpath,$(HOST_DIR)/lib. This ensures that host binaries will find their libraries. - In the "make sdk" target, when preparing the SDK to be relocatable, we use patchelf to replace those absolute RPATHs by relative RPATHs that use $ORIGIN. Unfortunately, the use of absolute RPATH during the build is not compatible with the move to per-package SDK, where a different host directory is used for the build of each package. Therefore, we need to move to a different strategy for RPATH handling. The new strategy is as follows: - During the build, we do not encode any RPATH into the host binaries. Instead, we specify LD_LIBRARY_PATH environment to point to the current HOST_DIR/lib directory (for the package being currently built). - At the end of the build, in order to make sure that binaries are usable without LD_LIBRARY_PATH, we fixup the host binaries to use $ORIGIN/../lib. This is more-or-less what was done previously in the "make sdk" target, except that instead of turning absolute paths into relative paths in the RPATH, we are simply setting the RPATH to $ORIGIN/../lib. In order to implement this, the fix-rpath script logic is adjusted for the "host" case. An alternative strategy would have been to keep the -Wl,-rpath,$(HOST_DIR) flag, and therefore the absolute RPATH in the host binaries, and fix such RPATH at the end of the build of every package. However, that would require calling fix-rpath after the installation of every package, which is a bit expensive. This change is independent from the per-package SDK functionality, and could be applied separately. Signed-off-by: Thomas Petazzoni --- Makefile | 4 +++- package/Makefile.in | 2 +- package/pkg-generic.mk | 8 -------- support/scripts/fix-rpath | 15 ++++++++++----- 4 files changed, 14 insertions(+), 15 deletions(-) diff --git a/Makefile b/Makefile index 79db7fe48a..5496273329 100644 --- a/Makefile +++ b/Makefile @@ -231,6 +231,8 @@ LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings LEGAL_REPORT = $(LEGAL_INFO_DIR)/README +export LD_LIBRARY_PATH = $(if $(PKG),$(HOST_DIR)/lib) + ################################################################################ # # staging and target directories do NOT list these as @@ -558,7 +560,6 @@ world: target-post-image .PHONY: sdk sdk: world @$(call MESSAGE,"Rendering the SDK relocatable") - $(TOPDIR)/support/scripts/fix-rpath host $(TOPDIR)/support/scripts/fix-rpath staging $(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location @@ -679,6 +680,7 @@ $(TARGETS_ROOTFS): target-finalize .PHONY: target-finalize target-finalize: $(PACKAGES) @$(call MESSAGE,"Finalizing target directory") + $(TOPDIR)/support/scripts/fix-rpath host $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \ diff --git a/package/Makefile.in b/package/Makefile.in index a1a5316051..e94a75c230 100644 --- a/package/Makefile.in +++ b/package/Makefile.in @@ -220,7 +220,7 @@ HOST_CPPFLAGS = -I$(HOST_DIR)/include HOST_CFLAGS ?= -O2 HOST_CFLAGS += $(HOST_CPPFLAGS) HOST_CXXFLAGS += $(HOST_CFLAGS) -HOST_LDFLAGS += -L$(HOST_DIR)/lib -Wl,-rpath,$(HOST_DIR)/lib +HOST_LDFLAGS += -L$(HOST_DIR)/lib # The macros below are taken from linux 4.11 and adapted slightly. # Copy more when needed. diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index cca94ba338..82f8c06821 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -105,14 +105,6 @@ endef GLOBAL_INSTRUMENTATION_HOOKS += check_bin_arch -# This hook checks that host packages that need libraries that we build -# have a proper DT_RPATH or DT_RUNPATH tag -define check_host_rpath - $(if $(filter install-host,$(2)),\ - $(if $(filter end,$(1)),support/scripts/check-host-rpath $(3) $(HOST_DIR))) -endef -GLOBAL_INSTRUMENTATION_HOOKS += check_host_rpath - define step_check_build_dir_one if [ -d $(2) ]; then \ printf "%s: installs files in %s\n" $(1) $(2) >&2; \ diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath index 15705a3b0d..eed751f5da 100755 --- a/support/scripts/fix-rpath +++ b/support/scripts/fix-rpath @@ -61,7 +61,7 @@ main() { local rootdir local tree="${1}" local find_args=( ) - local sanitize_extra_args=( ) + local sanitize_args=( ) if ! "${PATCHELF}" --version > /dev/null 2>&1; then echo "Error: can't execute patchelf utility '${PATCHELF}'" @@ -89,7 +89,8 @@ main() { cp "${PATCHELF}" "${PATCHELF}.__to_be_patched" # we always want $ORIGIN-based rpaths to make it relocatable. - sanitize_extra_args+=( "--relative-to-file" ) + sanitize_args+=( "--set-rpath" ) + sanitize_args+=( "\$ORIGIN/../lib" ) ;; staging) @@ -101,14 +102,18 @@ main() { done # should be like for the target tree below - sanitize_extra_args+=( "--no-standard-lib-dirs" ) + sanitize_args+=( "--no-standard-lib-dirs" ) + sanitize_args+=( "--make-rpath-relative" ) + sanitize_args+=( "${rootdir}" ) ;; target) rootdir="${TARGET_DIR}" # we don't want $ORIGIN-based rpaths but absolute paths without rootdir. # we also want to remove rpaths pointing to /lib or /usr/lib. - sanitize_extra_args+=( "--no-standard-lib-dirs" ) + sanitize_args+=( "--no-standard-lib-dirs" ) + sanitize_args+=( "--make-rpath-relative" ) + sanitize_args+=( "${rootdir}" ) ;; *) @@ -125,7 +130,7 @@ main() { # make files writable if necessary changed=$(chmod -c u+w "${file}") # call patchelf to sanitize the rpath - ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}" + ${PATCHELF} ${sanitize_args[@]} "${file}" # restore the original permission test "${changed}" != "" && chmod u-w "${file}" fi From patchwork Fri Nov 3 16:06:26 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 833969 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yT6KJ3TSJz9ryQ for ; Sat, 4 Nov 2017 03:07:00 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id EBDAF2FCC3; Fri, 3 Nov 2017 16:06:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 27NIXGocGjDc; Fri, 3 Nov 2017 16:06:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id B798D2FA77; Fri, 3 Nov 2017 16:06:55 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id DD5C51C0916 for ; Fri, 3 Nov 2017 16:06:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id D6CD02FA77 for ; Fri, 3 Nov 2017 16:06:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SDl1G2HJR6dI for ; Fri, 3 Nov 2017 16:06:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by silver.osuosl.org (Postfix) with ESMTP id 530CD2CE53 for ; Fri, 3 Nov 2017 16:06:50 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 1298721D47; Fri, 3 Nov 2017 17:06:49 +0100 (CET) Received: from localhost (132.230.147.77.rev.sfr.net [77.147.230.132]) by mail.free-electrons.com (Postfix) with ESMTPSA id 9FFFA21DAE; Fri, 3 Nov 2017 17:06:38 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Peter Korsgaard Date: Fri, 3 Nov 2017 17:06:26 +0100 Message-Id: <20171103160627.6468-4-thomas.petazzoni@free-electrons.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> References: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> Subject: [Buildroot] [RFCv1 3/4] toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The upcoming per-package SDK functionality is heavily based on the fact that HOST_DIR, STAGING_DIR and TARGET_DIR are evaluated during the configure/build/install steps of the packages. Therefore, any evaluation-during-assignment using := is going to cause problems, and need to be turned into evaluation-during-use using =. This patch fix up one such instance in the external toolchain code. This change is independent from the per-package SDK functionality, and could be applied separately. Signed-off-by: Thomas Petazzoni Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- toolchain/toolchain-external/pkg-toolchain-external.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/toolchain/toolchain-external/pkg-toolchain-external.mk b/toolchain/toolchain-external/pkg-toolchain-external.mk index dc0588c536..b9ad1720a1 100644 --- a/toolchain/toolchain-external/pkg-toolchain-external.mk +++ b/toolchain/toolchain-external/pkg-toolchain-external.mk @@ -77,7 +77,7 @@ ifneq ($(TOOLCHAIN_EXTERNAL_PREFIX),) TOOLCHAIN_EXTERNAL_BIN := $(dir $(shell which $(TOOLCHAIN_EXTERNAL_PREFIX)-gcc)) endif else -TOOLCHAIN_EXTERNAL_BIN := $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/bin +TOOLCHAIN_EXTERNAL_BIN = $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/bin endif # If this is a buildroot toolchain, it already has a wrapper which we want to From patchwork Fri Nov 3 16:06:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 833972 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yT6KV0Lrpz9ryQ for ; Sat, 4 Nov 2017 03:07:09 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4B16487FEF; Fri, 3 Nov 2017 16:07:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fdsXDUTvPN3a; Fri, 3 Nov 2017 16:07:06 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 49D3E87F74; Fri, 3 Nov 2017 16:07:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 419DB1C0916 for ; Fri, 3 Nov 2017 16:06:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3A4B586E81 for ; Fri, 3 Nov 2017 16:06:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xXS3NZsDIbh6 for ; Fri, 3 Nov 2017 16:06:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.free-electrons.com (mail.free-electrons.com [62.4.15.54]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7BE2486E71 for ; Fri, 3 Nov 2017 16:06:51 +0000 (UTC) Received: by mail.free-electrons.com (Postfix, from userid 110) id 40AAB21D9D; Fri, 3 Nov 2017 17:06:50 +0100 (CET) Received: from localhost (132.230.147.77.rev.sfr.net [77.147.230.132]) by mail.free-electrons.com (Postfix) with ESMTPSA id 8789021DAF; Fri, 3 Nov 2017 17:06:39 +0100 (CET) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Peter Korsgaard Date: Fri, 3 Nov 2017 17:06:27 +0100 Message-Id: <20171103160627.6468-5-thomas.petazzoni@free-electrons.com> X-Mailer: git-send-email 2.13.6 In-Reply-To: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> References: <20171103160627.6468-1-thomas.petazzoni@free-electrons.com> Subject: [Buildroot] [RFCv1 4/4] core: implement per-package SDK and target X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit implemnts the core of the move to per-package SDK and target directories. The main idea is that instead of having a global output/host and output/target in which all packages install files, we switch to per-package host and target folders, that only contain their explicit dependencies. There are two main benefits: - Packages will no longer discover dependencies that they do not explicitly indicate in their _DEPENDENCIES variable. - We can support top-level parallel build properly, because a package only "sees" its own host directory and target directory, isolated from the build of other packages that can happen in parallel. It works as follows: - A new output/per-package/ folder is created, which will contain one sub-folder per package, and inside it, a "host" folder and a "target" folder: output/per-package/busybox/target output/per-package/busybox/host output/per-package/host-fakeroot/target output/per-package/host-fakeroot/host This output/per-package/ folder is PER_PACKAGE_DIR, and each package defines _TARGET_DIR, _HOST_DIR and _STAGING_DIR pointing to its own target, host and staging directories. - The _TARGET_DIR, _HOST_DIR and _STAGING_DIR variable are passed as TARGET_DIR, HOST_DIR and STAGING_DIR to the various make targets used for the different build steps of the packages. Effectively this means that when a package references $(HOST_DIR), the value is in fact $(_HOST_DIR). - Of course, packages have dependencies, so those dependencies must be installed in the per-package host and target folders before configuring the package. To do so, we simply rsync (using hard links to save space and time) the host and target folders of the direct dependencies of the package to the current package host and target folders. We only need to take care of direct dependencies (and not recursively all dependencies), because we accumulate into those per-package host and target folders the files installed by the dependencies. - We fix LD_LIBRARY_PATH to make it point to the current package host directory. This is basically enough to make per-package SDK and target work. The only gotcha is that at the end of the build, output/target and output/host are empty, which means that: - The filesystem image creation code cannot work. - We don't have a SDK to build code outside of Buildroot. In order to fix this, this commit extends the target-finalize step so that it starts by populating output/target and output/host by rsync-ing into them the target and host directories of all packages listed in the $(PACKAGES) variable. [Note: in fact, output/host is not completely empty, even though they should be. It contains the following files: output/host/ output/host/lib64 output/host/usr output/host/share output/host/share/buildroot output/host/share/buildroot/Platform output/host/share/buildroot/Platform/Buildroot.cmake output/host/share/buildroot/toolchainfile.cmake output/host/lib Perhaps those files should be installed by some package instead, perhaps the 'toolchain' package.] Signed-off-by: Thomas Petazzoni --- Makefile | 17 ++++++++++++++--- package/pkg-generic.mk | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 5496273329..4e1ba62569 100644 --- a/Makefile +++ b/Makefile @@ -216,6 +216,8 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images TARGET_DIR := $(BASE_DIR)/target +PER_PACKAGE_DIR := $(BASE_DIR)/per-package + # initial definition so that 'make clean' works for most users, even without # .config. HOST_DIR will be overwritten later when .config is included. HOST_DIR := $(BASE_DIR)/host @@ -231,7 +233,7 @@ LEGAL_MANIFEST_CSV_HOST = $(LEGAL_INFO_DIR)/host-manifest.csv LEGAL_WARNINGS = $(LEGAL_INFO_DIR)/.warnings LEGAL_REPORT = $(LEGAL_INFO_DIR)/README -export LD_LIBRARY_PATH = $(if $(PKG),$(HOST_DIR)/lib) +export LD_LIBRARY_PATH = $(if $(PKG),$($(PKG)_HOST_DIR)/lib) ################################################################################ # @@ -239,7 +241,7 @@ export LD_LIBRARY_PATH = $(if $(PKG),$(HOST_DIR)/lib) # dependencies anywhere else # ################################################################################ -$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST): +$(BUILD_DIR) $(TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST) $(PER_PACKAGE_DIR): @mkdir -p $@ BR2_CONFIG = $(CONFIG_DIR)/.config @@ -675,10 +677,19 @@ endef TARGET_FINALIZE_HOOKS += PURGE_LOCALES endif +ALL_PACKAGES_TARGET_DIRS = $(foreach pkg,$(PACKAGES),$($(call UPPERCASE,$(pkg))_TARGET_DIR)) +ALL_PACKAGES_HOST_DIRS = $(foreach pkg,$(PACKAGES),$($(call UPPERCASE,$(pkg))_HOST_DIR)) + $(TARGETS_ROOTFS): target-finalize .PHONY: target-finalize target-finalize: $(PACKAGES) + @$(call MESSAGE,"Creating global target directory") + $(foreach dir,$(ALL_PACKAGES_TARGET_DIRS),\ + rsync -au --link-dest=$(dir) $(dir) $(TARGET_DIR)$(sep)) + @$(call MESSAGE,"Creating global host directory") + $(foreach dir,$(ALL_PACKAGES_HOST_DIRS),\ + rsync -au --link-dest=$(dir) $(dir) $(HOST_DIR)$(sep)) @$(call MESSAGE,"Finalizing target directory") $(TOPDIR)/support/scripts/fix-rpath host $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) @@ -972,7 +983,7 @@ printvars: clean: rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \ $(BUILD_DIR) $(BASE_DIR)/staging \ - $(LEGAL_INFO_DIR) $(GRAPHS_DIR) + $(LEGAL_INFO_DIR) $(GRAPHS_DIR) $(PER_PACKAGE_DIR) .PHONY: distclean distclean: clean diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 82f8c06821..aee4011f63 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -158,6 +158,7 @@ $(BUILD_DIR)/%/.stamp_actual_downloaded: $(BUILD_DIR)/%/.stamp_extracted: @$(call step_start,extract) @$(call MESSAGE,"Extracting") + @mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(foreach hook,$($(PKG)_PRE_EXTRACT_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D) $($(PKG)_EXTRACT_CMDS) @@ -215,6 +216,10 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\ $(BUILD_DIR)/%/.stamp_configured: @$(call step_start,configure) @$(call MESSAGE,"Configuring") + $(foreach dir,$($(PKG)_FINAL_DEPENDENCIES_HOST_DIRS),\ + rsync -au --link-dest=$(dir) $(dir) $(HOST_DIR)$(sep)) + $(foreach dir,$($(PKG)_FINAL_DEPENDENCIES_TARGET_DIRS),\ + rsync -au --link-dest=$(dir) $(dir) $(TARGET_DIR)$(sep)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) @@ -409,6 +414,10 @@ $(2)_NAME = $(1) $(2)_RAWNAME = $$(patsubst host-%,%,$(1)) $(2)_PKGDIR = $(pkgdir) +$(2)_TARGET_DIR = $$(PER_PACKAGE_DIR)/$(1)/target/ +$(2)_HOST_DIR = $$(PER_PACKAGE_DIR)/$(1)/host/ +$(2)_STAGING_DIR = $$(PER_PACKAGE_DIR)/$(1)/host/$(STAGING_SUBDIR) + # Keep the package version that may contain forward slashes in the _DL_VERSION # variable, then replace all forward slashes ('/') by underscores ('_') to # sanitize the package version that is used in paths, directory and file names. @@ -561,6 +570,13 @@ $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES)) $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES)) $(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES)) +$(2)_FINAL_DEPENDENCIES_HOST_DIRS = \ + $$(foreach dep,$$($(2)_FINAL_DEPENDENCIES),\ + $$($$(call UPPERCASE,$$(dep))_HOST_DIR)) +$(2)_FINAL_DEPENDENCIES_TARGET_DIRS = \ + $$(foreach dep,$$($(2)_FINAL_DEPENDENCIES),\ + $$($$(call UPPERCASE,$$(dep))_TARGET_DIR)) + $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_IMAGES ?= NO $(2)_INSTALL_TARGET ?= YES @@ -789,17 +805,38 @@ $(1)-reconfigure: $(1)-clean-for-reconfigure $(1) # define the PKG variable for all targets, containing the # uppercase package variable prefix $$($(2)_TARGET_INSTALL_TARGET): PKG=$(2) +$$($(2)_TARGET_INSTALL_TARGET): HOST_DIR=$$($(2)_HOST_DIR) +$$($(2)_TARGET_INSTALL_TARGET): STAGING_DIR=$$($(2)_STAGING_DIR) +$$($(2)_TARGET_INSTALL_TARGET): TARGET_DIR=$$($(2)_TARGET_DIR) $$($(2)_TARGET_INSTALL_STAGING): PKG=$(2) +$$($(2)_TARGET_INSTALL_STAGING): HOST_DIR=$$($(2)_HOST_DIR) +$$($(2)_TARGET_INSTALL_STAGING): STAGING_DIR=$$($(2)_STAGING_DIR) +$$($(2)_TARGET_INSTALL_STAGING): TARGET_DIR=$$($(2)_TARGET_DIR) $$($(2)_TARGET_INSTALL_IMAGES): PKG=$(2) +$$($(2)_TARGET_INSTALL_IMAGES): HOST_DIR=$$($(2)_HOST_DIR) +$$($(2)_TARGET_INSTALL_IMAGES): STAGING_DIR=$$($(2)_STAGING_DIR) +$$($(2)_TARGET_INSTALL_IMAGES): TARGET_DIR=$$($(2)_TARGET_DIR) $$($(2)_TARGET_INSTALL_HOST): PKG=$(2) +$$($(2)_TARGET_INSTALL_HOST): HOST_DIR=$$($(2)_HOST_DIR) +$$($(2)_TARGET_INSTALL_HOST): STAGING_DIR=$$($(2)_STAGING_DIR) +$$($(2)_TARGET_INSTALL_HOST): TARGET_DIR=$$($(2)_TARGET_DIR) $$($(2)_TARGET_BUILD): PKG=$(2) +$$($(2)_TARGET_BUILD): HOST_DIR=$$($(2)_HOST_DIR) +$$($(2)_TARGET_BUILD): STAGING_DIR=$$($(2)_STAGING_DIR) +$$($(2)_TARGET_BUILD): TARGET_DIR=$$($(2)_TARGET_DIR) $$($(2)_TARGET_CONFIGURE): PKG=$(2) +$$($(2)_TARGET_CONFIGURE): HOST_DIR=$$($(2)_HOST_DIR) +$$($(2)_TARGET_CONFIGURE): STAGING_DIR=$$($(2)_STAGING_DIR) +$$($(2)_TARGET_CONFIGURE): TARGET_DIR=$$($(2)_TARGET_DIR) $$($(2)_TARGET_RSYNC): SRCDIR=$$($(2)_OVERRIDE_SRCDIR) $$($(2)_TARGET_RSYNC): PKG=$(2) $$($(2)_TARGET_PATCH): PKG=$(2) $$($(2)_TARGET_PATCH): RAWNAME=$$(patsubst host-%,%,$(1)) $$($(2)_TARGET_PATCH): PKGDIR=$(pkgdir) $$($(2)_TARGET_EXTRACT): PKG=$(2) +$$($(2)_TARGET_EXTRACT): HOST_DIR=$$($(2)_HOST_DIR) +$$($(2)_TARGET_EXTRACT): STAGING_DIR=$$($(2)_STAGING_DIR) +$$($(2)_TARGET_EXTRACT): TARGET_DIR=$$($(2)_TARGET_DIR) $$($(2)_TARGET_SOURCE): PKG=$(2) $$($(2)_TARGET_SOURCE): PKGDIR=$(pkgdir) $$($(2)_TARGET_ACTUAL_SOURCE): PKG=$(2)