From patchwork Mon Jun 21 14:11:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495139 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s190DKyz9sf9 for ; Tue, 22 Jun 2021 00:12:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B0773607AA; Mon, 21 Jun 2021 14:12:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TkLWCiVB_mPJ; Mon, 21 Jun 2021 14:12:24 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 001E260017; Mon, 21 Jun 2021 14:12:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 085071BF355 for ; Mon, 21 Jun 2021 14:12:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 0532682F0F for ; Mon, 21 Jun 2021 14:12:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YcExTVXN1oZQ for ; Mon, 21 Jun 2021 14:12:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1316D82BBA for ; Mon, 21 Jun 2021 14:12:20 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 94016200004; Mon, 21 Jun 2021 14:12:18 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:16 +0200 Message-Id: <20210621141130.48654-2-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 01/15] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas Petazzoni Without per-package directory support, a package can happily overwrite files installed by other packages. Indeed, because the build order between packages is always guaranteed, Buildroot will always produce the same output. However, with per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package, due to how the final aggregation is done to create the complete target/, staging/ and host/ folders. Unfortunately, we currently don't have anything in Buildroot that detects this situation. We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57. This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps. Since preventing file overwrites is really only needed when per-package directory support is used, and due to this verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is however not too bad as on average, with per-package directory support, the per-package target/ and host/ directories will contain less files than with a build that doesn't use per-package directory support. This helps a bit in mitigating the additional cost of this verification. Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR. Signed-off-by: Thomas Petazzoni This commit is retreived from Thomas's work. The first version was discussed https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni@bootlin.com/ This new version was not already submitted by Thomas or I missed it. Signed-off-by: Herve Codina --- package/pkg-generic.mk | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 45589bcbb4..bb9ff4150a 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -102,6 +102,25 @@ define fixup-libtool-files endef endif +# Functions to detect overwritten files + +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_before + LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5; +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_after + if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \ + LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \ + { echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \ + fi +endef +endif + # Functions to collect statistics about installed files # $(1): base directory to search in @@ -235,6 +254,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) @@ -360,6 +381,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ # Remove package sources From patchwork Mon Jun 21 14:11:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495140 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s1N3l7Yz9sXG for ; Tue, 22 Jun 2021 00:12:40 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 22EEA827D9; Mon, 21 Jun 2021 14:12:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pHxffy_l2OyI; Mon, 21 Jun 2021 14:12:37 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp1.osuosl.org (Postfix) with ESMTP id 952E582656; Mon, 21 Jun 2021 14:12:36 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 6E8F61BF355 for ; Mon, 21 Jun 2021 14:12:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 5E28782F0F for ; Mon, 21 Jun 2021 14:12:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wGB5FU2P7e43 for ; Mon, 21 Jun 2021 14:12:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp1.osuosl.org (Postfix) with ESMTPS id 2936582BBA for ; Mon, 21 Jun 2021 14:12:23 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 035FF200008; Mon, 21 Jun 2021 14:12:20 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:17 +0200 Message-Id: <20210621141130.48654-3-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 02/15] package/e2fsprogs: fix fsck overwrite in HOST_DIR X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" host-e2fsprog overwrites fsck program and some man related files previously installed by host-util-linux. This patch simply disable fsck in host-e2fsprog. host-e2fsprog is used to build final ext{2,3,4} images. The missing host-e2fsprog fsck tool (filesystem integrity check tool) in HOST_DIR should not lead to issues. Signed-off-by: Herve Codina Reviewed-by: Thomas Petazzoni --- package/e2fsprogs/e2fsprogs.mk | 1 + 1 file changed, 1 insertion(+) diff --git a/package/e2fsprogs/e2fsprogs.mk b/package/e2fsprogs/e2fsprogs.mk index 0a69690e2f..bd147e737e 100644 --- a/package/e2fsprogs/e2fsprogs.mk +++ b/package/e2fsprogs/e2fsprogs.mk @@ -27,6 +27,7 @@ HOST_E2FSPROGS_CONF_OPTS = \ --disable-defrag \ --disable-e2initrd-helper \ --disable-fuse2fs \ + --disable-fsck \ --disable-libblkid \ --disable-libuuid \ --disable-testio-debug \ From patchwork Mon Jun 21 14:11:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495141 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s1c3wj7z9sXk for ; Tue, 22 Jun 2021 00:12:52 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8A9776070C; Mon, 21 Jun 2021 14:12:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ftt6e4piEwsz; Mon, 21 Jun 2021 14:12:49 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 003F960631; Mon, 21 Jun 2021 14:12:48 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id D84121BF355 for ; Mon, 21 Jun 2021 14:12:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id C7D6360631 for ; Mon, 21 Jun 2021 14:12:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vOxMtzMVwIpc for ; Mon, 21 Jun 2021 14:12:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp3.osuosl.org (Postfix) with ESMTPS id F29C760017 for ; Mon, 21 Jun 2021 14:12:37 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 0F379200003; Mon, 21 Jun 2021 14:12:33 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:18 +0200 Message-Id: <20210621141130.48654-4-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 03/15] package/pkg-generic.mk: Remove Info documents dir entry X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Some packages (autotools for instance) install documentation files using install-info. This program adds an entry in the Info directory file (share/info/dir) and this causes TARGET_DIR and/or HOST_DIR overwrite. In order to avoid this overwrite this patch removes the Info directory file right after any installation. Signed-off-by: Herve Codina --- package/pkg-generic.mk | 3 +++ 1 file changed, 3 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index bb9ff4150a..2499c94746 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -280,6 +280,7 @@ $(BUILD_DIR)/%/.stamp_host_installed: $(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(HOST_DIR)/share/info/dir @$(call step_end,install-host) $(Q)touch $@ @@ -309,6 +310,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed: $(foreach hook,$($(PKG)_PRE_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_STAGING_CMDS) $(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(STAGING_DIR)/share/info/dir $(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \ $(call MESSAGE,"Fixing package configuration files") ;\ $(SED) "s,$(HOST_DIR),@HOST_DIR@,g" \ @@ -368,6 +370,7 @@ $(BUILD_DIR)/%/.stamp_target_installed: $(or $($(PKG)_INSTALL_INIT_OPENRC), \ $($(PKG)_INSTALL_INIT_SYSV))) $(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep)) + $(Q) rm -f $(TARGET_DIR)/share/info/dir $(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \ $(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \ fi From patchwork Mon Jun 21 14:11:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495142 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s1w5ZDdz9sWQ for ; Tue, 22 Jun 2021 00:13:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0A3F360017; Mon, 21 Jun 2021 14:13:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZKnX2kFhbsLr; Mon, 21 Jun 2021 14:13:02 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 4DD91607C2; Mon, 21 Jun 2021 14:13:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 437841BF355 for ; Mon, 21 Jun 2021 14:12:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 331EC8274E for ; Mon, 21 Jun 2021 14:12:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CPqyA8AJFp1v for ; Mon, 21 Jun 2021 14:12:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp1.osuosl.org (Postfix) with ESMTPS id E7A7C82656 for ; Mon, 21 Jun 2021 14:12:38 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id C5E5E20000A; Mon, 21 Jun 2021 14:12:36 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:19 +0200 Message-Id: <20210621141130.48654-5-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 04/15] package/pkg-generic.mk: Fix .la files overwrite detection X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" During per-package build, original .la files are modified by fixup-libtool-files calls. But since fixup-libtool-files modifies files using sed --in-place, these modification are done using a temporary file and a call to rename. Rename breaks the hardlink to the original file and leave the temporary file in per-package TARGET dir. As the original file is not modified, this is no longer considered as an overwrite. To fix this detection, this patch simply considers the what is done by fixup-libtool-files is part of the original snapshot used to detect overwrites. And so, the original snapshot is taken after fixup-libtool-files call. Signed-off-by: Herve Codina Reviewed-by: Thomas Petazzoni --- package/pkg-generic.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 2499c94746..f9564831cc 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -254,9 +254,9 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) - $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) From patchwork Mon Jun 21 14:11:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495143 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s282dB1z9sWl for ; Tue, 22 Jun 2021 00:13:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B8CAF606DA; Mon, 21 Jun 2021 14:13:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V8hfMQMgUCvh; Mon, 21 Jun 2021 14:13:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id DD5C86088E; Mon, 21 Jun 2021 14:13:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 5EB5F1BF355 for ; Mon, 21 Jun 2021 14:12:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 5A30940360 for ; Mon, 21 Jun 2021 14:12:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 518QeP06vx2T for ; Mon, 21 Jun 2021 14:12:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp4.osuosl.org (Postfix) with ESMTPS id E73D540348 for ; Mon, 21 Jun 2021 14:12:40 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 79D50200009; Mon, 21 Jun 2021 14:12:38 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:20 +0200 Message-Id: <20210621141130.48654-6-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 05/15] package/pkg-generic.mk: Perform .la files fixup in per-package HOST_DIR X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" fixup-libtool-files was called on per-package STAGING_DIR. Some host-xxxx packages have their .la files with directories pointing outside their own per-package directory. This commit, calling fixup-libtool-files on HOST_DIR, fixes this issue. Signed-off-by: Herve Codina --- package/pkg-generic.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index f9564831cc..e437050175 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -97,7 +97,8 @@ endif # $2: staging directory of the package ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define fixup-libtool-files - $(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \ + $(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \ + -name "*.la" | xargs --no-run-if-empty \ $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" endef endif @@ -254,6 +255,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) From patchwork Mon Jun 21 14:11:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495144 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s2L5B0Tz9sWQ for ; Tue, 22 Jun 2021 00:13:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id DF383400DD; Mon, 21 Jun 2021 14:13:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jwSHMF0L8cV3; Mon, 21 Jun 2021 14:13:27 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 5947F4010D; Mon, 21 Jun 2021 14:13:26 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 999061BF355 for ; Mon, 21 Jun 2021 14:12:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 896D460017 for ; Mon, 21 Jun 2021 14:12:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lNoiPBk32VEw for ; Mon, 21 Jun 2021 14:12:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp3.osuosl.org (Postfix) with ESMTPS id C9B6060631 for ; Mon, 21 Jun 2021 14:12:42 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 3B93B20000B; Mon, 21 Jun 2021 14:12:40 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:21 +0200 Message-Id: <20210621141130.48654-7-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 06/15] package/pkg-generic.mk: Introduce _PER_PACKAGE_TWEAK_HOOKS X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This hook will be used by package infrastucture or some packages to tweak external files (ie files not provides by the package itself) when we build with BR2_PER_PACKAGE_DIRECTORIES set. This allows to have these tweaks in a specific hook instead of _POST_CONFIGURE_HOOK. And so we can call these specific tweaks whenever we need them. Signed-off-by: Herve Codina --- package/pkg-generic.mk | 2 ++ 1 file changed, 2 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index e437050175..4069d2cf3c 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -257,6 +257,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) @@ -844,6 +845,7 @@ $(2)_PRE_LEGAL_INFO_HOOKS ?= $(2)_POST_LEGAL_INFO_HOOKS ?= $(2)_TARGET_FINALIZE_HOOKS ?= $(2)_ROOTFS_PRE_CMD_HOOKS ?= +$(2)_PER_PACKAGE_TWEAK_HOOKS ?= ifeq ($$($(2)_TYPE),target) ifneq ($$(HOST_$(2)_KCONFIG_VAR),) From patchwork Mon Jun 21 14:11:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495145 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s2f4R3pz9sWQ for ; Tue, 22 Jun 2021 00:13:46 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6B6BA400DD; Mon, 21 Jun 2021 14:13:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G9gu3ARmNOYl; Mon, 21 Jun 2021 14:13:39 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id B760640398; Mon, 21 Jun 2021 14:13:38 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 6F0761BF355 for ; Mon, 21 Jun 2021 14:12:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 6C0668274E for ; Mon, 21 Jun 2021 14:12:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oLUSZxenom6u for ; Mon, 21 Jun 2021 14:12:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp1.osuosl.org (Postfix) with ESMTPS id AAD9682656 for ; Mon, 21 Jun 2021 14:12:44 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 30E48200003; Mon, 21 Jun 2021 14:12:42 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:22 +0200 Message-Id: <20210621141130.48654-8-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 07/15] package/apr-util: Use _PER_PACKAGE_TWEAK_HOOKS X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Original APR_UTIL_FIX_RULES_MK_LIBTOOL tweaked libtool and rules.mk. libtool is provided by a dependency (apr). It needs to be tweaked and, as an apr-util external file, this tweak is relevant in _PER_PACKAGE_TWEAK_HOOKS. rules.mk is generated by apr-util configure step and it is private to apr-util. The modification performed needs to be kept in _POST_CONFIGURE_HOOKS. This commit splits original APR_UTIL_FIX_RULES_MK_LIBTOOL and attaches each part to the correct hook. Signed-off-by: Herve Codina --- package/apr-util/apr-util.mk | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/package/apr-util/apr-util.mk b/package/apr-util/apr-util.mk index db4df91564..b26d523265 100644 --- a/package/apr-util/apr-util.mk +++ b/package/apr-util/apr-util.mk @@ -18,13 +18,17 @@ APR_UTIL_CONF_OPTS = \ APR_UTIL_CONFIG_SCRIPTS = apu-1-config ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define APR_UTIL_FIX_RULES_MK_LIBTOOL - $(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \ - $(@D)/build/rules.mk +define APR_UTIL_FIX_LIBTOOL $(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \ $(STAGING_DIR)/usr/build-1/libtool endef -APR_UTIL_POST_CONFIGURE_HOOKS += APR_UTIL_FIX_RULES_MK_LIBTOOL +APR_UTIL_PER_PACKAGE_TWEAK_HOOKS += APR_UTIL_FIX_LIBTOOL + +define APR_UTIL_FIX_RULES_MK + $(SED) 's,$(PER_PACKAGE_DIR)/apr/,$(PER_PACKAGE_DIR)/apr-util/,g' \ + $(@D)/build/rules.mk +endef +APR_UTIL_POST_CONFIGURE_HOOKS += APR_UTIL_FIX_RULES_MK endif # When iconv is available, then use it to provide charset conversion From patchwork Mon Jun 21 14:11:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495146 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s2r1JWbz9sWQ for ; Tue, 22 Jun 2021 00:13:56 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 5D6D840265; Mon, 21 Jun 2021 14:13:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IdhxEB5CdCjL; Mon, 21 Jun 2021 14:13:52 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 543DF40267; Mon, 21 Jun 2021 14:13:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id DD8CA1BF355 for ; Mon, 21 Jun 2021 14:12:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id CCA1660631 for ; Mon, 21 Jun 2021 14:12:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3gv0SVsIds_7 for ; Mon, 21 Jun 2021 14:12:47 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp3.osuosl.org (Postfix) with ESMTPS id 10FA960017 for ; Mon, 21 Jun 2021 14:12:46 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 66A3F200008; Mon, 21 Jun 2021 14:12:44 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:23 +0200 Message-Id: <20210621141130.48654-9-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 08/15] package/apache: Move APACHE_FIXUP_APR_LIBTOOL to _PER_PACKAGE_TWEAK_HOOKS X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" APACHE_FIXUP_APR_LIBTOOL tweaks files for per package directory build. This is typically the kind of operation expected to be in _PER_PACKAGE_TWEAK_HOOKS Signed-off-by: Herve Codina --- package/apache/apache.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/apache/apache.mk b/package/apache/apache.mk index 7dbd1a4512..f496333174 100644 --- a/package/apache/apache.mk +++ b/package/apache/apache.mk @@ -23,7 +23,7 @@ define APACHE_FIXUP_APR_LIBTOOL $(SED) "s@$(PER_PACKAGE_DIR)/[^/]\+/@$(PER_PACKAGE_DIR)/apache/@g" \ $(STAGING_DIR)/usr/build-1/libtool endef -APACHE_POST_CONFIGURE_HOOKS += APACHE_FIXUP_APR_LIBTOOL +APACHE_PER_PACKAGE_TWEAK_HOOKS += APACHE_FIXUP_APR_LIBTOOL endif APACHE_CONF_ENV= \ From patchwork Mon Jun 21 14:11:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495147 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s3464Ttz9sWl for ; Tue, 22 Jun 2021 00:14:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id EF2AF403B5; Mon, 21 Jun 2021 14:14:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6V_7_qJFJIFK; Mon, 21 Jun 2021 14:14:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 60EBD403A3; Mon, 21 Jun 2021 14:14:04 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id F3A351BF355 for ; Mon, 21 Jun 2021 14:12:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id F08F6400F0 for ; Mon, 21 Jun 2021 14:12:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id reb8wOWc6HPC for ; Mon, 21 Jun 2021 14:12:49 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp2.osuosl.org (Postfix) with ESMTPS id 28470400DD for ; Mon, 21 Jun 2021 14:12:48 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id A932E200003; Mon, 21 Jun 2021 14:12:46 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:24 +0200 Message-Id: <20210621141130.48654-10-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 09/15] package/pkg-python: Remove _sysconfigdata*.pyc files when _sysconfigdata*.py are changed X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In order to avoid _sysconfigdata*.pyc overwrites when they are generated based on _sysconfigdata*.py changes, this commit simply removes _sysconfigdata*.pyc whenever _sysconfigdata*.py are changed. As they are removed, overwrite detection will no longer trig and coherency between the two files (.py and .pyc) is ensured. Signed-off-by: Herve Codina --- package/pkg-python.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/package/pkg-python.mk b/package/pkg-python.mk index 59a48e5a87..b3fde77da5 100644 --- a/package/pkg-python.mk +++ b/package/pkg-python.mk @@ -96,7 +96,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define PKG_PYTHON_FIXUP_SYSCONFIGDATA find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\ + find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f endef endif From patchwork Mon Jun 21 14:11:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495152 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s3K5lf9z9sWk for ; Tue, 22 Jun 2021 00:14:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1D71D403F6; Mon, 21 Jun 2021 14:14:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A8RW6xUsVyiI; Mon, 21 Jun 2021 14:14:17 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id ECD11403EA; Mon, 21 Jun 2021 14:14:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id E40C41BF355 for ; Mon, 21 Jun 2021 14:12:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id DDF8B40360 for ; Mon, 21 Jun 2021 14:12:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8Gro7W7bgICr for ; Mon, 21 Jun 2021 14:12:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp4.osuosl.org (Postfix) with ESMTPS id 999CD40348 for ; Mon, 21 Jun 2021 14:12:50 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 5F1CD200016; Mon, 21 Jun 2021 14:12:48 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:25 +0200 Message-Id: <20210621141130.48654-11-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 10/15] package/pkg-generic.mk: Move python fixup to generic package infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Fixing _sysconfigdata*.{py,pyc} was previously done by python package infrastructure. Some packages use python stuff without using python package infrastructure. These packages perform overwrites and need the specific python fixup to fix them. In order to be sure to fix all of these packages, the python fixup is moved to the generic package infrastructure and applied to all packages. This follows the same principle as for the .la libtool files fixup. Signed-off-by: Herve Codina --- package/pkg-generic.mk | 17 +++++++++++++++++ package/pkg-python.mk | 12 ------------ 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 4069d2cf3c..918e2381af 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -103,6 +103,21 @@ define fixup-libtool-files endef endif +# Make sure python _sysconfigdata*.py files only reference the current +# per-package directory. + +# $1: package name (lower case) +# $2: staging or host directory of the package +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define fixup-python-files + $(Q)find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \ + -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g" ;\ + find $(2) \( -path '$(2)/lib/python*' -o -path '$(2)/usr/lib/python*' \) \ + -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f +endef +endif + # Functions to detect overwritten files ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) @@ -257,6 +272,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(HOST_DIR)) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) + $(call fixup-python-files,$(NAME),$(HOST_DIR)) + $(call fixup-python-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PER_PACKAGE_TWEAK_HOOKS),$(call $(hook))$(sep)) @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) diff --git a/package/pkg-python.mk b/package/pkg-python.mk index b3fde77da5..e6b81bdfd3 100644 --- a/package/pkg-python.mk +++ b/package/pkg-python.mk @@ -92,16 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \ --root=/ \ --single-version-externally-managed -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define PKG_PYTHON_FIXUP_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g" ;\ - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.pyc" -print0 | xargs -0 -r rm -f -endef -endif - ################################################################################ # inner-python-package -- defines how the configuration, compilation # and installation of a Python package should be done, implements a @@ -246,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON) endif endif -$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA - # # Build step. Only define it if not already defined by the package .mk # file. From patchwork Mon Jun 21 14:11:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495153 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s3b1cM4z9sWQ for ; Tue, 22 Jun 2021 00:14:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 563A1606DA; Mon, 21 Jun 2021 14:14:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jAkmzKG_78Qv; Mon, 21 Jun 2021 14:14:30 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 8E916606E9; Mon, 21 Jun 2021 14:14:29 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id B1EDF1BF355 for ; Mon, 21 Jun 2021 14:12:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id AF0F6828F7 for ; Mon, 21 Jun 2021 14:12:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ddp_L_keisN5 for ; Mon, 21 Jun 2021 14:12:52 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp1.osuosl.org (Postfix) with ESMTPS id 01B9C82656 for ; Mon, 21 Jun 2021 14:12:51 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id EEC30200003; Mon, 21 Jun 2021 14:12:49 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:26 +0200 Message-Id: <20210621141130.48654-12-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 11/15] package/owfs: Remove Python sysconfigdata fixup X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This fixup is done at pkg-generic level for all packages. So, it is no more needed in owfs package. Signed-off-by: Herve Codina --- package/owfs/owfs.mk | 9 --------- 1 file changed, 9 deletions(-) diff --git a/package/owfs/owfs.mk b/package/owfs/owfs.mk index ffc0b3098d..10543d3698 100644 --- a/package/owfs/owfs.mk +++ b/package/owfs/owfs.mk @@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig # override the PYSITEDIR variable in make. OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages -ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) -define OWFS_FIXUP_PYTHON_SYSCONFIGDATA - find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \ - -name "_sysconfigdata*.py" | xargs --no-run-if-empty \ - $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g" -endef -OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA -endif - else OWFS_CONF_OPTS += --disable-owpython --without-python endif From patchwork Mon Jun 21 14:11:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495154 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s3n5kMgz9sX1 for ; Tue, 22 Jun 2021 00:14:45 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 96767608A1; Mon, 21 Jun 2021 14:14:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G7EswiQc3AzC; Mon, 21 Jun 2021 14:14:42 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp3.osuosl.org (Postfix) with ESMTP id 03B0B60872; Mon, 21 Jun 2021 14:14:42 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 2F2A41BF355 for ; Mon, 21 Jun 2021 14:12:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2B965606DA for ; Mon, 21 Jun 2021 14:12:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WbSuUysFI0Bo for ; Mon, 21 Jun 2021 14:12:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4287860631 for ; Mon, 21 Jun 2021 14:12:54 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id DF4B5200010; Mon, 21 Jun 2021 14:12:51 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:27 +0200 Message-Id: <20210621141130.48654-13-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 12/15] package/pkg-generic.mk: Generate final rsync exclude file list X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The final rsync performed at {host,target}-finalize steps need to be done by rsyncing files generated by each packages without looking at files generated by other packages the current package depends on. This is needed to avoid overwrites in final {HOST,TARGET}_DIR. In order to prepare the final rsync, an exclusion list is generated. This list lists files that are not generated by the current package and so files that need to be excluded from the final rsync. Note also that the files list was not based on .files-list.{before,after}. During .file-list.{before,after} built for host directory, staging sub-directory (ie /sysroot) is filtered out. The final rsync exclusion list needs to take into account the full {host,target} directory to avoid final overwrites. Signed-off-by: Herve Codina --- package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 918e2381af..546370af7a 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -162,6 +162,30 @@ define pkg_size_after rm -f $($(PKG)_DIR)/.files-list$(2).after endef +# Functions to collect final rsync exclusion files + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_before + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_final_rsync_after + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after + LC_ALL=C comm -2 \ + $($(PKG)_DIR)/.files-final-rsync$(2).before \ + $($(PKG)_DIR)/.files-final-rsync$(2).after \ + | sed -r -e 's/^[^,]+,./- /' \ + > $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync + rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after +endef + define check_bin_arch support/scripts/check-bin-arch -p $($(PKG)_NAME) \ -l $($(PKG)_DIR)/.files-list.txt \ @@ -267,6 +291,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call MESSAGE,"Configuring") $(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR) $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) + @$(call pkg_final_rsync_before,$(TARGET_DIR)) + @$(call pkg_final_rsync_before,$(HOST_DIR),-host) @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) @@ -404,6 +430,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_final_rsync_after,$(TARGET_DIR)) + @$(call pkg_final_rsync_after,$(HOST_DIR),-host) @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ From patchwork Mon Jun 21 14:11:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495155 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s446RbYz9sWQ for ; Tue, 22 Jun 2021 00:15:00 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D1BBE401FC; Mon, 21 Jun 2021 14:14:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yOsQlLX38sCH; Mon, 21 Jun 2021 14:14:55 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 696AC403A7; Mon, 21 Jun 2021 14:14:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id F1D421BF355 for ; Mon, 21 Jun 2021 14:13:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E22D8827D9 for ; Mon, 21 Jun 2021 14:13:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u1ORTN6V0HbQ for ; Mon, 21 Jun 2021 14:12:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3B82F82948 for ; Mon, 21 Jun 2021 14:12:56 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id F12EE200005; Mon, 21 Jun 2021 14:12:53 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:28 +0200 Message-Id: <20210621141130.48654-14-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 13/15] Makefile: Rsync global {TARGET, HOST}_DIR using exclusion file list X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" On a per-package build, rsync final {TARGET,HOST}_DIR using exclusion file list computed for each packages. As we rsync only files generated by each packages, we need to to do this rsync recursively on each dependencies to collect all needed files. This is done based on existing _FINAL_RECURSIVE_DEPENDENCIES Signed-off-by: Herve Codina --- Makefile | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index 472af5a318..50bc6b4503 100644 --- a/Makefile +++ b/Makefile @@ -740,10 +740,28 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt)) HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt)) STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt)) +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# rsync the contents of per-package directories +# $1: space-separated list of packages to rsync from +# $2: 'host' or 'target' +# $3: destination directory +# $4: exclude file list to use +define per-package-rsync-delta + mkdir -p $(3) + $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ + $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + $(3)$(sep)) +endef +endif + .PHONY: host-finalize host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) @$(call MESSAGE,"Finalizing host directory") - $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + host,$(HOST_DIR),.files-final-rsync-host.exclude_rsync) .PHONY: staging-finalize staging-finalize: $(STAGING_DIR_SYMLINK) @@ -751,7 +769,9 @@ staging-finalize: $(STAGING_DIR_SYMLINK) .PHONY: target-finalize target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize @$(call MESSAGE,"Finalizing target directory") - $(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR)) + $(call per-package-rsync-delta, $(sort $(foreach pkg, $(PACKAGES), \ + $(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES))), \ + target,$(TARGET_DIR),.files-final-rsync.exclude_rsync) $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \ From patchwork Mon Jun 21 14:11:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495157 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s4f3RHhz9sWk for ; Tue, 22 Jun 2021 00:15:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 557CD40497; Mon, 21 Jun 2021 14:15:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VTznXT1VUHaj; Mon, 21 Jun 2021 14:15:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp4.osuosl.org (Postfix) with ESMTP id 947EA404B7; Mon, 21 Jun 2021 14:15:07 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 8540F1BF355 for ; Mon, 21 Jun 2021 14:13:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 7171760870 for ; Mon, 21 Jun 2021 14:13:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lCJ6WwIOffK7 for ; Mon, 21 Jun 2021 14:12:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp3.osuosl.org (Postfix) with ESMTPS id BB7BE60631 for ; Mon, 21 Jun 2021 14:12:57 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id AD5F4200009; Mon, 21 Jun 2021 14:12:55 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:29 +0200 Message-Id: <20210621141130.48654-15-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 14/15] Makefile: Breaks hardlinks in global {TARGET, HOST}_DIR on per-package build X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Without this patch, a make _rebuild detects overwrites. Indeed, in target_finalize steps some modifications are done on installed files (ie strip or TARGET_FINALIZE_HOOKS for instance). In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built using a full copy of the involved per-package files instead of hardlinks. Signed-off-by: Herve Codina --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 50bc6b4503..fe6e3f95af 100644 --- a/Makefile +++ b/Makefile @@ -749,7 +749,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) define per-package-rsync-delta mkdir -p $(3) $(foreach pkg,$(1),\ - rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ + rsync -a \ --filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \ $(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \ $(3)$(sep)) From patchwork Mon Jun 21 14:11:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Herve Codina X-Patchwork-Id: 1495156 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G7s4g0SS7z9sX1 for ; Tue, 22 Jun 2021 00:15:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 22346402DC; Mon, 21 Jun 2021 14:15:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cgacbucG06FR; Mon, 21 Jun 2021 14:15:25 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 19305403A3; Mon, 21 Jun 2021 14:15:24 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 9CC8B1BF355 for ; Mon, 21 Jun 2021 14:13:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 976BB400F0 for ; Mon, 21 Jun 2021 14:13:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LCWY9BmuVcL5 for ; Mon, 21 Jun 2021 14:12:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by smtp2.osuosl.org (Postfix) with ESMTPS id 76414400DD for ; Mon, 21 Jun 2021 14:12:59 +0000 (UTC) Received: from localhost.localdomain (unknown [88.163.90.193]) (Authenticated sender: herve.codina@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPA id 0EAB4200003; Mon, 21 Jun 2021 14:12:56 +0000 (UTC) From: Herve Codina To: buildroot@buildroot.org Date: Mon, 21 Jun 2021 16:11:30 +0200 Message-Id: <20210621141130.48654-16-herve.codina@bootlin.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210621141130.48654-1-herve.codina@bootlin.com> References: <20210621141130.48654-1-herve.codina@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 15/15] package/pkg-generic.mk: Fix per-package -{reconfigure, rebuild, reinstall} X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Bernd Kuhls , Herve Codina , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Many overwrites are detected on -{reconfigure,rebuild,reinstall}. Indeed, files previously installed by a package were detected as overwritten on next reconfigure, rebuild or reinstall. To avoid this, we recreate per-package host and target dir from scratch as it was done during the first configure step. Signed-off-by: Herve Codina --- package/pkg-generic.mk | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 546370af7a..0aafd865ad 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -470,6 +470,13 @@ define pkg-graph-depends $$(GRAPHS_DIR)/$$(@).dot endef +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define empty-per-package-directory + rm -rf $(HOST_DIR) $(TARGET_DIR) + mkdir -p $(HOST_DIR) $(TARGET_DIR) +endef +endif + ################################################################################ # inner-generic-package -- generates the make targets needed to build a # generic package @@ -1072,6 +1079,25 @@ endif rm -f $$($(2)_TARGET_INSTALL_TARGET) rm -f $$($(2)_TARGET_INSTALL_IMAGES) rm -f $$($(2)_TARGET_INSTALL_HOST) + $$(call empty-per-package-directory) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES)) + $$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES)) + @$$(call pkg_final_rsync_before,$$(TARGET_DIR)) + @$$(call pkg_final_rsync_before,$$(HOST_DIR),-host) + @$$(call pkg_size_before,$$(TARGET_DIR)) + @$$(call pkg_size_before,$$(STAGING_DIR),-staging) + @$$(call pkg_size_before,$$(HOST_DIR),-host) + $$(call fixup-libtool-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-libtool-files,$$(NAME),$$(STAGING_DIR)) + $$(call fixup-python-files,$$(NAME),$$(HOST_DIR)) + $$(call fixup-python-files,$$(NAME),$$(STAGING_DIR)) + $$(foreach hook,$$($(2)_PER_PACKAGE_TWEAK_HOOKS),$$(call $$(hook))$$(sep)) + @$$(call pkg_detect_overwrite_before,$$(TARGET_DIR)) + @$$(call pkg_detect_overwrite_before,$$(HOST_DIR),-host) + +$(1)-clean-for-reinstall: PKG=$(2) +$(1)-clean-for-reinstall: NAME=$(1) $(1)-reinstall: $(1)-clean-for-reinstall $(1)