From patchwork Thu Apr 30 09:52:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280102 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0M65y0z9sPF for ; Thu, 30 Apr 2020 19:53:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 157E38868D; Thu, 30 Apr 2020 09:53:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2UIkfW2fOfAA; Thu, 30 Apr 2020 09:53:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 45A8A88689; Thu, 30 Apr 2020 09:53:04 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id BF7691BF9B4 for ; Thu, 30 Apr 2020 09:53:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B952C86BB9 for ; Thu, 30 Apr 2020 09:53:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rKpBcnyAYnT0 for ; Thu, 30 Apr 2020 09:52:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 2F3C486A29 for ; Thu, 30 Apr 2020 09:52:58 +0000 (UTC) X-Originating-IP: 86.210.146.109 Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id B419CC0002; Thu, 30 Apr 2020 09:52:55 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:38 +0200 Message-Id: <20200430095249.782597-2-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 01/11] package/pkg-generic.mk: use $(PKG)_NAME in step_pkg_size_after X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Since we're already using $(PKG)_DIR in step_pkg_size_after, we can also just use $(PKG)_NAME. This allows to make $(1) useless, which means it can be dropped in a follow-up commit. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 8cd5a7ff62..d8df34ed14 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -76,7 +76,7 @@ define step_pkg_size_after LC_ALL=C comm -13 \ $($(PKG)_DIR)/.files-list$(3).before \ $($(PKG)_DIR)/.files-list$(3).after \ - | sed -r -e 's/^[^,]+/$(1)/' \ + | sed -r -e 's/^[^,]+/$($(PKG)_NAME)/' \ > $($(PKG)_DIR)/.files-list$(3).txt rm -f $($(PKG)_DIR)/.files-list$(3).before rm -f $($(PKG)_DIR)/.files-list$(3).after From patchwork Thu Apr 30 09:52:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280101 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0L1vfZz9sPF for ; Thu, 30 Apr 2020 19:53:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id D565D86A29; Thu, 30 Apr 2020 09:53:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z3MLZh5iAxaN; Thu, 30 Apr 2020 09:53:03 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3A64786BB9; Thu, 30 Apr 2020 09:53:03 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id BCBA81BF86C for ; Thu, 30 Apr 2020 09:53:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B832287E17 for ; Thu, 30 Apr 2020 09:53:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b9Q5-QQ+Ghrq for ; Thu, 30 Apr 2020 09:52:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id 944F287DF9 for ; Thu, 30 Apr 2020 09:52:58 +0000 (UTC) X-Originating-IP: 86.210.146.109 Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id E5045240009; Thu, 30 Apr 2020 09:52:56 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:39 +0200 Message-Id: <20200430095249.782597-3-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 02/11] package/pkg-generic.mk: drop useless $(1) argument in step_pkg_size_{before, after} X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The $(1) argument passed to step_pkg_size_{before,after}, which contains the package name, is no longer used. We simply use $(PKG) to get the upper-case version of the package name. So, let's drop this first argument that isn't needed. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 40 +++++++++++++++++++--------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index d8df34ed14..fc43807bb0 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -57,45 +57,43 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files -# $(1): package name -# $(2): base directory to search in -# $(3): suffix of file (optional) +# $(1): base directory to search in +# $(2): suffix of file (optional) define step_pkg_size_before - cd $(2); \ + cd $(1); \ LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ - | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(3).before + | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).before endef -# $(1): package name -# $(2): base directory to search in -# $(3): suffix of file (optional) +# $(1): base directory to search in +# $(2): suffix of file (optional) define step_pkg_size_after - cd $(2); \ + cd $(1); \ LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ - | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(3).after + | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).after LC_ALL=C comm -13 \ - $($(PKG)_DIR)/.files-list$(3).before \ - $($(PKG)_DIR)/.files-list$(3).after \ + $($(PKG)_DIR)/.files-list$(2).before \ + $($(PKG)_DIR)/.files-list$(2).after \ | sed -r -e 's/^[^,]+/$($(PKG)_NAME)/' \ - > $($(PKG)_DIR)/.files-list$(3).txt - rm -f $($(PKG)_DIR)/.files-list$(3).before - rm -f $($(PKG)_DIR)/.files-list$(3).after + > $($(PKG)_DIR)/.files-list$(2).txt + rm -f $($(PKG)_DIR)/.files-list$(2).before + rm -f $($(PKG)_DIR)/.files-list$(2).after endef define step_pkg_size $(if $(filter start-install-target,$(1)-$(2)),\ - $(call step_pkg_size_before,$(3),$(TARGET_DIR))) + $(call step_pkg_size_before,$(TARGET_DIR))) $(if $(filter start-install-staging,$(1)-$(2)),\ - $(call step_pkg_size_before,$(3),$(STAGING_DIR),-staging)) + $(call step_pkg_size_before,$(STAGING_DIR),-staging)) $(if $(filter start-install-host,$(1)-$(2)),\ - $(call step_pkg_size_before,$(3),$(HOST_DIR),-host)) + $(call step_pkg_size_before,$(HOST_DIR),-host)) $(if $(filter end-install-target,$(1)-$(2)),\ - $(call step_pkg_size_after,$(3),$(TARGET_DIR))) + $(call step_pkg_size_after,$(TARGET_DIR))) $(if $(filter end-install-staging,$(1)-$(2)),\ - $(call step_pkg_size_after,$(3),$(STAGING_DIR),-staging)) + $(call step_pkg_size_after,$(STAGING_DIR),-staging)) $(if $(filter end-install-host,$(1)-$(2)),\ - $(call step_pkg_size_after,$(3),$(HOST_DIR),-host)) + $(call step_pkg_size_after,$(HOST_DIR),-host)) endef GLOBAL_INSTRUMENTATION_HOOKS += step_pkg_size From patchwork Thu Apr 30 09:52:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280103 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0Q0lvvz9sSd for ; Thu, 30 Apr 2020 19:53:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7E7AF86D04; Thu, 30 Apr 2020 09:53:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DcJH-N81Oe67; Thu, 30 Apr 2020 09:53:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id D37A386CC5; Thu, 30 Apr 2020 09:53:05 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 6474E1BF86C for ; Thu, 30 Apr 2020 09:53:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6159F87DF9 for ; Thu, 30 Apr 2020 09:53:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2H6zXHg7-CXQ for ; Thu, 30 Apr 2020 09:53:00 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by whitealder.osuosl.org (Postfix) with ESMTPS id 221CA87E0B for ; Thu, 30 Apr 2020 09:52:59 +0000 (UTC) X-Originating-IP: 86.210.146.109 Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 1C023C000C; Thu, 30 Apr 2020 09:52:58 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:40 +0200 Message-Id: <20200430095249.782597-4-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 03/11] package/pkg-generic.mk: introduce final 'install' step X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" We currently have four different install steps: target installation, staging installation, images installation and host installation. These steps are directly triggered from the $(1)-install make target, so there is no place where we can run some logic once all installation steps have completed. However, as part of improving the reliability of the logic done in step_pkg_size_before and step_pkg_size_after to detect the files installed by packages, we would in fact need to run some logic after all installation steps have completed. This will allow us to make sure that all files are detected, even if a host package installs something in the target directory, or if a target package installs something in the host directory. To achieve this, this commit implements a new stamp file, .stamp_installed, which is a step that depends on all four install steps. Currently, this step does nothing except creating the stamp file. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index fc43807bb0..43f6f86b40 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -374,6 +374,11 @@ $(BUILD_DIR)/%/.stamp_target_installed: @$(call step_end,install-target) $(Q)touch $@ +# Final installation step, completed when all installation steps +# (host, images, staging, target) have completed +$(BUILD_DIR)/%/.stamp_installed: + $(Q)touch $@ + # Remove package sources $(BUILD_DIR)/%/.stamp_dircleaned: $(if $(BR2_PER_PACKAGE_DIRECTORIES),rm -Rf $(PER_PACKAGE_DIR)/$(NAME)) @@ -705,6 +710,7 @@ $(2)_FINAL_RECURSIVE_RDEPENDENCIES = $$(sort \ $$($(2)_FINAL_RECURSIVE_RDEPENDENCIES__X)) # define sub-target stamps +$(2)_TARGET_INSTALL = $$($(2)_DIR)/.stamp_installed $(2)_TARGET_INSTALL_TARGET = $$($(2)_DIR)/.stamp_target_installed $(2)_TARGET_INSTALL_STAGING = $$($(2)_DIR)/.stamp_staging_installed $(2)_TARGET_INSTALL_IMAGES = $$($(2)_DIR)/.stamp_images_installed @@ -760,14 +766,23 @@ endif # human-friendly targets and target sequencing $(1): $(1)-install +$(1)-install: $$($(2)_TARGET_INSTALL) ifeq ($$($(2)_TYPE),host) -$(1)-install: $(1)-install-host +$$($(2)_TARGET_INSTALL): $$($(2)_TARGET_INSTALL_HOST) else $(2)_INSTALL_STAGING ?= NO $(2)_INSTALL_IMAGES ?= NO $(2)_INSTALL_TARGET ?= YES -$(1)-install: $(1)-install-staging $(1)-install-target $(1)-install-images +ifeq ($$($(2)_INSTALL_TARGET),YES) +$$($(2)_TARGET_INSTALL): $$($(2)_TARGET_INSTALL_TARGET) +endif +ifeq ($$($(2)_INSTALL_STAGING),YES) +$$($(2)_TARGET_INSTALL): $$($(2)_TARGET_INSTALL_STAGING) +endif +ifeq ($$($(2)_INSTALL_IMAGES),YES) +$$($(2)_TARGET_INSTALL): $$($(2)_TARGET_INSTALL_IMAGES) +endif endif ifeq ($$($(2)_INSTALL_TARGET),YES) @@ -936,6 +951,7 @@ $(1)-reconfigure: $(1)-clean-for-reconfigure $(1) # define the PKG variable for all targets, containing the # uppercase package variable prefix +$$($(2)_TARGET_INSTALL): PKG=$(2) $$($(2)_TARGET_INSTALL_TARGET): PKG=$(2) $$($(2)_TARGET_INSTALL_STAGING): PKG=$(2) $$($(2)_TARGET_INSTALL_IMAGES): PKG=$(2) From patchwork Thu Apr 30 09:52:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280104 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0V0Bznz9sSd for ; Thu, 30 Apr 2020 19:53:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B1ABD87E15; Thu, 30 Apr 2020 09:53:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id k0def2koehV2; Thu, 30 Apr 2020 09:53:07 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id E38B387E0B; Thu, 30 Apr 2020 09:53:06 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 37E5C1BF86C for ; Thu, 30 Apr 2020 09:53:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3460386BB9 for ; Thu, 30 Apr 2020 09:53:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1qkVWtRT9jD4 for ; Thu, 30 Apr 2020 09:53:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 1A34586A29 for ; Thu, 30 Apr 2020 09:53:00 +0000 (UTC) X-Originating-IP: 86.210.146.109 Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 6BCE9240015; Thu, 30 Apr 2020 09:52:59 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:41 +0200 Message-Id: <20200430095249.782597-5-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 04/11] package/pkg-generic.mk: create directories upfront in the configure step X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" We currently create HOST_DIR, TARGET_DIR, STAGING_DIR and BINARIES_DIR in their respective installation steps. However, as we are about to change how the logic to capture files installed by packages is implemented, we will need these directories to exist at the configure step to keep things simple. Note that when BR2_PER_PACKAGE_DIRECTORIES=y, the HOST_DIR, TARGET_DIR and STAGING_DIR are anyway already created at the configure step, when populating the per-package HOST_DIR and TARGET_DIR. This also means that we can drop the "mkdir" from per-package-rsync. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 5 +---- package/pkg-utils.mk | 1 - 2 files changed, 1 insertion(+), 5 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 43f6f86b40..efdc32e355 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -249,6 +249,7 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\ # Configure $(BUILD_DIR)/%/.stamp_configured: + @mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR) @$(call step_start,configure) @$(call MESSAGE,"Configuring") $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) @@ -271,7 +272,6 @@ $(BUILD_DIR)/%/.stamp_built:: # Install to host dir $(BUILD_DIR)/%/.stamp_host_installed: - @mkdir -p $(HOST_DIR) @$(call step_start,install-host) @$(call MESSAGE,"Installing to host directory") $(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep)) @@ -301,7 +301,6 @@ $(BUILD_DIR)/%/.stamp_host_installed: # empty when we use an internal toolchain. # $(BUILD_DIR)/%/.stamp_staging_installed: - @mkdir -p $(STAGING_DIR) @$(call step_start,install-staging) @$(call MESSAGE,"Installing to staging directory") $(foreach hook,$($(PKG)_PRE_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep)) @@ -344,7 +343,6 @@ $(BUILD_DIR)/%/.stamp_staging_installed: # Install to images dir $(BUILD_DIR)/%/.stamp_images_installed: - @mkdir -p $(BINARIES_DIR) @$(call step_start,install-image) @$(call MESSAGE,"Installing to images directory") $(foreach hook,$($(PKG)_PRE_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep)) @@ -355,7 +353,6 @@ $(BUILD_DIR)/%/.stamp_images_installed: # Install to target dir $(BUILD_DIR)/%/.stamp_target_installed: - @mkdir -p $(TARGET_DIR) @$(call step_start,install-target) @$(call MESSAGE,"Installing to target") $(foreach hook,$($(PKG)_PRE_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep)) diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index 66504d0be2..dd7b06ac62 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -156,7 +156,6 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) # $2: 'host' or 'target' # $3: destination directory define per-package-rsync - mkdir -p $(3) $(foreach pkg,$(1),\ rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ $(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ From patchwork Thu Apr 30 09:52:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280109 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0m4ZRgz9sPF for ; Thu, 30 Apr 2020 19:53:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id E3EEB24C8E; Thu, 30 Apr 2020 09:53:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oef3Xb5vlh3M; Thu, 30 Apr 2020 09:53:16 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 30C37204D3; Thu, 30 Apr 2020 09:53:15 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 2F6F31BF86C for ; Thu, 30 Apr 2020 09:53:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 259D6204FA for ; Thu, 30 Apr 2020 09:53:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E-GeoKzLk3-k for ; Thu, 30 Apr 2020 09:53:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by silver.osuosl.org (Postfix) with ESMTPS id 2EFDD204E4 for ; Thu, 30 Apr 2020 09:53:05 +0000 (UTC) Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id E427E20000E; Thu, 30 Apr 2020 09:53:00 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:42 +0200 Message-Id: <20200430095249.782597-6-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 05/11] package/pkg-generic.mk: rework pkg_size logic with the "installed" step X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commits reworks the pkg_size logic to no longer use the GLOBAL_INSTRUMENTATION_HOOKS mechanism, but instead be directly implemented within the configure step and install step. The problem with the current implementation in the GLOBAL_INSTRUMENTATION_HOOKS is that we only capture what is installed in $(HOST_DIR) during the "host installation step", what is installed in $(TARGET_DIR) during the "target installation step" and what is installed in "$(STAGING_DIR)" during the staging installation step. While this seems reasonable in principle, it is in fact not completely true. For example, "toolchain", which is a target package, installs tons of files in $(HOST_DIR). "qt5base", which is also a target package, also installs things in $(HOST_DIR). Same with the "syslinux" package. The idea behind this patch is pretty simple: - At the beginning of the configure step, right after the per-package directories have been populated (if BR2_PER_PACKAGE_DIRECTORIES=y), we capture the state of the HOST_DIR, TARGET_DIR and STAGING_DIR. - At the end of all install steps (which is possible thanks to the newly introduced "install" step), we capture again the state of HOST_DIR, TARGET_DIR and STAGING_DIR, and compare it to what we have saved at the configure step. So regardless of whether a file has been installed in $(HOST_DIR) during the target or staging installation steps of a target package, or if a host package has installed a file in $(TARGET_DIR), we will detect it. The pkg_size_before and pkg_size_after macros are intentionally left where they are (even if they now fall in the middle of the GLOBAL_INSTRUMENTATION_HOOKS implementations) to minimize the diffstat and facilitate review. Note that we also have to change check_bin_arch to be explicitly called from the install step rather than through a GLOBAL_INSTRUMENTATION_HOOKS as it depends on the .files-list.txt file produced by the pkg_size_after function. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 42 ++++++++++++++---------------------------- 1 file changed, 14 insertions(+), 28 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index efdc32e355..3fb1e5f8c3 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -59,7 +59,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # $(1): base directory to search in # $(2): suffix of file (optional) -define step_pkg_size_before +define pkg_size_before cd $(1); \ LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).before @@ -67,7 +67,7 @@ endef # $(1): base directory to search in # $(2): suffix of file (optional) -define step_pkg_size_after +define pkg_size_after cd $(1); \ LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).after @@ -80,35 +80,14 @@ define step_pkg_size_after rm -f $($(PKG)_DIR)/.files-list$(2).after endef -define step_pkg_size - $(if $(filter start-install-target,$(1)-$(2)),\ - $(call step_pkg_size_before,$(TARGET_DIR))) - $(if $(filter start-install-staging,$(1)-$(2)),\ - $(call step_pkg_size_before,$(STAGING_DIR),-staging)) - $(if $(filter start-install-host,$(1)-$(2)),\ - $(call step_pkg_size_before,$(HOST_DIR),-host)) - - $(if $(filter end-install-target,$(1)-$(2)),\ - $(call step_pkg_size_after,$(TARGET_DIR))) - $(if $(filter end-install-staging,$(1)-$(2)),\ - $(call step_pkg_size_after,$(STAGING_DIR),-staging)) - $(if $(filter end-install-host,$(1)-$(2)),\ - $(call step_pkg_size_after,$(HOST_DIR),-host)) -endef -GLOBAL_INSTRUMENTATION_HOOKS += step_pkg_size - -# Relies on step_pkg_size, so must be after define check_bin_arch - $(if $(filter end-install-target,$(1)-$(2)),\ - support/scripts/check-bin-arch -p $(3) \ - -l $($(PKG)_DIR)/.files-list.txt \ - $(foreach i,$($(PKG)_BIN_ARCH_EXCLUDE),-i "$(i)") \ - -r $(TARGET_READELF) \ - -a $(BR2_READELF_ARCH_NAME)) + support/scripts/check-bin-arch -p $($(PKG)_NAME) \ + -l $($(PKG)_DIR)/.files-list.txt \ + $(foreach i,$($(PKG)_BIN_ARCH_EXCLUDE),-i "$(i)") \ + -r $(TARGET_READELF) \ + -a $(BR2_READELF_ARCH_NAME) endef -GLOBAL_INSTRUMENTATION_HOOKS += check_bin_arch - # This hook checks that host packages that need libraries that we build # have a proper DT_RPATH or DT_RUNPATH tag define check_host_rpath @@ -253,6 +232,9 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call step_start,configure) @$(call MESSAGE,"Configuring") $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) + @$(call pkg_size_before,$(TARGET_DIR)) + @$(call pkg_size_before,$(STAGING_DIR),-staging) + @$(call pkg_size_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) @@ -374,6 +356,10 @@ $(BUILD_DIR)/%/.stamp_target_installed: # Final installation step, completed when all installation steps # (host, images, staging, target) have completed $(BUILD_DIR)/%/.stamp_installed: + @$(call pkg_size_after,$(TARGET_DIR)) + @$(call pkg_size_after,$(STAGING_DIR),-staging) + @$(call pkg_size_after,$(HOST_DIR),-host) + @$(call check_bin_arch) $(Q)touch $@ # Remove package sources From patchwork Thu Apr 30 09:52:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280106 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0b2cMdz9sSd for ; Thu, 30 Apr 2020 19:53:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9BD44235AE; Thu, 30 Apr 2020 09:53:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 38Yci2sqPc+O; Thu, 30 Apr 2020 09:53:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id ED9A9204E4; Thu, 30 Apr 2020 09:53:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 797B81BF86C for ; Thu, 30 Apr 2020 09:53:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 6D680204FA for ; Thu, 30 Apr 2020 09:53:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JkwsBVtogfuu for ; Thu, 30 Apr 2020 09:53:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by silver.osuosl.org (Postfix) with ESMTPS id 44091204D3 for ; Thu, 30 Apr 2020 09:53:04 +0000 (UTC) X-Originating-IP: 86.210.146.109 Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id 8933A240009; Thu, 30 Apr 2020 09:53:02 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:43 +0200 Message-Id: <20200430095249.782597-7-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 06/11] package/pkg-generic.mk: exclude the staging sub-directory X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Now that we are checking the host directory changes throughout all installation steps and not just during the "host installation step", it means that changes done within the staging directory (which is a subdir of the host directory) are also visible in the .files-list-host.txt file. Note that this problem already potentially occurs if a host package is installing files in the staging directory: they would be listed in .files-list-host.txt even without the changes in this series. To fix this up, we simply exclude files that are beneath the $(STAGING_SUBDIR). Note that we do that in all cases, so when searching $(HOST_DIR), $(HOST_DIR)/$(STAGING_SUBDIR) is excluded, but when searching $(TARGET_DIR), $(TARGET_DIR)/$(STAGING_SUBDIR) is excluded, and when search $(STAGING_DIR), $(STAGING_DIR)/$(STAGING_SUBDIR) is excluded. This is not a problem in practice since $(TARGET_DIR)/$(STAGING_SUBDIR) and $(STAGING_DIR)/$(STAGING_SUBDIR) don't exist, but it's not very nice. However, it allows to keep the code simple. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 3fb1e5f8c3..2ae269bb3d 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -61,7 +61,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # $(2): suffix of file (optional) define pkg_size_before cd $(1); \ - LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + LC_ALL=C find . -not -path './$(STAGING_SUBDIR)/*' \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).before endef @@ -69,7 +69,7 @@ endef # $(2): suffix of file (optional) define pkg_size_after cd $(1); \ - LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + LC_ALL=C find . -not -path './$(STAGING_SUBDIR)/*' \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).after LC_ALL=C comm -13 \ $($(PKG)_DIR)/.files-list$(2).before \ From patchwork Thu Apr 30 09:52:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280105 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0b0wY7z9sPF for ; Thu, 30 Apr 2020 19:53:19 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 7D93A88692; Thu, 30 Apr 2020 09:53:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GAdz3ptVUSK2; Thu, 30 Apr 2020 09:53:16 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id C6AD688688; Thu, 30 Apr 2020 09:53:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 4405F1BF9B4 for ; Thu, 30 Apr 2020 09:53:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 39F2B204E4 for ; Thu, 30 Apr 2020 09:53:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f6lmFqOigEF6 for ; Thu, 30 Apr 2020 09:53:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by silver.osuosl.org (Postfix) with ESMTPS id BED01204F1 for ; Thu, 30 Apr 2020 09:53:05 +0000 (UTC) Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id E208E20000B; Thu, 30 Apr 2020 09:53:03 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:44 +0200 Message-Id: <20200430095249.782597-8-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 07/11] package/pkg-generic.mk: move pkg_size_{before, after} and check_bin_arch functions X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" These functions are no longer using the GLOBAL_INSTRUMENTATION_HOOKS mechanism, so it doesn't make much sense for them to be in the section of pkg-generic.mk related to those hooks. Move them to the "Helper functions" section. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 66 +++++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 33 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 2ae269bb3d..6e06d735ad 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -55,39 +55,6 @@ define step_time endef GLOBAL_INSTRUMENTATION_HOOKS += step_time -# Hooks to collect statistics about installed files - -# $(1): base directory to search in -# $(2): suffix of file (optional) -define pkg_size_before - cd $(1); \ - LC_ALL=C find . -not -path './$(STAGING_SUBDIR)/*' \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ - | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).before -endef - -# $(1): base directory to search in -# $(2): suffix of file (optional) -define pkg_size_after - cd $(1); \ - LC_ALL=C find . -not -path './$(STAGING_SUBDIR)/*' \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ - | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).after - LC_ALL=C comm -13 \ - $($(PKG)_DIR)/.files-list$(2).before \ - $($(PKG)_DIR)/.files-list$(2).after \ - | sed -r -e 's/^[^,]+/$($(PKG)_NAME)/' \ - > $($(PKG)_DIR)/.files-list$(2).txt - rm -f $($(PKG)_DIR)/.files-list$(2).before - rm -f $($(PKG)_DIR)/.files-list$(2).after -endef - -define check_bin_arch - support/scripts/check-bin-arch -p $($(PKG)_NAME) \ - -l $($(PKG)_DIR)/.files-list.txt \ - $(foreach i,$($(PKG)_BIN_ARCH_EXCLUDE),-i "$(i)") \ - -r $(TARGET_READELF) \ - -a $(BR2_READELF_ARCH_NAME) -endef - # This hook checks that host packages that need libraries that we build # have a proper DT_RPATH or DT_RUNPATH tag define check_host_rpath @@ -135,6 +102,39 @@ define fixup-libtool-files endef endif +# Functions to collect statistics about installed files + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_size_before + cd $(1); \ + LC_ALL=C find . -not -path './$(STAGING_SUBDIR)/*' \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).before +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_size_after + cd $(1); \ + LC_ALL=C find . -not -path './$(STAGING_SUBDIR)/*' \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $($(PKG)_DIR)/.files-list$(2).after + LC_ALL=C comm -13 \ + $($(PKG)_DIR)/.files-list$(2).before \ + $($(PKG)_DIR)/.files-list$(2).after \ + | sed -r -e 's/^[^,]+/$($(PKG)_NAME)/' \ + > $($(PKG)_DIR)/.files-list$(2).txt + rm -f $($(PKG)_DIR)/.files-list$(2).before + rm -f $($(PKG)_DIR)/.files-list$(2).after +endef + +define check_bin_arch + support/scripts/check-bin-arch -p $($(PKG)_NAME) \ + -l $($(PKG)_DIR)/.files-list.txt \ + $(foreach i,$($(PKG)_BIN_ARCH_EXCLUDE),-i "$(i)") \ + -r $(TARGET_READELF) \ + -a $(BR2_READELF_ARCH_NAME) +endef + ################################################################################ # Implicit targets -- produce a stamp file for each step of a package build ################################################################################ From patchwork Thu Apr 30 09:52:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280107 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0c4ZQTz9sPF for ; Thu, 30 Apr 2020 19:53:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 24CDD86D28; Thu, 30 Apr 2020 09:53:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oX-MZHYUU4RN; Thu, 30 Apr 2020 09:53:18 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7D31086CC5; Thu, 30 Apr 2020 09:53:18 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 7A4991BF86C for ; Thu, 30 Apr 2020 09:53:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 76165204E4 for ; Thu, 30 Apr 2020 09:53:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZC-LcS8qNGjp for ; Thu, 30 Apr 2020 09:53:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by silver.osuosl.org (Postfix) with ESMTPS id 22DBC204D3 for ; Thu, 30 Apr 2020 09:53:06 +0000 (UTC) Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 3BFBF200008; Thu, 30 Apr 2020 09:53:05 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:45 +0200 Message-Id: <20200430095249.782597-9-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 08/11] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" With per-package directory support, it is absolutely critical that a given package does not overwrite files already installed by another package. However, we currently don't have anything in Buildroot that detects this situation. We used to have check-uniq-files, but it was dropped in commit 2496189a4207173e4cd5bbab90256f911175ee57. This commit is a new implementation of such a detection, which is based on calculating and verifying MD5 hashes of installed files: the calculation is done at the beginning of the configure step, the verification during the newly introduced "install" step that takes place after all installation steps. Due to this calculation and verification having some overhead, it is only enabled when BR2_PER_PACKAGE_DIRECTORIES=y. Note that having BR2_PER_PACKAGE_DIRECTORIES=y also means that on average the HOST_DIR and TARGET_DIR contain less files than on non-per-package build: we only have in HOST_DIR and TARGET_DIR the dependencies of the current package being built. This helps a bit in mitigating the additional cost of this verification. Note that we are not handling separately HOST_DIR and STAGING_DIR, like we're doing with the pkg_size_{before,after} functions. Instead, the verification on HOST_DIR walks down into the STAGING_DIR. Signed-off-by: Thomas Petazzoni --- package/pkg-generic.mk | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 6e06d735ad..82af4afaee 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -102,6 +102,27 @@ define fixup-libtool-files endef endif +# Functions to detect overwritten files + +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_before + cd $(1); \ + LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5 +endef + +# $(1): base directory to search in +# $(2): suffix of file (optional) +define pkg_detect_overwrite_after + cd $(1); \ + if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \ + LC_ALL=C md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \ + { echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \ + fi +endef +endif + # Functions to collect statistics about installed files # $(1): base directory to search in @@ -235,6 +256,8 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call pkg_size_before,$(TARGET_DIR)) @$(call pkg_size_before,$(STAGING_DIR),-staging) @$(call pkg_size_before,$(HOST_DIR),-host) + @$(call pkg_detect_overwrite_before,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_before,$(HOST_DIR),-host) $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) @@ -360,6 +383,8 @@ $(BUILD_DIR)/%/.stamp_installed: @$(call pkg_size_after,$(STAGING_DIR),-staging) @$(call pkg_size_after,$(HOST_DIR),-host) @$(call check_bin_arch) + @$(call pkg_detect_overwrite_after,$(TARGET_DIR)) + @$(call pkg_detect_overwrite_after,$(HOST_DIR),-host) $(Q)touch $@ # Remove package sources From patchwork Thu Apr 30 09:52:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280111 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0v2VXPz9sSd for ; Thu, 30 Apr 2020 19:53:35 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 08BEB24E00; Thu, 30 Apr 2020 09:53:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t040Q6sfqAaV; Thu, 30 Apr 2020 09:53:26 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 6E79B24C01; Thu, 30 Apr 2020 09:53:20 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 104681BF86C for ; Thu, 30 Apr 2020 09:53:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0BCF1204D3 for ; Thu, 30 Apr 2020 09:53:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4gSp3hCux5jD for ; Thu, 30 Apr 2020 09:53:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by silver.osuosl.org (Postfix) with ESMTPS id E9AAD204E4 for ; Thu, 30 Apr 2020 09:53:10 +0000 (UTC) X-Originating-IP: 86.210.146.109 Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id 36CF2FF807; Thu, 30 Apr 2020 09:53:06 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:46 +0200 Message-Id: <20200430095249.782597-10-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 09/11] support/testing/infra: add log_file_path() function X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni , Ricardo Martincoski Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Some tests will need to grep through the build log to verify that some features are working are expected. In order to allow them to open the build log, we provide a new function called log_file_path(), which returns the path to the log file if available. We also use this function in open_log_file(). Note that open_log_file() cannot be used directly to grep through the log file at the end of a build: because it opens in "a+" mode, it greps starting from the end of the file. Signed-off-by: Thomas Petazzoni --- support/testing/infra/__init__.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py index 6392aa679b..aaee055ee6 100644 --- a/support/testing/infra/__init__.py +++ b/support/testing/infra/__init__.py @@ -10,14 +10,23 @@ ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/" BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../..")) +def log_file_path(builddir, stage, logtofile=True): + """Return path to log file""" + if logtofile: + return "{}-{}.log".format(builddir, stage) + else: + return None + + def open_log_file(builddir, stage, logtofile=True): """ Open a file for logging and return its handler. If logtofile is True, returns sys.stdout. Otherwise opens a file with a suitable name in the build directory. """ - if logtofile: - fhandle = open("{}-{}.log".format(builddir, stage), 'a+') + logf = log_file_path(builddir, stage, logtofile) + if logf: + fhandle = open(logf, 'a+') else: fhandle = sys.stdout return fhandle From patchwork Thu Apr 30 09:52:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280108 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0f66fxz9sPF for ; Thu, 30 Apr 2020 19:53:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 4B8D887E38; Thu, 30 Apr 2020 09:53:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vzBgEC-56Qdx; Thu, 30 Apr 2020 09:53:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 7C40187E1F; Thu, 30 Apr 2020 09:53:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 93CC51BF86C for ; Thu, 30 Apr 2020 09:53:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8FEC9204F1 for ; Thu, 30 Apr 2020 09:53:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tgjkNOhNpjTU for ; Thu, 30 Apr 2020 09:53:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by silver.osuosl.org (Postfix) with ESMTPS id 53C84204D3 for ; Thu, 30 Apr 2020 09:53:10 +0000 (UTC) Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 8971E200010; Thu, 30 Apr 2020 09:53:08 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:47 +0200 Message-Id: <20200430095249.782597-11-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 10/11] support/testing/tests: add test for check_bin_arch X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This tests build a bogus package that installs a binary built for the host architecture into $(TARGET_DIR), which should cause a build failure, at least as long as the host architecture isn't ARM. Signed-off-by: Thomas Petazzoni --- .gitlab-ci.yml | 1 + .../br2-external/detect-bad-arch/Config.in | 1 + .../detect-bad-arch/external.desc | 1 + .../br2-external/detect-bad-arch/external.mk | 1 + .../package/detect-bad-arch/Config.in | 4 ++++ .../detect-bad-arch/detect-bad-arch.mk | 15 +++++++++++++++ support/testing/tests/core/test_bad_arch.py | 19 +++++++++++++++++++ 7 files changed, 42 insertions(+) create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk create mode 100644 support/testing/tests/core/test_bad_arch.py diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index fa8e077a07..dd69fb9d50 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -354,6 +354,7 @@ tests.boot.test_syslinux.TestSysLinuxX86EFI: { extends: .runtime_test } tests.boot.test_syslinux.TestSysLinuxX86LegacyBios: { extends: .runtime_test } tests.boot.test_syslinux.TestSysLinuxX86_64EFI: { extends: .runtime_test } tests.boot.test_syslinux.TestSysLinuxX86_64LegacyBios: { extends: .runtime_test } +tests.core.test_bad_arch.DetectBadArchTest: { extends: .runtime_test } tests.core.test_file_capabilities.TestFileCapabilities: { extends: .runtime_test } tests.core.test_hardening.TestFortifyConserv: { extends: .runtime_test } tests.core.test_hardening.TestFortifyNone: { extends: .runtime_test } diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/Config.in b/support/testing/tests/core/br2-external/detect-bad-arch/Config.in new file mode 100644 index 0000000000..530c077bbe --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-bad-arch/Config.in @@ -0,0 +1 @@ +source "$BR2_EXTERNAL_DETECT_BAD_ARCH_PATH/package/detect-bad-arch/Config.in" diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/external.desc b/support/testing/tests/core/br2-external/detect-bad-arch/external.desc new file mode 100644 index 0000000000..3c4232c90d --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-bad-arch/external.desc @@ -0,0 +1 @@ +name: DETECT_BAD_ARCH diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/external.mk b/support/testing/tests/core/br2-external/detect-bad-arch/external.mk new file mode 100644 index 0000000000..71b9821ddc --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-bad-arch/external.mk @@ -0,0 +1 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_BAD_ARCH_PATH)/package/*/*.mk)) diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in new file mode 100644 index 0000000000..9893e9afc1 --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in @@ -0,0 +1,4 @@ +config BR2_PACKAGE_DETECT_BAD_ARCH + bool + default y + diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk new file mode 100644 index 0000000000..5e78c55f1f --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk @@ -0,0 +1,15 @@ +################################################################################ +# +# detect-bad-arch +# +################################################################################ + +define DETECT_BAD_ARCH_BUILD_CMDS + echo "int main(void) { return 0; }" | $(HOSTCC) -x c -o $(@D)/foo - +endef + +define DETECT_BAD_ARCH_INSTALL_TARGET_CMDS + $(INSTALL) -D -m 0755 $(@D)/foo $(TARGET_DIR)/usr/bin/foo +endef + +$(eval $(generic-package)) diff --git a/support/testing/tests/core/test_bad_arch.py b/support/testing/tests/core/test_bad_arch.py new file mode 100644 index 0000000000..8f4bd57b0e --- /dev/null +++ b/support/testing/tests/core/test_bad_arch.py @@ -0,0 +1,19 @@ +import infra +import infra.basetest +import subprocess + + +class DetectBadArchTest(infra.basetest.BRConfigTest): + config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + infra.basetest.MINIMAL_CONFIG + br2_external = [infra.filepath("tests/core/br2-external/detect-bad-arch")] + + def test_run(self): + with self.assertRaises(SystemError): + self.b.build() + logf_path = infra.log_file_path(self.b.builddir, "build", + infra.basetest.BRConfigTest.logtofile) + if logf_path: + s = 'ERROR: architecture for "/usr/bin/foo" is "Advanced Micro Devices X86-64", should be "ARM"' + logf = open(logf_path, "r") + ret = subprocess.call(["grep", "-q", s], stdin=logf) + self.assertEqual(ret, 0) From patchwork Thu Apr 30 09:52:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1280110 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49CW0q1LgBz9sPF for ; Thu, 30 Apr 2020 19:53:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id AA90586D17; Thu, 30 Apr 2020 09:53:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WEGYn_Wgd4As; Thu, 30 Apr 2020 09:53:28 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7296686D02; Thu, 30 Apr 2020 09:53:28 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 207981BF86C for ; Thu, 30 Apr 2020 09:53:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1D44788688 for ; Thu, 30 Apr 2020 09:53:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7soxswpoHkSE for ; Thu, 30 Apr 2020 09:53:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by hemlock.osuosl.org (Postfix) with ESMTPS id 9F3C88867F for ; Thu, 30 Apr 2020 09:53:12 +0000 (UTC) X-Originating-IP: 86.210.146.109 Received: from localhost (lfbn-tou-1-915-109.w86-210.abo.wanadoo.fr [86.210.146.109]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id C92C7FF80A; Thu, 30 Apr 2020 09:53:09 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Thu, 30 Apr 2020 11:52:48 +0200 Message-Id: <20200430095249.782597-12-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.25.4 In-Reply-To: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> References: <20200430095249.782597-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 11/11] support/testing/tests: add test for file overwrite detection X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , "Yann E. MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Thomas Petazzoni --- .gitlab-ci.yml | 2 + .../br2-external/detect-overwrite/Config.in | 1 + .../detect-overwrite/external.desc | 1 + .../br2-external/detect-overwrite/external.mk | 1 + .../package/detect-overwrite/Config.in | 5 ++ .../detect-overwrite/detect-overwrite.mk | 19 ++++++++ .../testing/tests/core/test_file_overwrite.py | 47 +++++++++++++++++++ 7 files changed, 76 insertions(+) create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk create mode 100644 support/testing/tests/core/test_file_overwrite.py diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index dd69fb9d50..64266d8783 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -356,6 +356,8 @@ tests.boot.test_syslinux.TestSysLinuxX86_64EFI: { extends: .runtime_test } tests.boot.test_syslinux.TestSysLinuxX86_64LegacyBios: { extends: .runtime_test } tests.core.test_bad_arch.DetectBadArchTest: { extends: .runtime_test } tests.core.test_file_capabilities.TestFileCapabilities: { extends: .runtime_test } +tests.core.test_file_overwrite.DetectHostFileOverwriteTest: { extends: .runtime_test } +tests.core.test_file_overwrite.DetectTargetFileOverwriteTest: { extends: .runtime_test } tests.core.test_hardening.TestFortifyConserv: { extends: .runtime_test } tests.core.test_hardening.TestFortifyNone: { extends: .runtime_test } tests.core.test_hardening.TestRelro: { extends: .runtime_test } diff --git a/support/testing/tests/core/br2-external/detect-overwrite/Config.in b/support/testing/tests/core/br2-external/detect-overwrite/Config.in new file mode 100644 index 0000000000..b5514510bd --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-overwrite/Config.in @@ -0,0 +1 @@ +source "$BR2_EXTERNAL_DETECT_OVERWRITE_PATH/package/detect-overwrite/Config.in" diff --git a/support/testing/tests/core/br2-external/detect-overwrite/external.desc b/support/testing/tests/core/br2-external/detect-overwrite/external.desc new file mode 100644 index 0000000000..6fedc276e8 --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-overwrite/external.desc @@ -0,0 +1 @@ +name: DETECT_OVERWRITE diff --git a/support/testing/tests/core/br2-external/detect-overwrite/external.mk b/support/testing/tests/core/br2-external/detect-overwrite/external.mk new file mode 100644 index 0000000000..90927b33ef --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-overwrite/external.mk @@ -0,0 +1 @@ +include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_OVERWRITE_PATH)/package/*/*.mk)) diff --git a/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in new file mode 100644 index 0000000000..fff8b0320f --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in @@ -0,0 +1,5 @@ +config BR2_PACKAGE_DETECT_OVERWRITE + bool "detect-overwrite" + +config BR2_PACKAGE_HOST_DETECT_OVERWRITE + bool "host-detect-overwrite" diff --git a/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk new file mode 100644 index 0000000000..c6df2a339d --- /dev/null +++ b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk @@ -0,0 +1,19 @@ +################################################################################ +# +# detect-overwrite +# +################################################################################ + +define DETECT_OVERWRITE_INSTALL_TARGET_CMDS + grep -q "^foo" $(TARGET_DIR)/etc/passwd || \ + echo "foo" >> $(TARGET_DIR)/etc/passwd +endef + +HOST_DETECT_OVERWRITE_DEPENDENCIES = host-pkgconf + +define HOST_DETECT_OVERWRITE_INSTALL_CMDS + $(SED) 's/manipulating/tweaking/' $(HOST_DIR)/lib/pkgconfig/libpkgconf.pc +endef + +$(eval $(generic-package)) +$(eval $(host-generic-package)) diff --git a/support/testing/tests/core/test_file_overwrite.py b/support/testing/tests/core/test_file_overwrite.py new file mode 100644 index 0000000000..526ed55e43 --- /dev/null +++ b/support/testing/tests/core/test_file_overwrite.py @@ -0,0 +1,47 @@ +import infra +import infra.basetest +import subprocess + + +class DetectTargetFileOverwriteTest(infra.basetest.BRConfigTest): + config = \ + infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ + infra.basetest.MINIMAL_CONFIG + \ + """ + BR2_PER_PACKAGE_DIRECTORIES=y + BR2_PACKAGE_DETECT_OVERWRITE=y + """ + br2_external = [infra.filepath("tests/core/br2-external/detect-overwrite")] + + def test_run(self): + with self.assertRaises(SystemError): + self.b.build() + logf_path = infra.log_file_path(self.b.builddir, "build", + infra.basetest.BRConfigTest.logtofile) + if logf_path: + s = './etc/passwd: FAILED' + logf = open(logf_path, "r") + ret = subprocess.call(["grep", "-q", s], stdin=logf) + self.assertEqual(ret, 0) + + +class DetectHostFileOverwriteTest(infra.basetest.BRConfigTest): + config = \ + infra.basetest.BASIC_TOOLCHAIN_CONFIG + \ + infra.basetest.MINIMAL_CONFIG + \ + """ + BR2_PER_PACKAGE_DIRECTORIES=y + BR2_PACKAGE_HOST_DETECT_OVERWRITE=y + """ + br2_external = [infra.filepath("tests/core/br2-external/detect-overwrite")] + + def test_run(self): + with self.assertRaises(SystemError): + self.b.build() + logf_path = infra.log_file_path(self.b.builddir, "build", + infra.basetest.BRConfigTest.logtofile) + if logf_path: + s = './lib/pkgconfig/libpkgconf.pc: FAILED' + logf = open(logf_path, "r") + ret = subprocess.call(["grep", "-q", s], stdin=logf) + self.assertEqual(ret, 0)