From patchwork Fri Feb 14 19:57:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1238338 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=fs3IrigU; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48K40v01qnz9sT6 for ; Sat, 15 Feb 2020 06:57:33 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 2135A2272C; Fri, 14 Feb 2020 19:57:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dKsIJ54ChrBz; Fri, 14 Feb 2020 19:57:29 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 78B922269B; Fri, 14 Feb 2020 19:57:29 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 7A1381BF2C8 for ; Fri, 14 Feb 2020 19:57:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 71942882DD for ; Fri, 14 Feb 2020 19:57:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id riUq5vO+W8fU for ; Fri, 14 Feb 2020 19:57:26 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-oi1-f193.google.com (mail-oi1-f193.google.com [209.85.167.193]) by hemlock.osuosl.org (Postfix) with ESMTPS id 999AD882F7 for ; Fri, 14 Feb 2020 19:57:26 +0000 (UTC) Received: by mail-oi1-f193.google.com with SMTP id p125so10537245oif.10 for ; Fri, 14 Feb 2020 11:57:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XCwoCAlIgAa3kb0D84fFhfyjs2ibfVUWwxj2ClegwMk=; b=fs3IrigUw0PYfEbr8pQHt0G7jLbOy/73kDqHHKFjjwdwAdGlUGPWHMq3I8LcayW4Du KX9Z0LKAy5Nb7OsVSUiyf3OvIhoxv0G4ajkDd48MlOcPFVTpQBxQ6vusxKPmhH12tYDw /h7izoSi600YID2XNrvzfT8kUJZCXbrcOHVj+44nIsJLRn0q4RucG6OHGyc4sxMcUPUc rMK0CJfGL9Fo2JIkGpA8jWcIkHGgISPTiCM7+rGh6B6irg7McCJeWikTej4WJW4D7sSm sCUHMLmjcHAiZDbYhgGFwsG2rVEWVyPg2xdo5WaWNhQk+rNDG3K5LfUJPUe+tByJ0sST ikFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=XCwoCAlIgAa3kb0D84fFhfyjs2ibfVUWwxj2ClegwMk=; b=p2eN7soNtdyqTEDX+EVJ+ntQgBqDEnLpvFUZ9+KuGdBm7owDhULfgvTGDSEYpK+Hli xcNlVN1kalbexVl2ypOJVCaGGOddtVs8Z1yeOKYnA93hux3aTl8VCPaQXRqZ+9HrmH77 YhzfECaZJbM7HV+sp6m2LZBzrA1k/xtfFoni2O0iiSMAIfi5p7oxDfjxX8jHnGopWx16 QRr+VGIijmvw9R7JnCtpyjUUagN5g6zSPIMenYMn/T+3FxK9SHDYWa4/gy+yB9/US43/ tak0dOE9Ek+xbraUsmOVZXY8LFtFckJXhuTklG6PQwsTtew7qAa+YM3AvDDlonNpDUJM QRag== X-Gm-Message-State: APjAAAVDt2aYZMbZ45HCK4a+osb+sO0ST27k69Lno6k06r3wAMy+dJxb cn6NmJbVX5tD24fRbT1qWt/pgeQ6Wpc= X-Google-Smtp-Source: APXvYqxlpFZ5ZR4EaIxye7OG9a+6XRGhgqUp+2Xu+hrzu0Wwq8R0/BN4pF6lHSnT851dGyd2N02+fA== X-Received: by 2002:aca:1204:: with SMTP id 4mr3048494ois.143.1581710245521; Fri, 14 Feb 2020 11:57:25 -0800 (PST) Received: from localhost ([131.228.32.167]) by smtp.gmail.com with ESMTPSA id y22sm1472680otq.31.2020.02.14.11.57.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 Feb 2020 11:57:25 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Fri, 14 Feb 2020 20:57:33 +0100 Message-Id: <20200214195734.32081-1-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Subject: [Buildroot] [PATCH] core: fix packages-file-list.txt after an incremental build X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire The package instrumentation step 'step_pkg_size' is populating the files: output/build/packages-file-list.txt output/build/packages-file-list-staging.txt output/build/packages-file-list-host.txt by comparing the list of files before and after installation of a package, with some clever tricks to detect changes to existing files etc. As an optimization, instead of gathering this list before and after each package, where the 'after-state' of one package is the same as the 'before-state' of the next package, only the 'after-state' is used and is shared between packages. This works fine, except at the end of the build, as explained next. In the target-finalize step, many files will be touched. For example, files like /etc/hosts, /etc/os-release, but also all object files that are stripped, and all files touched by post-build scripts or created by rootfs overlays. This means that the 'after-state' of the last package does not reflect the actual situation after target-finalize is run. For a single complete build this poses no problem. But, if one incrementally rebuilds a package after the initial build, e.g. with 'make foo-rebuild', then all changes that happened in target-finalize at the end of the initial build (the 'after-state' of the last package built) will be detected as changes caused by the rebuild of package foo. As a result, all these files will incorrectly be treated as 'owned' by package foo. Correct this situation by capturing a new state at the end of target-finalize, so that the 'before-state' of an incremental build will be correct. Note: the reasoning above talks about packages-file-list.txt and target-finalize, but also applies to packages-file-list-staging.txt/staging-finalize and packages-file-list-host.txt/host-finalize. Signed-off-by: Thomas De Schampheleire --- Makefile | 10 ++++++++++ package/pkg-generic.mk | 23 ++++++++++++++++++----- 2 files changed, 28 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index a52f1c75fd..af24630d2f 100644 --- a/Makefile +++ b/Makefile @@ -801,6 +801,16 @@ endif # merged /usr touch $(TARGET_DIR)/usr +# AFTER ALL FILE-CHANGING ACTIONS: +# Update timestamps in internal file list to fix attribution of files +# to packages on subsequent builds + $(call step_pkg_size_file_list,$(TARGET_DIR)) + $(call step_pkg_size_finalize) + $(call step_pkg_size_file_list,$(STAGING_DIR),-staging) + $(call step_pkg_size_finalize,-staging) + $(call step_pkg_size_file_list,$(HOST_DIR),-host) + $(call step_pkg_size_finalize,-host) + .PHONY: target-post-image target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize @rm -f $(ROOTFS_COMMON_TAR) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 268d999efb..2f11e3903f 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -57,6 +57,22 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time # Hooks to collect statistics about installed files +# Helper function to create the file list -- also used from target-finalize +# $(1): base directory to search in +# $(2): suffix of file (optional) +define step_pkg_size_file_list + cd $(1); \ + LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ + | LC_ALL=C sort > $(BUILD_DIR)/.files-list$(2).new +endef +# Helper function to mark the latest file list as the reference for next +# iteration -- also used from target-finalize +# $(1): suffix of file (optional) +define step_pkg_size_finalize + mv $(BUILD_DIR)/.files-list$(1).new \ + $(BUILD_DIR)/.files-list$(1).stat +endef + # The suffix is typically empty for the target variant, for legacy backward # compatibility. # $(1): package name @@ -66,9 +82,7 @@ define step_pkg_size_inner @touch $(BUILD_DIR)/.files-list$(3).stat @touch $(BUILD_DIR)/packages-file-list$(3).txt $(SED) '/^$(1),/d' $(BUILD_DIR)/packages-file-list$(3).txt - cd $(2); \ - LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \ - | LC_ALL=C sort > $(BUILD_DIR)/.files-list$(3).new + $(call step_pkg_size_file_list,$(2),$(3)) LC_ALL=C comm -13 \ $(BUILD_DIR)/.files-list$(3).stat \ $(BUILD_DIR)/.files-list$(3).new \ @@ -76,8 +90,7 @@ define step_pkg_size_inner sed -r -e 's/^[^,]+/$(1)/' \ $($(PKG)_BUILDDIR)/.files-list$(3).txt \ >> $(BUILD_DIR)/packages-file-list$(3).txt - mv $(BUILD_DIR)/.files-list$(3).new \ - $(BUILD_DIR)/.files-list$(3).stat + $(call step_pkg_size_finalize,$(3)) endef define step_pkg_size