From patchwork Mon Jan 7 22:05:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1021632 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="qZ6hAqEV"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43YTyY45TVz9sDT for ; Tue, 8 Jan 2019 09:07:17 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id D72A6248F6; Mon, 7 Jan 2019 22:07:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OVRFOH7heZFu; Mon, 7 Jan 2019 22:07:13 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id D4D2A2441E; Mon, 7 Jan 2019 22:07:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id E55D51C3353 for ; Mon, 7 Jan 2019 22:06:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 81B9985F7E for ; Mon, 7 Jan 2019 22:06:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G-n-JV8OWhW8 for ; Mon, 7 Jan 2019 22:06:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 55A4B85F83 for ; Mon, 7 Jan 2019 22:06:15 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id b3so2385872ede.1 for ; Mon, 07 Jan 2019 14:06:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=1HubKRGWlhVqTsVbfs5gOWZlVwuV172UiWvIJXQQRNo=; b=qZ6hAqEVleTnk3B2fdTT/Dj1RWWFYW3MLwZ35Ad7ZR1tOxmuQd7h9N4+7Z9iRmcx8C mOiRRGnNWX0qe42GbcdAJlhAK94M0qUwqNF569rk0XWx2BHFxEWjfdfpAMhxK6Vo6GYZ jebjb2yKincY9Iqv3qFEDqZOyFfP37mZcfwyTy40nmkrUnT5C4gICsg9x5UqnlpMyE2d C2T406RtcdJ3G7U3e6FBON6Ezq2f4u8w0I3m112i91v3Ict90THigVPZdc6BUYj6ppfN xv+Vd1b/I3eKcPd/D7WR82oKwK4m7RrOHCyj6MPz05pN+5zsJ/PJCn2Xm7AIjdg57Ysi FLWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=1HubKRGWlhVqTsVbfs5gOWZlVwuV172UiWvIJXQQRNo=; b=lLnYaKtJzQN47Amt91EkQJjTdnlkG5dFYwOPUko15cIsyKg6FzHoYzy2zp3q21zbfh ZB6UMqh9Rti4YUopBCVLvc3h5/MiUd0xkNeepUJnxAfILbu13UR/rQCnvbYkSkWg9NdH /zoiRdnU9JWDxz0KgC6s6Zod/qNDKNiJkmHQuakVDR+J54wu64DF5Ej4812HLHfAV9l9 NQo56ih9++sqzPEH15Yc7p/FWD/DiNPx1ydRKCdCtXM9rut60mzQcCTloaS9fx7cdnkx l5qsmtopp5ixWBHsAAteg+jmSJ6vveyh7TnsaYhvSUfgS7FexvorAwPM/ipl1v6pLfcF BDhg== X-Gm-Message-State: AA+aEWbj+XhQZd4BElsfHDommAaVUB75UUgcO+LPUv+US9uO3XU1n/iE 63EtFz4hOnsDWzgL7kjuVXDJTSRv X-Google-Smtp-Source: AFSGD/U52vp9plQ+h9vQjL4vyC0eGfkDedy/brPHfo8xYGrvsci/LGMfNSLReoWWZB3F7JKzeWpBFQ== X-Received: by 2002:a17:906:655:: with SMTP id t21-v6mr47499715ejb.81.1546898773471; Mon, 07 Jan 2019 14:06:13 -0800 (PST) Received: from scaer.home ([2a01:cb19:829a:2800:68e8:7a61:9bb9:12a]) by smtp.gmail.com with ESMTPSA id d56sm31799589ede.76.2019.01.07.14.06.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Jan 2019 14:06:12 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Mon, 7 Jan 2019 23:05:41 +0100 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 19/19] core: add optional failure when 2+ packages touch the same file X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , "Yann E. MORIN" MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" For top-level parallel build, we will really want to avoid the situation where two packages touch the same file. The two problematic situations being: - different packages install different content for a file, so we would not know which to provide: currently build ordering guarantees the content to be reproducible (i.e. last wins), but with parallel builds, we may lose such a guarantee; - pacakge update an existing file, e.g. by registering new content to it: currently, build ordering again guarantees that the file is properly expanded by successive packages, but with top-level parallel build, that would not longer be the case. Consequently, we can't accept that two packages touch the same file. We currently only warn about the situation, but we have no real hard numbers about how many packages are affected and cause the issue. Add a user-settable option (mostly for the autobuilders for now) to turn the warning into a hard error. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni --- Config.in | 8 ++++++++ Makefile | 10 +++++++--- support/scripts/check-uniq-files | 12 ++++++++++-- 3 files changed, 25 insertions(+), 5 deletions(-) diff --git a/Config.in b/Config.in index f965e9d6d8..d424ea6b0b 100644 --- a/Config.in +++ b/Config.in @@ -677,6 +677,14 @@ config BR2_COMPILER_PARANOID_UNSAFE_PATH and external toolchain backends (through the toolchain wrapper). +config BR2_UNIQ_FILES + bool "Forbid two packages from touching the same files" + help + By default, Buildroot will detect and warn when two packages + (or more) touch the same files. + + Say 'y' here to turn the warning into an error. + config BR2_REPRODUCIBLE bool "Make the build reproducible (experimental)" # SOURCE_DATE_EPOCH support in toolchain-wrapper requires GCC 4.4 diff --git a/Makefile b/Makefile index cc9ac91647..cc135e0ad3 100644 --- a/Makefile +++ b/Makefile @@ -703,6 +703,10 @@ endef TARGET_FINALIZE_HOOKS += PURGE_LOCALES endif +ifeq ($(BR2_UNIQ_FILES),y) +UNIQ_FILES_OPTS = --fail +endif + $(TARGETS_ROOTFS): target-finalize # Avoid the rootfs name leaking down the dependency chain @@ -712,7 +716,7 @@ target-finalize: ROOTFS= host-finalize: $(HOST_DIR_SYMLINK) # Check files that are touched by more than one package ./support/scripts/check-uniq-files \ - -t host -d $(HOST_DIR) \ + -t host -d $(HOST_DIR) $(UNIQ_FILES_OPTS) \ $(BUILD_DIR)/packages-file-list-host.txt .PHONY: staging-finalize @@ -720,7 +724,7 @@ staging-finalize: @ln -snf $(STAGING_DIR) $(BASE_DIR)/staging # Check files that are touched by more than one package ./support/scripts/check-uniq-files \ - -t staging -d $(STAGING_DIR) \ + -t staging -d $(STAGING_DIR) $(UNIQ_FILES_OPTS) \ $(BUILD_DIR)/packages-file-list-staging.txt .PHONY: target-finalize @@ -790,7 +794,7 @@ endif # Check files that are touched by more than one package ./support/scripts/check-uniq-files \ - -t target -d $(TARGET_DIR) \ + -t target -d $(TARGET_DIR) $(UNIQ_FILES_OPTS) \ $(BUILD_DIR)/packages-file-list.txt touch $(TARGET_DIR)/usr diff --git a/support/scripts/check-uniq-files b/support/scripts/check-uniq-files index 3b33626c73..4507cda4a8 100755 --- a/support/scripts/check-uniq-files +++ b/support/scripts/check-uniq-files @@ -6,7 +6,7 @@ import os.path from collections import defaultdict from brpkgutil import parse_pkg_file_list as parse_pkg_file_list -warn = 'Warning: {0} file "{1}" is touched by more than one package: {2}\n' +warn = '{0}: {1} file "{2}" is touched by more than one package: {3}\n' # If possible, try to decode the binary string s with the user's locale. @@ -27,6 +27,8 @@ def main(): help='Report as a TYPE file (TYPE is either target, staging, or host)') parser.add_argument('-d', '--dir', metavar="DIR", required=True, help='Directory used as base (target/, staging/ or host/))') + parser.add_argument('-f', '--fail', action='store_true', + help='Fail if a file is touched by more than one package') args = parser.parse_args() @@ -36,6 +38,7 @@ def main(): file_to_pkg[record['file']].add(record['pkg']) file_md5[record['file']].add(record['md5']) + ret = 0 for file in file_to_pkg: if len(file_to_pkg[file]) == 1: continue @@ -46,10 +49,15 @@ def main(): if not os.path.exists(os.path.join(args.dir, file)): continue - sys.stderr.write(warn.format(args.type, str_decode(file), + ret = 1 if args.fail else ret + sys.stderr.write(warn.format("Error" if args.fail else "Warning", + args.type, + str_decode(file), ", ".join([str_decode(p) for p in file_to_pkg[file]]))) + return ret + if __name__ == "__main__": sys.exit(main())