From patchwork Sun Oct 22 15:15:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 829057 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nMIP30pD"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yKjm84frhz9sRm for ; Mon, 23 Oct 2017 02:16:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id B9CF38782E; Sun, 22 Oct 2017 15:16:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xgAnMwU6W1Z5; Sun, 22 Oct 2017 15:16:02 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 72B9A87817; Sun, 22 Oct 2017 15:16:02 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id C4FF71BFDB4 for ; Sun, 22 Oct 2017 15:15:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C03EB860FD for ; Sun, 22 Oct 2017 15:15:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2374yUwPULXu for ; Sun, 22 Oct 2017 15:15:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f50.google.com (mail-wm0-f50.google.com [74.125.82.50]) by fraxinus.osuosl.org (Postfix) with ESMTPS id C8A6A85FDB for ; Sun, 22 Oct 2017 15:15:58 +0000 (UTC) Received: by mail-wm0-f50.google.com with SMTP id m72so5300820wmc.1 for ; Sun, 22 Oct 2017 08:15:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=E4PLPKKSOPI8UU4w5jCOvK8jIkpVeMFnhlgoeYl38P4=; b=nMIP30pDRGwCbbxYFcfET5xCD1PG1sYZ5IdulCLWc40ryzW8renMik3doZ79N/kuMc gKInnHj5yDbe8zzxRSeinD3l3Op3dtmJtTls1wYXuOHwUXFPKkrQ63MJzTsM7YCteEtT 6WMMohdLWKdsVIGKkbwXCyCkcWGczuFbWPh2m/w3E5UfQK8fntiJhsuWYqtm02+dROlN h244OfcdzArlegoB3MkI3AFAIn7NXZ6xZVhnKlXb/8IN1q3KbftBxaKBQRpiEL7EeMFw a/WStgannY8libgSHkef0ZNotIx3zKmY8ByhySZRcNqKsh2SO1ZVPXySPJrSedAY0HuE 9p3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=E4PLPKKSOPI8UU4w5jCOvK8jIkpVeMFnhlgoeYl38P4=; b=EL7cGueaEmUAqRD3q4icCBjN1BYvN5oQtL8FreiGwtnKBVhJK6XH59pwSYMPvl2Em7 nBWvrDFyWm1bUyqcUUomQqQVRPZ7os61t2wch8m5oGtZTjYX3n4W1u6Aa4BxKM1SgjYk ZeTD6Nz03UNQATWzFVhnMQ+xG+tR44A6JZGFf+FfSyCal3MCFH20XFJpEOLG2lSrDxkB k2Y7rgE5+ZcOVzWEe9rL6Ydg8a8LXkIqRp+GPeev0vBfvNp7cDepHV8H3YXbeEQPJgTv 4NAH7Lb7Vm1YMvTPcaOIG0+jzdDGJnelZdAKDapuYhXzkdHPrArqUt5anDOx0ufeor3i lJ8A== X-Gm-Message-State: AMCzsaVmteudnT9BOwv8QgJWmgEPSOz5Efv4PBJDY8gAeIk3EG1FKwdS 6AK6/LPPEx+ZTXoXJgf6/yHpzg== X-Google-Smtp-Source: ABhQp+SZupmZxpQRN8sWv9tt3jZxq9D0EmjFiswlR9R6iPEO9b9PVhdzwo2MNFz36Cl+bYUOYMhhTg== X-Received: by 10.28.91.194 with SMTP id p185mr3041230wmb.84.1508685357068; Sun, 22 Oct 2017 08:15:57 -0700 (PDT) Received: from localhost.localdomain (ns304657.ip-46-105-103.eu. [46.105.103.66]) by smtp.gmail.com with ESMTPSA id w29sm5489708wra.52.2017.10.22.08.15.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 22 Oct 2017 08:15:56 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Sun, 22 Oct 2017 17:15:40 +0200 Message-Id: <64a5b7666e47d87d58845028f9576ab68a186f44.1508685316.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.11.0 In-Reply-To: References: Cc: Thomas Petazzoni , "Yann E. MORIN" Subject: [Buildroot] [PATCH 4/4 v2] core: check files are nto touched by more than on package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Currently, we do nothing about packages that touch the same file: given a specific configuration, the result is reproducible (even though it might not be what the user expected) because the build order is guaranteed. Hwever, when we later introduce top-level parallel buil, we will no longer be able to guarantee a build order, by the mere way of it being parallel. Reconcialliting all those modified files will be impossible to do automatically. The only way will be to refuse such situations. As a preliminary step, introduce a helper script that detects files that are being moified by two or more packages, and reports them and the impacted packages, at the end of the build. The list being reported at the end of the build will make it prominently visible in autobuilder results, so we can assess the problem, if any. Later on, calling that helper script can be done right after the package installation step, to bail out early. Thanks Arnout for the pythonist way to write default dictionaries! ;-) Note: doing it in python rather than a shell script is impressively faster: where the shell script takes ~1.2s on a minimalist build, the python script only takes ~0.015s, that is about 80 times faster. Signed-off-by: "Yann E. MORIN" Cc: Arnout Vandecappelle Cc: Thomas Petazzoni Cc: Peter Korsgaard --- Makefile | 4 ++++ support/scripts/check-uniq-files.py | 40 +++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) create mode 100755 support/scripts/check-uniq-files.py diff --git a/Makefile b/Makefile index 79db7fe48a..f0b60bae5c 100644 --- a/Makefile +++ b/Makefile @@ -679,6 +679,10 @@ $(TARGETS_ROOTFS): target-finalize .PHONY: target-finalize target-finalize: $(PACKAGES) @$(call MESSAGE,"Finalizing target directory") + # Check files that are touched by more than one package + ./support/scripts/check-uniq-files.py -t target $(BUILD_DIR)/packages-file-list.txt + ./support/scripts/check-uniq-files.py -t staging $(BUILD_DIR)/packages-file-list-staging.txt + ./support/scripts/check-uniq-files.py -t host $(BUILD_DIR)/packages-file-list-host.txt $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ $(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \ diff --git a/support/scripts/check-uniq-files.py b/support/scripts/check-uniq-files.py new file mode 100755 index 0000000000..e77472cf80 --- /dev/null +++ b/support/scripts/check-uniq-files.py @@ -0,0 +1,40 @@ +#!/usr/bin/env python2 + +import sys +import csv +import argparse +from collections import defaultdict + +warn = 'Warnng: {}s file "{}" is touched by more than one package: {}\n' + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument('packages_file_list', nargs='*', + help='The packages-file-list to check from') + parser.add_argument('-t', '--type', metavar="TYPE", + help='Report as a TYPE file (TYPE is either target, staging, or host)') + + args = parser.parse_args() + + if not len(args.packages_file_list) == 1: + sys.stderr.write('No packages-file-list was provided.\n') + return False + + if args.type is None: + sys.stderr.write('No type was provided\n') + return False + + file_to_pkg = defaultdict(list) + with open(args.packages_file_list[0], 'rb') as pkg_file_list: + r = csv.reader(pkg_file_list, delimiter=',') + for row in r: + pkg = row[0] + file = row[1] + file_to_pkg[file].append(pkg) + + for file in file_to_pkg: + if len(file_to_pkg[file]) > 1: + sys.stderr.write(warn.format(args.type, file, file_to_pkg[file])) + +if __name__ == "__main__": + sys.exit(main())