From patchwork Sat Aug 10 03:58:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1145050 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HAmqw/L9"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4657f460lZz9sMr for ; Sat, 10 Aug 2019 13:59:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 87D2D8573E; Sat, 10 Aug 2019 03:59:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8RLkqcitDM6V; Sat, 10 Aug 2019 03:59:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 72D4B84B82; Sat, 10 Aug 2019 03:59:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 9E6021BF20D for ; Sat, 10 Aug 2019 03:59:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9B9472039F for ; Sat, 10 Aug 2019 03:59:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AOawub8S9Xm6 for ; Sat, 10 Aug 2019 03:59:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by silver.osuosl.org (Postfix) with ESMTPS id B2E5220391 for ; Sat, 10 Aug 2019 03:59:11 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id m30so47060259pff.8 for ; Fri, 09 Aug 2019 20:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=L67q3gtt92rlVoOLry+6ElPmQ3G1q0Vq73NY2+bHIDY=; b=HAmqw/L91LJCIXIXKgCY5OdFZFxOglxAH/5G3Y5XBmXpdg5rfOxH2Brzt4oFVJ702r bpDjmTp3JXJ5bgj9NnOXeeceHhOwPPgOZsxWe2kl7S6XYZ4DMmcB5GEkQIUdolWeWa/f QcMv5tPijXE6dWSI5M8hqe9tWmmILwvd6R1rOFEf56HKQcUQdEG0n50AtrvseQBs8zdr mxSyY2tcllPnIYEVsCnreLxER6lEdKXrRSQyb25JZfGmK0Z+44u/i91KiZwiYaqoHUyY /AbzzdT8qdvazZ2IYt57OjhwoUGr5/1vkegwlchr6lzxA4f433uQ9Vovg47H+ILufvn/ pdWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=L67q3gtt92rlVoOLry+6ElPmQ3G1q0Vq73NY2+bHIDY=; b=UmsKscJYi2/hrcsqneNVy/hNjRCNIaFwi2k1VO14yXxtcdiA3f3jRhrpCPxtFwgeEj uf5FdpX/SyUkaTcorXyMrzHTjSi04YM4YEwCx5LeK3135vx6zdNPfJ7EegySROb2niMs VM391xODyIkEZLEef0GHN2yC4agtnOZD2bFzZJoabwDqVItX6VjZGhztcWKYsEF5p3TB D0atjUwOHrlNQQqjvTvXRvQrHmhP7gRjgryqUYchx1YT1Qwn7xPIjekXGfm8r2zWjeF+ bo2OguA394EQfBflabZDRdcMqGhzANj8ugPofnugFPo5RWhT7gsBjzcIt1vtZay7/V0x f/Ew== X-Gm-Message-State: APjAAAVwFulzgG83chyWiRPyK3sGTNVW7jR3BQyumEzO9t4rnE/ve7Ub mQ7yfy82EK9AxZwlGPPCvZFzqhnWRjw= X-Google-Smtp-Source: APXvYqwu4ZeL1ncd4u3wnQsPhYKNN0GnD34aYketz7viLqCmFiwdt+WzYTq0tjvD2wCG8Et2fUmupA== X-Received: by 2002:a17:90a:8a0b:: with SMTP id w11mr12591548pjn.125.1565409550782; Fri, 09 Aug 2019 20:59:10 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.122]) by smtp.gmail.com with ESMTPSA id o32sm6527465pje.9.2019.08.09.20.59.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2019 20:59:09 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sat, 10 Aug 2019 09:28:58 +0530 Message-Id: <20190810035902.14047-1-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 1/5] autobuild-run: use different output directories for reproducible builds testing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Having different output directories increases variation in the build. This commit also changes the directory name of the single output directory to "output-1", as having varied length names for output directories introduces a lot of reproducibility issues. Official Debian efforts also currently use same length names for different directories: "/build/1st/$pkg-$ver" and "/build/$pkg-$ver/2nd" Cfr: https://tests.reproducible-builds.org/debian/index_variations.html Reproducible-Builds has proposed a spec called BUILD_PATH_PREFIX_MAP to help with this. However, that is still in the works and not fully solved yet. (As told on the #reproducible-builds IRC). So I think until further work is done on BUILD_PATH_PREFIX_MAP, we will have to stay with output directories of same length. See: https://wiki.debian.org/ReproducibleBuilds/GCC-build-path and https://reproducible-builds.org/specs/build-path-prefix-map/ Signed-off-by: Atharva Lele --- Changes v1 -> v2: - Added more details in commit log (suggested by Thomas) --- scripts/autobuild-run | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index f208a56..21eb47a 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -275,7 +275,10 @@ class Builder: # path to the output directory here is not relative to the # Buildroot sources, but to the location of the autobuilder # script. - self.outputdir = os.path.abspath(os.path.join(self.idir, "output")) + self.outputdir = os.path.abspath(os.path.join(self.idir, "output-1")) + # We use a different output directory for more variance in reproducibility + # testing. + self.outputdir_2 = os.path.abspath(os.path.join(self.idir, "output-2")) self.resultdir = os.path.join(self.outputdir, "results") # If it doesn't exist, create the instance directory @@ -434,8 +437,8 @@ class Builder: reproducible_results = os.path.join(self.outputdir, "results", "reproducible_results") # Using only tar images for now - build_1_image = os.path.join(self.outputdir, "images-1", "rootfs.tar") - build_2_image = os.path.join(self.outputdir, "images", "rootfs.tar") + build_1_image = os.path.join(self.outputdir, "images", "rootfs.tar") + build_2_image = os.path.join(self.outputdir_2, "images", "rootfs.tar") with open(reproducible_results, 'w') as diff: if self.sysinfo.has("diffoscope"): @@ -523,16 +526,17 @@ class Builder: log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret - # First build has been built, move files and start build 2 - os.rename(os.path.join(self.outputdir, "images"), os.path.join(self.outputdir, "images-1")) + # Create the second output directory if it does not exist + if not os.path.exists(self.outputdir_2): + os.mkdir(self.outputdir_2) - # Clean up build 1 - f = open(os.path.join(self.outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % self.outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) + # Copy .config to the other output directory + shutil.copyfile(os.path.join(self.outputdir, ".config"), + os.path.join(self.outputdir_2, ".config")) # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") - ret = self.do_build(self.outputdir) + ret = self.do_build(self.outputdir_2) if ret != 0: log_write(self.log, "INFO: build 2 failed") return ret From patchwork Sat Aug 10 03:58:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1145052 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="g31talVY"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4657f6157nz9sN6 for ; Sat, 10 Aug 2019 13:59:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5BA4B884C5; Sat, 10 Aug 2019 03:59:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FlFVD6cP+0Oy; Sat, 10 Aug 2019 03:59:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 58EA68815F; Sat, 10 Aug 2019 03:59:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 7C4021BF20D for ; Sat, 10 Aug 2019 03:59:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 795818815F for ; Sat, 10 Aug 2019 03:59:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3hjWnN97cK7I for ; Sat, 10 Aug 2019 03:59:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by hemlock.osuosl.org (Postfix) with ESMTPS id E46B187F81 for ; Sat, 10 Aug 2019 03:59:13 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id t16so47017621pfe.11 for ; Fri, 09 Aug 2019 20:59:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Bj8iYfrExeuOyiVeMZZ5PWZZePloozM8rNTQxPPtYL8=; b=g31talVYnO5l6BY5sf21akPXm4tQiNG8PtuihuJzW17dL9xDhw0pyQp455up5Re1Cq DYdNQY7Mv7BReCjoFUR9RjOifhH28Y7NhbSbHb5AwBiiKB+8kR0DZFzsOPmlJUDzvHx3 DBHOO3Ozjw3eN4vlkcNNQHTGGIF8rlD9Sdiux3QvKq458mHINWDIWkpsEiW2iXlT2hA6 WZhH3kxa7KrEgL+71KGEcJL7RCsUV92GNGL+JL1D1qrnC4IJiX8VpUGkKigQ7GGZ8knF 4Omihsl7NzpsuCXVs5uskTRd5Ns7ZXkKaowdr14X9Jvx6d2egC+sxLNJUBhnQzoDkcC3 fOtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bj8iYfrExeuOyiVeMZZ5PWZZePloozM8rNTQxPPtYL8=; b=YSLWJranEehFMUCRJdsYDoGA/nlDXOkJjyJo7fStKj6URWcxPaolM19lolp8ihqSoy /HkdMgo1oWyLyqrM7y78NALPx/TCxQ3lqsSSwy8P0jbEgvtMxtKViVtQsSCvcWv78aFS YQP2/Bi/kdj+hpsHpaiYiNNR1F6uHxYezT8rHVjuVCaUf9CufPAYQmmMvLqoQJ+xoaKF k2BJlgPWnlVoweERRIzW4lJKMUer9RYZ3b1VfRFlHpfqZ/HgCAfvnJQ2MAi2WeVz8t+O jrQEB7oEmeKNVOmJKoiv/KuiPITADTJifd60FO9lvj8rRfidaljcy8dx0jA38SF74TuF RLTw== X-Gm-Message-State: APjAAAVvK9/VcYxUZiNI9zLiIUiAZ59Oeq+hbzUsBiXvX1ruUI8nYlef Y7uITQ1mxKBBNwSJhd3Y/1MEMLeHs50= X-Google-Smtp-Source: APXvYqyed0pje8CX9qVeRuUAEMVtUgsgr1Bagu7DRXjsnoAmvoJiDfqaFpfCNn6zXWPdhryizHqXBw== X-Received: by 2002:a62:e901:: with SMTP id j1mr25569506pfh.189.1565409553119; Fri, 09 Aug 2019 20:59:13 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.122]) by smtp.gmail.com with ESMTPSA id o32sm6527465pje.9.2019.08.09.20.59.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2019 20:59:12 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sat, 10 Aug 2019 09:28:59 +0530 Message-Id: <20190810035902.14047-2-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190810035902.14047-1-itsatharva@gmail.com> References: <20190810035902.14047-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 2/5] autobuild-run: make prepare_build() clean the output directory used for reproducibility testing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The previous patch introduced a new output directory for the second build in a reproducibile build test. This patch will make sure that it is cleaned before the subsequent build process begins. Signed-off-by: Atharva Lele --- scripts/autobuild-run | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 21eb47a..69766b2 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -383,6 +383,10 @@ class Builder: # shutil.rmtree doesn't remove write-protected files subprocess.call(["rm", "-rf", self.outputdir]) os.mkdir(self.outputdir) + + # If it exists, remove the other output directory used for reproducibility testing + if os.path.exists(self.outputdir_2): + subprocess.call(["rm", "-rf", self.outputdir_2]) with open(os.path.join(self.outputdir, "branch"), "w") as branchf: branchf.write(branch) From patchwork Sat Aug 10 03:59:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1145051 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="SkF590dk"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4657f46wqsz9sN4 for ; Sat, 10 Aug 2019 13:59:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id CF798868DC; Sat, 10 Aug 2019 03:59:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6RXT0tZd3kz4; Sat, 10 Aug 2019 03:59:20 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4417E86A02; Sat, 10 Aug 2019 03:59:20 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id C128C1BF20D for ; Sat, 10 Aug 2019 03:59:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id BE95C8815F for ; Sat, 10 Aug 2019 03:59:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VxPbMX2p647n for ; Sat, 10 Aug 2019 03:59:16 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 598F687F81 for ; Sat, 10 Aug 2019 03:59:16 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id o13so46767396pgp.12 for ; Fri, 09 Aug 2019 20:59:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Bm+G2ZT5hYBdw0KGNwaqxmT0IkuF8KaDQ04duZ4OrFY=; b=SkF590dkvZX0sYUd4tELXJ6AJRVnO9EUT2FXqULBkND92H8Ng+ViExIsiYzhDxtnwP fPjrR2Jefm8zBZnmSpG07Fe75r0Luco1jFSSHg8Fwf3XqBCQC2T6gjQf1HY5KCVEXyC0 2Z2XrEzMARo4qhQRkVRcwAXxi4kQhnvAt33Y1I0jnCZzTUcrIYYrrLi37aXVKdob0z/W KlA9jxXmt7yNCTvp4WJs32etpDwrMndrGOUSnlR6ttQTRR7Cntg2hIgylRLjNEAa8quC AlxM/zj+1k0SMzy9cv9pNSd6Ft1iFnXVVQC+jOKZREiyGTNFMc3Rw5s/9aMOvQsHAEai L40w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Bm+G2ZT5hYBdw0KGNwaqxmT0IkuF8KaDQ04duZ4OrFY=; b=GWh1N/fJydqQrgpfVd3BZDty1ysLcd2Zbq/nealpL8WCBxYMXXS3b+7D9ifO/Rrd8H 9INwKnZkXMUvQJXgpX7DhLwEIiSQz6BjDLSbN/kiJsVEHmiSgwp++H0D6BZdq6nkeh/v wCkJRmVvJQFQNAC6UyYmNyP0kB7q00M3dKZSG6MjA6KHaa3WH456uAChMBxPsCHggj8c TvjXDY/BUlL0KcgJdQ4wP29a+ludNyhG2X9nisDuAjV4JYvRqFg5A9wYFIo5TkCqL8lW 6NlEz5pBb2AG1CvVPHZUFz1ATAmCD7cvtZZ+QlPvTfRgEwQTn7cfmj4loiig9lYTDmxa 8hAw== X-Gm-Message-State: APjAAAUsDXmFetILijuxIyCpbuj7XnI9PglKJr0hu10bgCfrqNP4JD45 IfXAYy0Xfg6riBM7I7S3qvGzrE341sg= X-Google-Smtp-Source: APXvYqxL4A1oyh30xOpvreDi5LwKcuxRj0sy67H7I1XlMxY2VPlr8J20n1hld5YziYIWFbqdzdcIFw== X-Received: by 2002:a62:f250:: with SMTP id y16mr25107962pfl.50.1565409555552; Fri, 09 Aug 2019 20:59:15 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.122]) by smtp.gmail.com with ESMTPSA id o32sm6527465pje.9.2019.08.09.20.59.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2019 20:59:15 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sat, 10 Aug 2019 09:29:00 +0530 Message-Id: <20190810035902.14047-3-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190810035902.14047-1-itsatharva@gmail.com> References: <20190810035902.14047-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 3/5] autobuild-run: make diffoscope output a JSON-formatted file as well X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Normal diffoscope output is readable by humans but not really convenient when working with it in code. JSON can be easily read and extracted information from. We still keep the normal text version because it's easy to parse by a human, but we limit its size to 40KB. Signed-off-by: Atharva Lele --- Changes v1 -> v2: - Save text output as well as JSON --- scripts/autobuild-run | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 69766b2..6adfa99 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -440,6 +440,7 @@ class Builder: """ reproducible_results = os.path.join(self.outputdir, "results", "reproducible_results") + reproducible_results_text = os.path.join(self.outputdir, "results", "reproducible_results_text") # Using only tar images for now build_1_image = os.path.join(self.outputdir, "images", "rootfs.tar") build_2_image = os.path.join(self.outputdir_2, "images", "rootfs.tar") @@ -453,7 +454,9 @@ class Builder: prefix = prefix[13:-1] log_write(self.log, "INFO: running diffoscope on images") subprocess.call(["diffoscope", build_1_image, build_2_image, - "--tool-prefix-binutils", prefix], stdout=diff, stderr=self.log) + "--tool-prefix-binutils", prefix, "--json", "-", + "--text", reproducible_results_text, "--max-text-report-size", "40000"], + stdout=diff, stderr=self.log) else: log_write(self.log, "INFO: diffoscope not installed, falling back to cmp") subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=self.log) From patchwork Sat Aug 10 03:59:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1145054 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="uCfTw9EA"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4657fB2j4xz9sMr for ; Sat, 10 Aug 2019 13:59:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8A50286B01; Sat, 10 Aug 2019 03:59:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OcVkY-eRbOqG; Sat, 10 Aug 2019 03:59:23 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 19AD586AC1; Sat, 10 Aug 2019 03:59:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 446E81BF20D for ; Sat, 10 Aug 2019 03:59:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 417AA8506F for ; Sat, 10 Aug 2019 03:59:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yhitPKwmBD-V for ; Sat, 10 Aug 2019 03:59:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by whitealder.osuosl.org (Postfix) with ESMTPS id DA82384E4E for ; Sat, 10 Aug 2019 03:59:18 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id w10so46786425pgj.7 for ; Fri, 09 Aug 2019 20:59:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xABifVLIE63CPUlp9bXy4nXyooI7F8DnX0DhX5GImiI=; b=uCfTw9EA/KX+jhIZ12188cB/YaaNt8KYC5RO1Uqe9iU67pTpQu7ubmaQVfGA8Dm2na Id0Seq/fxAfC6hgCXuXiFtNDhbWapPkaJ/NUAUK9VZq4OXucvuS9H/YJWqHzf9x4pN+w qGt7HtRaUZclWRdB2PzrbpH49fpLtk+gRCUKc6afPbzCRfw8jxhlGfXtBpSA9g1kDf36 V7KhjMJXVWXoIo7UYf2+8HDVxk3mV+J8wnDs5m6HU2QgKjrffsEirOz5h38GFM64ivzK a7ADok2I1yHo5f85k654lUXcMM6aLixFDlZL2ILzgkmZGZM+8nv0EvNqErL26t1ypbO3 +nzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xABifVLIE63CPUlp9bXy4nXyooI7F8DnX0DhX5GImiI=; b=VuIb7P4xpJ7STrtCYuwrQe6JenE9uDPBCFDwSACgEMCpUYdyLyfGJXLopS1HHzlbu5 eq6u9gnMoNBwDO6wPefcbe+gPG7mpDx6GXOaX48KyLQC2Ys0TNZK3P4d/HHXHbinxH2S SNVfSQrSS1IRm/hDd4MrlsDscLcDmDct79AVxa6/OTvLZyU0DeFGuFp3LG0A6P4dD0JP 1KkesRcBd1qd0Yz5b9kYGzhShInljfEHU9b49RpNqvv6W4wrYR2eTnIvaU/+BB/+vl3X sFyUb/364zSnm9HawpVozQGL0JzRPUHCiGPyd5RbUJN2WzlKcfgEiN6qg8OqDrI9ggWe 7RRA== X-Gm-Message-State: APjAAAXGPTmOGqlbx5glvdA6UQLAlw94bPakoTWuQJpI4pvNxGokF/p2 A+LraR0OQ4faa4NVhlu4v8OipK/fZIE= X-Google-Smtp-Source: APXvYqxyj23qVDJ2cOHFcliMG20ixOsEXJCj5jkMLi4xMKtO6Etxa1ue4Ez/7ypcifubc+PnmdmawQ== X-Received: by 2002:a62:82c1:: with SMTP id w184mr25613311pfd.8.1565409558082; Fri, 09 Aug 2019 20:59:18 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.122]) by smtp.gmail.com with ESMTPSA id o32sm6527465pje.9.2019.08.09.20.59.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2019 20:59:17 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sat, 10 Aug 2019 09:29:01 +0530 Message-Id: <20190810035902.14047-4-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190810035902.14047-1-itsatharva@gmail.com> References: <20190810035902.14047-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 4/5] autobuild-run: initial implementation of get_reproducibility_failure_reason() X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Analyze the JSON formatted output from diffoscope and check if the differences are due to a filesystem reproducibility issue or a package reproducibility issue. Also, discard the deltas because they might take up too much space. Signed-off-by: Atharva Lele --- Changes v1 -> v2: - Refactor using subfunctions and local variables (suggested by Thomas) - Added comments (suggested by Thomas) - Use more pythonic loops (suggested by Thomas) --- scripts/autobuild-run | 89 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 6adfa99..1be392a 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -131,6 +131,7 @@ import csv import docopt import errno import hashlib +import json import mmap import multiprocessing import os @@ -599,6 +600,94 @@ class Builder: if reject_results(): return + def get_reproducibility_failure_reason(reproducible_results): + def split_delta(delta): + # Take a delta and split it into added, deleted lines. + added = [] + deleted = [] + for line in delta: + if line.startswith("+"): + added.append(line) + if line.startswith("-"): + deleted.append(line) + return added, deleted + + def get_package(sourcef): + # Returns which package the source file belongs to. + with open(packages_file_list, "r") as packagef: + for line in packagef: + if sourcef in line: + package = line.split(',')[0] + + if package: + # Get package version + package_info = json.loads(subprocess.check_output(["make", "--no-print-directory", + "O=%s" % self.outputdir, + "-C", self.srcdir, + "%s-show-info" % package])) + if "version" in package_info[package]: + version = package_info[package]["version"] + return [package, version] + else: + return [package] + else: + return ["not found"] + + def cleanup(l): + # Takes a list and removes data which is redundant (source2) or data + # that might take up too much space (like huge diffs). + if "unified_diff" in l: + l.pop("unified_diff") + if "source2" in l: + l.pop("source2") + + + packages_file_list = os.path.join(self.outputdir, "build", "packages-file-list.txt") + + with open(reproducible_results, "r") as reproduciblef: + json_data = json.load(reproduciblef) + + if json_data["unified_diff"] == None: + # Remove the file list because it is not useful, i.e. it only shows + # which files vary, and nothing more. + if json_data["details"][0]["source1"] == "file list": + json_data["details"].pop(0) + + # Iterate over details in the diffoscope output. + for item in json_data["details"]: + diff_src = item["source1"] + item["package"] = get_package(diff_src) + + # In some cases, diffoscope uses multiple commands to get various + # diffs. Due to this, it generates a "details" key for those files + # instead of just storing the diff in the "unified_diff" key. + if item["unified_diff"] == None: + for item_details in item["details"]: + diff = item_details["unified_diff"].split("\n") + split_deltas = split_delta(diff) + item_details["added"] = split_deltas[0][:100] + item_details["deleted"] = split_deltas[1][:100] + cleanup(item_details) + else: + diff = item["unified_diff"].split("\n") + split_deltas = split_delta(diff) + item["added"] = split_deltas[0][:100] + item["deleted"] = split_deltas[1][:100] + cleanup(item) + # We currently just set the reason from first non-reproducible package in the + # dictionary. + reason = json_data["details"][0]["package"] + + # If there does exist a unified_diff directly for the .tar images, it is probably + # a filesystem reproducibility issue. + else: + reason = ["filesystem"] + + with open(reproducible_results, "w") as reproduciblef: + json.dump(json_data, reproduciblef, sort_keys=True, indent=4) + + return reason + def get_failure_reason(): # Output is a tuple (package, version), or None. lastlines = decode_bytes(subprocess.Popen( From patchwork Sat Aug 10 03:59:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1145053 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HrCUTtzY"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4657f82w7rz9sMr for ; Sat, 10 Aug 2019 13:59:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 49642856ED; Sat, 10 Aug 2019 03:59:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JzToRti9C7TW; Sat, 10 Aug 2019 03:59:23 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id DDE898506F; Sat, 10 Aug 2019 03:59:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id A5B3C1BF20D for ; Sat, 10 Aug 2019 03:59:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id A0BAA88289 for ; Sat, 10 Aug 2019 03:59:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SO2pknCu8eGu for ; Sat, 10 Aug 2019 03:59:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 2E125884B9 for ; Sat, 10 Aug 2019 03:59:21 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id i18so46787688pgl.11 for ; Fri, 09 Aug 2019 20:59:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZN0WI7Abtil0NLegKVeKKziPmLAiyXeAjHRT1NeHU1Q=; b=HrCUTtzY+nFW55+aFq9Uy7o+tFj+vPZOVVlPbTXuJsm7Q4LIrkicO96U00U+p/BWtE nJumLW/tEksrqIrc2IZjDSJeHNqHcC92qoA8HS7ztgnTxVlKWQdaveVv1XFEiwKa/cbj NlPchLi9vr1LXJOrSW1dczUMRtIDikPKu6/tZrJg/Bd8qVPbk5Q9XurDyp7i21vXonR5 SRBRze2V/bklUWnnk88efKa/O3s6m+fGYyFxzhZPz65NBzyCAW9KRkv6mV7PV3UlpgJJ GdeYKxUzkHsI9CBSDXagTzULADkgxpaWWBsTBk91dY7l4vauRxGeUIsYRvmY+VFC+X93 5E7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZN0WI7Abtil0NLegKVeKKziPmLAiyXeAjHRT1NeHU1Q=; b=nDKTlC0nMcelYQEe4TdLfkEqvWWiUzDePivb3AZczXLX3MMv5+5E7YHQR12xaBUhsm 6M3rs80fX/kZumwrUU8qMAD8s5xjIL6s48cDNZBQuuWI4dmCwwCLfnAbEYcMazTNl9TJ zK30wWzm7b/pZyL3hknRMkdT91blnpNL2kavf5xoOCrlHYgDEpmZGBo68UbVncFsdB2v StvRSv2FrkjeT/XLpe0hPAqpqzxGTKjk7o2nJboI1ciGheS9zJEy78AiASu7bWnh0lDu 2c1qttryPcPeMEwqvcZ+cwGyIbLXMvP/ZoEENo7QBp+MXvyP4tJsyejomcd5LEIFWwym iu9A== X-Gm-Message-State: APjAAAUaSz8nj7P3B+9j7zKenCTogTWYe+UYNKY4dqXFiu9x3oThCiNr MBcjGquUzSM8uwEBmlKe8RQuaf9YNU0= X-Google-Smtp-Source: APXvYqybJCgewtB1PM2PjzGFab3xe7Yj+WG0201a7casBG42xteQz3rqT83cKr7WepsvDjV2qLChEA== X-Received: by 2002:a63:2a08:: with SMTP id q8mr20298192pgq.415.1565409560370; Fri, 09 Aug 2019 20:59:20 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.122]) by smtp.gmail.com with ESMTPSA id o32sm6527465pje.9.2019.08.09.20.59.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 09 Aug 2019 20:59:19 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sat, 10 Aug 2019 09:29:02 +0530 Message-Id: <20190810035902.14047-5-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190810035902.14047-1-itsatharva@gmail.com> References: <20190810035902.14047-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 5/5] autobuild-run: account for reproducibility failures in get_failure_reason() X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- scripts/autobuild-run | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 1be392a..7fc8d3a 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -690,15 +690,26 @@ class Builder: def get_failure_reason(): # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() - - regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') - for line in lastlines: - m = regexp.search(line) - if m: - return m.group(1).rsplit('-', 1) + # Output is "package-reproducible" in case of reproducibility failure. + + reproducible_results = os.path.join(self.resultdir, "reproducible_results") + if os.path.exists(reproducible_results) and os.stat(reproducible_results).st_size > 0: + if self.sysinfo.has("diffoscope"): + reason = get_reproducibility_failure_reason(reproducible_results) + reason.append("nonreproducible") + return reason + else: + return ["nonreproducible"] + else: + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() + + regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') + for line in lastlines: + m = regexp.search(line) + if m: + return m.group(1).rsplit('-', 1) # not found return None