From patchwork Tue Aug 20 14:52:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1150216 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="pGh3VAOZ"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46CYgw31JJz9sR0 for ; Wed, 21 Aug 2019 00:53:15 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id BECFC2286C; Tue, 20 Aug 2019 14:53:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sI4SoIo2SPXh; Tue, 20 Aug 2019 14:53:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id D21462040F; Tue, 20 Aug 2019 14:53:10 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 6EEE01BF82B for ; Tue, 20 Aug 2019 14:52:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 6BE3E87F92 for ; Tue, 20 Aug 2019 14:52:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5VBy6rFIysco for ; Tue, 20 Aug 2019 14:52:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 08A0D81FB0 for ; Tue, 20 Aug 2019 14:52:38 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id d1so3393822pgp.4 for ; Tue, 20 Aug 2019 07:52:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=TnGZlgPDbRkgy0mpPk/5kH8efatk8cqB2xTYXLUzqFU=; b=pGh3VAOZTrzCX7/PnmPZIFOQWZUdIVjdgfKd5smnxPJ/4VyQfQdTClo2WCTnT0hYnI 9wg9igKqw0HgMB3FxCT+kx+eJ/7F1ck3U7JBYEQ3zpburVVK8egwsL5K/AOdBZT0KZyp Q7u/xhM8n5v01/1rqahIftNdtVbMV5lSgbEJib/iDkiomFDa4YKWqgMqxnkmVldz+iWy wyHVOV93R6OBZj2ieE4mjQXvx0S8PjZ3O0TpCkKXewjUrzMTEmE9e300ZqxYyra9i+5Y e4Xtmm0H45OMxeOwbvLoDWJFtkf01RiUGUM/ShLt4j/HBNQLWqoKR87bZTaZsKSfgyuU /uuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=TnGZlgPDbRkgy0mpPk/5kH8efatk8cqB2xTYXLUzqFU=; b=XnuE9we0fUyOKAZsFHltDEb3yRXM5WdgMf8RWEw86FoJmXmz3rJ9Tkd/XhqDhEWBOB fwlBJ9CPx2aKlWXni40zXfa7UYOXIeV+VLhiJkeTRmvz69bm+2v2j4cHVrljD/A2+oRz Z0WufmlyFTmQKigmUVKZGF++2f4Gncc75lGmhNf85TpxyoJH5PmAgwChBAOWFL0JRYcW 1FsYI2fnoTNRTtKUfv+L89LQyPkkxzmftQ/oQXObgjHRUkw3Pz0ARSxxFpKy/15qliFw y++xocpa6eYTxlccFEsZjOCBIIekm0qvch6jWWtlZcri/ZVXHrWQ+PHsWTUZpOiWGE/w rIrQ== X-Gm-Message-State: APjAAAX/QpPZGERNbIbAMgNY7+eebxjjMnhlV/S6oqxZuo4Lj5ZXJT8k autjFiWgiVUaFuu6eQ5PW7GsrryVhxE= X-Google-Smtp-Source: APXvYqwfhTKbBLw5Ok56A51jPbaZEGwMew2gQxAtA0aVK2HQ0FEgyZxjMdXMMKdV/8nvGihR3V3pfg== X-Received: by 2002:a65:464d:: with SMTP id k13mr22422690pgr.99.1566312756992; Tue, 20 Aug 2019 07:52:36 -0700 (PDT) Received: from localhost.localdomain ([123.201.116.168]) by smtp.gmail.com with ESMTPSA id a1sm24502643pgh.61.2019.08.20.07.52.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2019 07:52:36 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Tue, 20 Aug 2019 20:22:27 +0530 Message-Id: <20190820145231.15507-1-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 1/5] autobuild-run: check if reproducibile_results exists before checking its size X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This was causing some reproducible configs to crash autobuild script since it tried to check the size of a file that did not exist. Signed-off-by: Atharva Lele --- scripts/autobuild-run | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index ead81a0..99b57dd 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -461,7 +461,7 @@ class Builder: log_write(self.log, "INFO: diffoscope not installed, falling back to cmp") subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=self.log) - if os.stat(reproducible_results).st_size > 0: + if os.path.exists(reproducible_results) and os.stat(reproducible_results).st_size > 0: log_write(self.log, "INFO: Build is non-reproducible.") return -1 From patchwork Tue Aug 20 14:52:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1150217 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Mj5Z+Ied"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46CYgx0Ngbz9sQn for ; Wed, 21 Aug 2019 00:53:16 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id ACDA8876B6; Tue, 20 Aug 2019 14:53:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1VBIJEjX5TdG; Tue, 20 Aug 2019 14:53:11 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id AC7F1876AE; Tue, 20 Aug 2019 14:53:11 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 010061BF82B for ; Tue, 20 Aug 2019 14:52:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id F1F7D2040F for ; Tue, 20 Aug 2019 14:52:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UqYQlexdDuxu for ; Tue, 20 Aug 2019 14:52:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by silver.osuosl.org (Postfix) with ESMTPS id 317D5203F8 for ; Tue, 20 Aug 2019 14:52:40 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id d85so3544952pfd.2 for ; Tue, 20 Aug 2019 07:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1CzxpWlLXrBWv5tz2fudf6raKJI7v0M9p2WLgCH5lKQ=; b=Mj5Z+IediqL6DlMH/kMxVab9U3Asw+F/sN8zcoah7wJz8GklXMIz8L/p4OSvfnH8ss m09SajvRUEOaQJN+OXeOR7Xtmv+UW9OicMyzWM1yBYRkqD8urUR+cbHaTo0F4LDyywzt NBIzMCYgVR2I752IrLEICpurK1K+tsiaFFGVVuQPKZJRszBeXhCUnhJdiCenEwJF4yjh 9ONc3US7e/rgCgNMwuBNYwADwYkbrt72rJ9Ggron/zio5s86eSeimqNFM/Lnk1AfP5GR aW1zO5dHXU0JvJSAnCu8TgChj7lY3tJ9rZmkG5oCN0pOxQZkBIHErK00Ndy+PVVdPfxb ypXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1CzxpWlLXrBWv5tz2fudf6raKJI7v0M9p2WLgCH5lKQ=; b=COpPpqIvMumXW/9TBQFPNQA5/dvfRSZPH051cKiRRhBwpzhXv8ifFH6fEHMrTd65Db xqlarYjVnggKIhBchmCNMvu8QzJso/4qVUoOXLSKb/ZKtdv45IVgqTPudN8JalUhy0vq g9R2npQVU2tGOP9QG/5fkz2cb4F14H54EwcGcJrcagcbllPhEMeEERcVl3K3naVBZA4Q Kiajr8YRHQiYd2reM/3YhZEuwUGWKm/3uSGCuZZ5xrpLcoU9l7/RLBTAB2WaYcIIIZUR SkIW04R2TV4iNomoWMK8/TOlbTkbWJNUr06RyUWQYndHQx8n/OuXcmrUjIoBsX0ifnfP ghIQ== X-Gm-Message-State: APjAAAX3VypDi8SDmOI0PvRNCJrDr4heR7kM6iLRoNZyKkhSYRnmZi10 jNV0C4fbAh5432lJUdY/46JOnPbVd/A= X-Google-Smtp-Source: APXvYqxYnfEnAYkjZ5Pxal0ShGxRfLgyDgA7JGcw3LqBXuYCw8uKlYEjDkcCKRqHgmhEbh3OAMLQyA== X-Received: by 2002:a63:784c:: with SMTP id t73mr25947910pgc.268.1566312759260; Tue, 20 Aug 2019 07:52:39 -0700 (PDT) Received: from localhost.localdomain ([123.201.116.168]) by smtp.gmail.com with ESMTPSA id a1sm24502643pgh.61.2019.08.20.07.52.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2019 07:52:38 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Tue, 20 Aug 2019 20:22:28 +0530 Message-Id: <20190820145231.15507-2-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190820145231.15507-1-itsatharva@gmail.com> References: <20190820145231.15507-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 2/5] autobuild-run: initial implementation of get_reproducibility_failure_reason() X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Analyze the JSON formatted output from diffoscope and check if the differences are due to a filesystem reproducibility issue or a package reproducibility issue. Also, discard the deltas because they might take up too much space. Signed-off-by: Atharva Lele --- Changes v2 -> v4: - Change if-else to try-except - remove blank line Changes v1 -> v2: - Refactor using subfunctions and local variables (suggested by Thomas) - Added comments (suggested by Thomas) - Use more pythonic loops (suggested by Thomas) --- scripts/autobuild-run | 88 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 99b57dd..384cf73 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -131,6 +131,7 @@ import csv import docopt import errno import hashlib +import json import mmap import multiprocessing import os @@ -599,6 +600,93 @@ class Builder: if reject_results(): return + def get_reproducibility_failure_reason(reproducible_results): + def split_delta(delta): + # Take a delta and split it into added, deleted lines. + added = [] + deleted = [] + for line in delta: + if line.startswith("+"): + added.append(line) + if line.startswith("-"): + deleted.append(line) + return added, deleted + + def get_package(sourcef): + # Returns which package the source file belongs to. + with open(packages_file_list, "r") as packagef: + for line in packagef: + if sourcef in line: + package = line.split(',')[0] + + try: + # Get package version + package_info = json.loads(subprocess.check_output(["make", "--no-print-directory", + "O=%s" % self.outputdir, + "-C", self.srcdir, + "%s-show-info" % package])) + if "version" in package_info[package]: + version = package_info[package]["version"] + return [package, version] + else: + return [package] + except: + return ["not found"] + + def cleanup(l): + # Takes a list and removes data which is redundant (source2) or data + # that might take up too much space (like huge diffs). + if "unified_diff" in l: + l.pop("unified_diff") + if "source2" in l: + l.pop("source2") + + packages_file_list = os.path.join(self.outputdir, "build", "packages-file-list.txt") + + with open(reproducible_results, "r") as reproduciblef: + json_data = json.load(reproduciblef) + + if json_data["unified_diff"] == None: + # Remove the file list because it is not useful, i.e. it only shows + # which files vary, and nothing more. + if json_data["details"][0]["source1"] == "file list": + json_data["details"].pop(0) + + # Iterate over details in the diffoscope output. + for item in json_data["details"]: + diff_src = item["source1"] + item["package"] = get_package(diff_src) + + # In some cases, diffoscope uses multiple commands to get various + # diffs. Due to this, it generates a "details" key for those files + # instead of just storing the diff in the "unified_diff" key. + if item["unified_diff"] == None: + for item_details in item["details"]: + diff = item_details["unified_diff"].split("\n") + split_deltas = split_delta(diff) + item_details["added"] = split_deltas[0][:100] + item_details["deleted"] = split_deltas[1][:100] + cleanup(item_details) + else: + diff = item["unified_diff"].split("\n") + split_deltas = split_delta(diff) + item["added"] = split_deltas[0][:100] + item["deleted"] = split_deltas[1][:100] + cleanup(item) + # We currently just set the reason from first non-reproducible package in the + # dictionary. + reason = json_data["details"][0]["package"] + + # If there does exist a unified_diff directly for the .tar images, it is probably + # a filesystem reproducibility issue. + else: + reason = ["filesystem"] + + with open(reproducible_results, "w") as reproduciblef: + json.dump(json_data, reproduciblef, sort_keys=True, indent=4) + + return reason + def get_failure_reason(): # Output is a tuple (package, version), or None. lastlines = decode_bytes(subprocess.Popen( From patchwork Tue Aug 20 14:52:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1150219 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="rzxw0P1e"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46CYh25N6pz9sPx for ; Wed, 21 Aug 2019 00:53:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 02372227CC; Tue, 20 Aug 2019 14:53:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hCfpk-JTGVJO; Tue, 20 Aug 2019 14:53:13 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 9738822850; Tue, 20 Aug 2019 14:53:12 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id CCE3A1BF82B for ; Tue, 20 Aug 2019 14:52:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C99502040F for ; Tue, 20 Aug 2019 14:52:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QFAq+0sIHNdr for ; Tue, 20 Aug 2019 14:52:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by silver.osuosl.org (Postfix) with ESMTPS id 48957203F8 for ; Tue, 20 Aug 2019 14:52:42 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id n4so3401726pgv.2 for ; Tue, 20 Aug 2019 07:52:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aLxGwz1J4lUuW+a7bzUI6zucMcFqBIVRz99y0SIxcbc=; b=rzxw0P1ew1fsDpdJc6GVygKGlsjXqw8+GQNXDa2tUfIfKRmfrdcUmLHL6Gac0YiT2c G94OHZYoBExZd9h/0nnUYhAayWlhMP8i2z91/BXKLVrHPp/6qYkhFKElfxvEEwR45Ibw IslGwAe+FItKXxAmUGIxJdQk+cPTTVpwQK19WXn6ken8TWpMsQkKR46/Hyd3GSeVRaNi zzbv7JxM1HX2Kcjc1JjfIuZACC3YImBQ3jCjKbMa7Xc3huTu2NqFNDZz30kUkLVxnmVJ zY/9m7QFvJWx/i7lZgPYBKpOnnVksGjT5+Qi6H6fsS4XqeX1Kk5eeUR+MWb1FQsa4Z41 eiGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aLxGwz1J4lUuW+a7bzUI6zucMcFqBIVRz99y0SIxcbc=; b=KGYIGgH0jY+vZ5NdzjwIsVS/ayDlO/z0FMxCAujm2SYncnHMv5/w1yw1jD8RVL+ICk SwU+05q2f+d93p89A9xWlYakDhSlpbIUYH4jU1L17qAtHTebt8tHXiMxtif9ILiiwQ7M LPbyERgykdX3/qzvm1qrdBmtdhLHW5BOCJHRLFiHgwmPgy2HtYIULlJXHoIA3pMYHhqE vKlQOq8W2IEzCdkMNU2+5IqzZsyWG5uvsR8B0JerCHjvOtEFzdvhnHF53GJZZDZVw7vV YJjEpEfBrpsnQl27geZV2+deQv2fZ1NDvmtdYmgc4Kb5sWSEGPGaOabsHe/fQxYBzKPm OJxA== X-Gm-Message-State: APjAAAXO8aZFokamCcwSW0/gCzXhQ0UnicqKAHKfgG+nXPlAQp29vsb5 K8ctAmJvr3dvyUvCrmhpllXW1rJbwqE= X-Google-Smtp-Source: APXvYqx8yPX4BV9gIWHTvnAHtvtIjfWNMtPb+dIJfO5XS9Vpxjd/4Q6CRSTcYrHdOP530zfOQZ0TOQ== X-Received: by 2002:a62:c584:: with SMTP id j126mr30479495pfg.21.1566312761515; Tue, 20 Aug 2019 07:52:41 -0700 (PDT) Received: from localhost.localdomain ([123.201.116.168]) by smtp.gmail.com with ESMTPSA id a1sm24502643pgh.61.2019.08.20.07.52.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2019 07:52:41 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Tue, 20 Aug 2019 20:22:29 +0530 Message-Id: <20190820145231.15507-3-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190820145231.15507-1-itsatharva@gmail.com> References: <20190820145231.15507-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 3/5] autobuild-run: account for reproducibility failures in get_failure_reason() X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- Changes v1 -> v3: - Account for changed name of diffoscope output --- scripts/autobuild-run | 29 ++++++++++++++++++++--------- 1 file changed, 20 insertions(+), 9 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 384cf73..876feb2 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -689,15 +689,26 @@ class Builder: def get_failure_reason(): # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() - - regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') - for line in lastlines: - m = regexp.search(line) - if m: - return m.group(1).rsplit('-', 1) + # Output is "package-reproducible" in case of reproducibility failure. + + reproducible_results = os.path.join(self.resultdir, "diffoscope-results.json") + if os.path.exists(reproducible_results) and os.stat(reproducible_results).st_size > 0: + if self.sysinfo.has("diffoscope"): + reason = get_reproducibility_failure_reason(reproducible_results) + reason.append("nonreproducible") + return reason + else: + return ["nonreproducible"] + else: + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() + + regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') + for line in lastlines: + m = regexp.search(line) + if m: + return m.group(1).rsplit('-', 1) # not found return None From patchwork Tue Aug 20 14:52:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1150220 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="biSeV8qV"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46CYh44zG8z9sNk for ; Wed, 21 Aug 2019 00:53:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C88B322850; Tue, 20 Aug 2019 14:53:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jHZ0Dp9b42Gw; Tue, 20 Aug 2019 14:53:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 3072B2288E; Tue, 20 Aug 2019 14:53:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 27C381BF82B for ; Tue, 20 Aug 2019 14:52:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 24DC187F92 for ; Tue, 20 Aug 2019 14:52:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iY6DH-dsRvyT for ; Tue, 20 Aug 2019 14:52:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by hemlock.osuosl.org (Postfix) with ESMTPS id A882281FB0 for ; Tue, 20 Aug 2019 14:52:44 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id w16so3533986pfn.7 for ; Tue, 20 Aug 2019 07:52:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0WabEcigge+baquWxE4XqX72KDWIXQvs3iGlK5MC+lo=; b=biSeV8qVWwc7Ych2x9goVNLwIynZ/0lOyiAsAAQf+kFSp0rNkiBt1GvvPnJT1/MF0t 7JSNA1ILtnHitHD9iJfjSfKrKBKDnEj+LttVnd/fIBd6MzyZ6798lS6IoYBFG26VjcxG /FfzgNMYXNjrmr9UkEnvoYC2uvOb7wvQZ+0k83JZESc5lAWIP6dCiTMpmhvO/bow4rTc zZC7EaJ6lu7B2uLxucOpNg53qtV8Wf/2g8nkU1O3pstEp2W2+mLISOx6bfomTZcvsgnF 5AHDcDxp3NUuhaASqgFWCKzHSF41fVudyzuaVbjz4QBrsRw8KZxDscYTa/NBukFkroxb Y/yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0WabEcigge+baquWxE4XqX72KDWIXQvs3iGlK5MC+lo=; b=l71tnLtZafJpSOa0NRAFZF0sGhhJAr/KTKrD9IApgzaP4aNDhLrnBzCVmB44xpDQNE JX/VPb6CBhbTjhHuzZEGwz3j2aoHnhnFFMGTyNx5sa/rfqu+LqIX8E++edMCrEQo+ujw jx5LyoO2fyDiarQX2HOtJSRPgMvBjguvuj/ZRP89SNEmNw1gQkH4agwmlRDkKGADNk7c ie9o8vVYPKBGpGMgucTEsvNhCZc5vHXLf4V5XMMoBHqkN+Z3u0JvdOvgpSgmw1tpr+di EyoWYMYfjbP0DJVAYs/vxpSAcSVbb/dI8SCVrGSMwKAmwgQN9nL6i/CiT7iTcCyo9su8 SGOw== X-Gm-Message-State: APjAAAXVjBfpZ6to7mr4/CsHHfQbgtWZf64jGWsZXaJTEj4NIsMNdGgA VQx5OgSGHLjtiSr9XX9u2Clwte/GsHA= X-Google-Smtp-Source: APXvYqy6iKmJ0kmwT3+pl3s38JjJ8a1riE2UbCpul6a/R/rPr4K8XjywPLnAuOxSWtg/fezqPhEuxw== X-Received: by 2002:a63:593:: with SMTP id 141mr16132562pgf.78.1566312763690; Tue, 20 Aug 2019 07:52:43 -0700 (PDT) Received: from localhost.localdomain ([123.201.116.168]) by smtp.gmail.com with ESMTPSA id a1sm24502643pgh.61.2019.08.20.07.52.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2019 07:52:43 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Tue, 20 Aug 2019 20:22:30 +0530 Message-Id: <20190820145231.15507-4-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190820145231.15507-1-itsatharva@gmail.com> References: <20190820145231.15507-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 4/5] autobuild-run: move with open to appropriate place in check_reproducibility() X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Since we switched to using diffoscope's ability to write to a file, having a with open() does not make sense while running the command. Instead, move it to the else condition where it can be used by cmp. Signed-off-by: Atharva Lele --- scripts/autobuild-run | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 876feb2..c25413b 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -446,19 +446,19 @@ class Builder: build_1_image = os.path.join(self.outputdir, "images", "rootfs.tar") build_2_image = os.path.join(self.outputdir_2, "images", "rootfs.tar") - with open(reproducible_results, 'w') as diff: - if self.sysinfo.has("diffoscope"): - # Prefix to point diffoscope towards cross-tools - prefix = subprocess.check_output(["make", "--no-print-directory", "O=%s" % self.outputdir, - "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) - # Remove TARGET_CROSS= and \n from the string - prefix = prefix[13:-1] - log_write(self.log, "INFO: running diffoscope on images") - subprocess.call(["diffoscope", build_1_image, build_2_image, - "--tool-prefix-binutils", prefix, "--json", reproducible_results, - "--text", reproducible_results_txt, "--max-text-report-size", "40000"], - stderr=self.log) - else: + if self.sysinfo.has("diffoscope"): + # Prefix to point diffoscope towards cross-tools + prefix = subprocess.check_output(["make", "--no-print-directory", "O=%s" % self.outputdir, + "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) + # Remove TARGET_CROSS= and \n from the string + prefix = prefix[13:-1] + log_write(self.log, "INFO: running diffoscope on images") + subprocess.call(["diffoscope", build_1_image, build_2_image, + "--tool-prefix-binutils", prefix, "--json", reproducible_results, + "--text", reproducible_results_txt, "--max-text-report-size", "40000"], + stderr=self.log) + else: + with open(reproducible_results_txt, 'w') as diff: log_write(self.log, "INFO: diffoscope not installed, falling back to cmp") subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=self.log) From patchwork Tue Aug 20 14:52:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1150218 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="puc1evAY"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46CYh067mgz9sNk for ; Wed, 21 Aug 2019 00:53:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 2FCE5876E2; Tue, 20 Aug 2019 14:53:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pHwTHgpsPUt2; Tue, 20 Aug 2019 14:53:16 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 2B420876A6; Tue, 20 Aug 2019 14:53:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 4DE2D1BF82B for ; Tue, 20 Aug 2019 14:52:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 48BA686969 for ; Tue, 20 Aug 2019 14:52:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CRAmN1K8rd4Z for ; Tue, 20 Aug 2019 14:52:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by fraxinus.osuosl.org (Postfix) with ESMTPS id A8A1086938 for ; Tue, 20 Aug 2019 14:52:46 +0000 (UTC) Received: by mail-pl1-f194.google.com with SMTP id m9so2873017pls.8 for ; Tue, 20 Aug 2019 07:52:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xX+L02EFaJFUv2prw/q3I55mqegRJVYWzJfMDAcU8TQ=; b=puc1evAYSKJjnlb9Hvb3ZHZtBE4jQn4NP6HsAxKizQtUDrS1nsWitGRMLjSbeLZHvy HoD/TqjEcsM/d1R2TxM9v6CpVQ+pAhUud5sFQY3xFrVaqnp4/DRj/u2xmpiQ46+xl89G 4eXjBh96BoaChDJPJeAu42/+8kFNFr8663c4WGOMGGbwb522suvqHqle1M7Z9jyG9FAT ClRR+128E4qakB+ORdyCNodjjxBlgMCTzzzGLmiuKueF4ivsSf++DZmgH/bSyLSAx7gn sWDmQtkAeg1bqMim21DKOmjvhofZfbkU8ZzlXMOGptADXxOvaPc+04dq2hbAqzybYSPD Q80w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xX+L02EFaJFUv2prw/q3I55mqegRJVYWzJfMDAcU8TQ=; b=g9OnUC6IU0snHSwuxIH8b4iC1peNwvXDBFBAySbz8tc/o1KWaxIAw2dLwfvekvlbsx SnRTH6PchT1NDTqlUXOG+IjMzU4HMXVOUFPU853HwnfSybxJ5rkx3Iyy4JKUEgfII00x ro9QCaPN0Qpsxe8JrIYPA5AhiViSGNvpcPgxyiCZzpvdmk80Ntyw41ELtwuYfSQxsRHJ etjBVdOwDHsyzzsMAFK3tyqNRO552NEJEjGjheVJCtcI8hh4k6WNM2iWUZprunVx/mgG b6b+nNjh73UXtQvRQjDsrSErH5/EH/ML4ko00VAhaqu3v23vZtMhTVhpkVIBCv9vVI5W r1sQ== X-Gm-Message-State: APjAAAV41ckIFkaxVzS6Nihftv2w30BtuHpYvW2w6XWXIODEefs5JMT6 Pr+6fC5EQZJWLX451jztuJhORyPs0sc= X-Google-Smtp-Source: APXvYqxA1vHv5ZngDSViZEH5DVi0G2TEIevyvNY67fI8gdJTyH811SWSfvHYSSPl4r/eCbQQScitug== X-Received: by 2002:a17:902:b710:: with SMTP id d16mr23047530pls.165.1566312765801; Tue, 20 Aug 2019 07:52:45 -0700 (PDT) Received: from localhost.localdomain ([123.201.116.168]) by smtp.gmail.com with ESMTPSA id a1sm24502643pgh.61.2019.08.20.07.52.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Aug 2019 07:52:45 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Tue, 20 Aug 2019 20:22:31 +0530 Message-Id: <20190820145231.15507-5-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190820145231.15507-1-itsatharva@gmail.com> References: <20190820145231.15507-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 5/5] autobuild-run: initial implementation of categorization() of nonreproducibility X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Build ID and Build Path reproducibility issues are easy to identify and thus we start categorization with these issues. Signed-off-by: Atharva Lele --- scripts/autobuild-run | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index c25413b..83acaad 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -131,6 +131,7 @@ import csv import docopt import errno import hashlib +from itertools import izip import json import mmap import multiprocessing @@ -641,6 +642,26 @@ class Builder: if "source2" in l: l.pop("source2") + def categorize(added, deleted): + # In some deltas, the only part of output directory is captured. + # For eg. "put-1" or "tput-2", thus we must check all such possibilities. + # Start with 3 letter combinations to avoid false positives. + path_1 = "output-1" + path_2 = "output-2" + paths = [path_1[i:j] for i in range(len(path_1)) for j in range(i+3, len(path_1)+1)] + paths_2 = [path_2[i:j] for i in range(len(path_1)) for j in range(i+3, len(path_1)+1)] + paths = paths + paths_2 + # We need to iterate over the deltas simultaneously. + for a, d in izip(added, deleted): + for p in paths: + if p in a or p in d: + return "Embedded Path" + if "Build ID" in a or "Build ID" in d: + return "Build ID variation" + else: + continue + return "not found" + packages_file_list = os.path.join(self.outputdir, "build", "packages-file-list.txt") with open(reproducible_results, "r") as reproduciblef: @@ -667,12 +688,18 @@ class Builder: item_details["added"] = split_deltas[0][:100] item_details["deleted"] = split_deltas[1][:100] cleanup(item_details) + category = categorize(item_details["added"], item_details["deleted"]) + if category is not "not found": + item["category"] = category + break else: diff = item["unified_diff"].split("\n") split_deltas = split_delta(diff) item["added"] = split_deltas[0][:100] item["deleted"] = split_deltas[1][:100] cleanup(item) + if "added" in item or "deleted" in item: + item["category"] = categorize(item["added"], item["deleted"]) # We currently just set the reason from first non-reproducible package in the # dictionary. reason = json_data["details"][0]["package"]