From patchwork Sun Jul 7 05:28:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128553 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Jf58Lg8e"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFV04gBz9sCJ for ; Sun, 7 Jul 2019 15:29:17 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9FD1B20013; Sun, 7 Jul 2019 05:29:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cOmkOXAQOa03; Sun, 7 Jul 2019 05:29:01 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 46A0F203C3; Sun, 7 Jul 2019 05:29:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 126381BF5DD for ; Sun, 7 Jul 2019 05:29:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 04DCD8588A for ; Sun, 7 Jul 2019 05:29:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3MKyCG7KRYK1 for ; Sun, 7 Jul 2019 05:28:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by whitealder.osuosl.org (Postfix) with ESMTPS id 57097857EB for ; Sun, 7 Jul 2019 05:28:57 +0000 (UTC) Received: by mail-pl1-f194.google.com with SMTP id a93so6538446pla.7 for ; Sat, 06 Jul 2019 22:28:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=JWsr5krGaAh5zk5p3jkSRk1l6yA/OJpF2OW6YBQgf8I=; b=Jf58Lg8eYZKvc1Vvdz0YntNkbVMk4zhZR0vX0WzRBPJdxx549uy023tFRE2jmUIiU0 LUi/9BenEVlFGFoMMSXaP3b/Mndmrh0wx+NsPJXHaz+M2verJGgcPtaxpAJn4rJU5WmA +9gnD1m+WkxvMaFoHhTihZKgzDzXAPRqA49JpCo9knx4VHs3aDQHUB1PZoBTKJeeb8LG fdLuTtppif03Vnm1GDM7Gz8nvIug43nxUg+BVxmn6+Z5O84a4M/TejtK8E19rRuCJJVY LetOmgL0ASp9s4BuKSKavmDdBasWCHvRxZWhVBh+x5txlXCSTYGtmGk33tJnS/hcS/aP KeXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=JWsr5krGaAh5zk5p3jkSRk1l6yA/OJpF2OW6YBQgf8I=; b=iwfuaz14C10D6ShD4tb2/5M7cKpVBMi5sJYDMWKkRf2TTelq085tj0mXfrim1omflr s/kTZo+1tRirXdHSmECXL7rx21tbzO//KyvQk6sD6Sf9ukZjV1CIbJ3M08SGwSUADgca GvJduHaFpSgKZ4tXfuOsoQBuAW6HwF06jVQy2BhZTg66/RU7zwZDAuUEJL3EwuVt6ygy XXvDhynVQSbypeaStKLhn/NkySJSc5Jl+Pw2F4KPLah4IbtaxEJPAX40Bm+i27jyA4xV wrfjI+ezljTktaIl8Xchaj+qZitPE3V/EokBXlq2DOOGkOLcOfrzlYfeWCNfAzJA8iho kL8w== X-Gm-Message-State: APjAAAXYJUptNmZPdcQds7fRtj+T72ytpxCDsnFF+m1ctMcPkfRoE7g+ y35nl5o84PW0XGTNGqgZpXs7XccLrd1iMA== X-Google-Smtp-Source: APXvYqxH/hd4aOx5IoyYO2BsexO4eUHYZL6n/HtqWZEVk6IsZtQ4Ng03tmy8l0qNDFKarnpeR6q6Ig== X-Received: by 2002:a17:902:2983:: with SMTP id h3mr15069370plb.45.1562477336172; Sat, 06 Jul 2019 22:28:56 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.28.53 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:28:55 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:01 +0530 Message-Id: <20190707052831.9469-1-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 01/31] autobuild-run: introduce Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Various functions in the autobuild-run script use a lot of common data. To make it easier to work with, create a Builder class. For ease of review, this commit only introduces the Builder class but does not actually use it for anything. Subsequent patches will do that. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Fix indentation issues (suggested by Arnout) --- scripts/autobuild-run | 858 +++++++++++++++++++++--------------------- 1 file changed, 430 insertions(+), 428 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 601fb31..6bd6856 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -270,473 +270,474 @@ class SystemInfo: return not missing_requirements -def prepare_build(**kwargs): - """Prepare for the next build of the specified instance +class Builder: + def prepare_build(self, **kwargs): + """Prepare for the next build of the specified instance - This function prepares the build by making sure all the needed - directories are created, cloning or updating the Buildroot source - code, and cleaning up remaining stuff from previous builds. - """ + This function prepares the build by making sure all the needed + directories are created, cloning or updating the Buildroot source + code, and cleaning up remaining stuff from previous builds. + """ - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - - log_write(log, "INFO: preparing a new build") - - # Create the download directory if it doesn't exist - dldir = os.path.join(idir, "dl") - if not os.path.exists(dldir): - os.mkdir(dldir) - - # recursively find files under root - def find_files(root): - for r, d, f in os.walk(root): - # do not remove individual files from git caches. 'git' can - # be either dl//git or dl/git and we want to - # eventually remove tarballs for the git package, so check - # for '.git' instead to match only dl//git/.git . - if '.git' in d: - del d[:] - continue - for i in f: - yield os.path.join(r, i) - - # Remove 5 random files from the download directory. Removing - # random files from the download directory allows to ensure we - # regularly re-download files to check that their upstream - # location is still correct. - for i in range(0, 5): - flist = list(find_files(dldir)) - if not flist: - break - f = flist[randint(0, len(flist) - 1)] - log_write(log, "INFO: removing %s from downloads" % - os.path.relpath(f, dldir)) - os.remove(f) - - branch = get_branch() - log_write(log, "INFO: testing branch '%s'" % branch) - - # Clone Buildroot. This only happens if the source directory - # didn't exist already. - srcdir = os.path.join(idir, "buildroot") - if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", kwargs['repo'], srcdir], - stdout=log, stderr=log) + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] + + log_write(log, "INFO: preparing a new build") + + # Create the download directory if it doesn't exist + dldir = os.path.join(idir, "dl") + if not os.path.exists(dldir): + os.mkdir(dldir) + + # recursively find files under root + def find_files(root): + for r, d, f in os.walk(root): + # do not remove individual files from git caches. 'git' can + # be either dl//git or dl/git and we want to + # eventually remove tarballs for the git package, so check + # for '.git' instead to match only dl//git/.git . + if '.git' in d: + del d[:] + continue + for i in f: + yield os.path.join(r, i) + + # Remove 5 random files from the download directory. Removing + # random files from the download directory allows to ensure we + # regularly re-download files to check that their upstream + # location is still correct. + for i in range(0, 5): + flist = list(find_files(dldir)) + if not flist: + break + f = flist[randint(0, len(flist) - 1)] + log_write(log, "INFO: removing %s from downloads" % + os.path.relpath(f, dldir)) + os.remove(f) + + branch = get_branch() + log_write(log, "INFO: testing branch '%s'" % branch) + + # Clone Buildroot. This only happens if the source directory + # didn't exist already. + srcdir = os.path.join(idir, "buildroot") + if not os.path.exists(srcdir): + ret = subprocess.call(["git", "clone", kwargs['repo'], srcdir], + stdout=log, stderr=log) + if ret != 0: + log_write(log, "ERROR: could not clone Buildroot sources") + return -1 + + # Update the Buildroot sources. + abssrcdir = os.path.abspath(srcdir) + ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=log, stderr=log) if ret != 0: - log_write(log, "ERROR: could not clone Buildroot sources") + log_write(log, "ERROR: could not fetch Buildroot sources") return -1 - # Update the Buildroot sources. - abssrcdir = os.path.abspath(srcdir) - ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=log, stderr=log) - if ret != 0: - log_write(log, "ERROR: could not fetch Buildroot sources") - return -1 - - ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=log, stderr=log) - if ret != 0: - log_write(log, "ERROR: could not check out Buildroot sources") - return -1 - - # Create an empty output directory. We remove it first, in case a previous build was aborted. - outputdir = os.path.join(idir, "output") - if os.path.exists(outputdir): - # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", outputdir]) - os.mkdir(outputdir) - with open(os.path.join(outputdir, "branch"), "w") as branchf: - branchf.write(branch) - - return 0 - -def gen_config(**kwargs): - """Generate a new random configuration.""" - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - - log_write(log, "INFO: generate the configuration") - - if kwargs['debug']: - devnull = log - else: - devnull = open(os.devnull, "w") - - args = [os.path.join(srcdir, "utils/genrandconfig"), - "-o", outputdir, "-b", srcdir] - - toolchains_csv = kwargs['toolchains_csv'] - if toolchains_csv: - if not os.path.isabs(toolchains_csv): - toolchains_csv = os.path.join(srcdir, toolchains_csv) - args.extend(["--toolchains-csv", toolchains_csv]) - - ret = subprocess.call(args, stdout=devnull, stderr=log) - return ret - -def stop_on_build_hang(monitor_thread_hung_build_flag, - monitor_thread_stop_flag, - sub_proc, outputdir, log): - build_time_logfile = os.path.join(outputdir, "build/build-time.log") - while True: - if monitor_thread_stop_flag.is_set(): - return - if os.path.exists(build_time_logfile): - mtime = datetime.datetime.fromtimestamp(os.stat(build_time_logfile).st_mtime) - - if mtime < datetime.datetime.now() - datetime.timedelta(minutes=HUNG_BUILD_TIMEOUT): - if sub_proc.poll() is None: - monitor_thread_hung_build_flag.set() # Used by do_build() to determine build hang - log_write(log, "INFO: build hung") - sub_proc.kill() - break - monitor_thread_stop_flag.wait(30) + ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=log, stderr=log) + if ret != 0: + log_write(log, "ERROR: could not check out Buildroot sources") + return -1 -def check_reproducibility(**kwargs): - """Check reproducibility of builds + # Create an empty output directory. We remove it first, in case a previous build was aborted. + outputdir = os.path.join(idir, "output") + if os.path.exists(outputdir): + # shutil.rmtree doesn't remove write-protected files + subprocess.call(["rm", "-rf", outputdir]) + os.mkdir(outputdir) + with open(os.path.join(outputdir, "branch"), "w") as branchf: + branchf.write(branch) + + return 0 + + def gen_config(self, **kwargs): + """Generate a new random configuration.""" + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") - Use diffoscope on the built images, if diffoscope is not - installed, fallback to cmp - """ + log_write(log, "INFO: generate the configuration") - log = kwargs['log'] - idir = "instance-%d" % kwargs['instance'] - outputdir = os.path.join(idir, "output") - srcdir = os.path.join(idir, "buildroot") - reproducible_results = os.path.join(outputdir, "results", "reproducible_results") - # Using only tar images for now - build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") - build_2_image = os.path.join(outputdir, "images", "rootfs.tar") - - with open(reproducible_results, 'w') as diff: - if kwargs['sysinfo'].has("diffoscope"): - # Prefix to point diffoscope towards cross-tools - prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) - # Remove TARGET_CROSS= and \n from the string - prefix = prefix[13:-1] - log_write(log, "INFO: running diffoscope on images") - subprocess.call(["diffoscope", build_1_image, build_2_image, - "--tool-prefix-binutils", prefix], stdout=diff, stderr=log) + if kwargs['debug']: + devnull = log else: - log_write(log, "INFO: diffoscope not installed, falling back to cmp") - subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=log) - - if os.stat(reproducible_results).st_size > 0: - log_write(log, "INFO: Build is non-reproducible.") - return -1 - - # rootfs images match byte-for-byte -> reproducible image - log_write(log, "INFO: Build is reproducible!") - return 0 - -def do_build(**kwargs): - """Run the build itself""" - - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - nice = kwargs['nice'] - - # We need the absolute path to use with O=, because the relative - # path to the output directory here is not relative to the - # Buildroot sources, but to the location of the autobuilder - # script. - dldir = os.path.abspath(os.path.join(idir, "dl")) - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - f = open(os.path.join(outputdir, "logfile"), "w+") - log_write(log, "INFO: build started") - - cmd = ["nice", "-n", str(nice), - "make", "O=%s" % outputdir, - "-C", srcdir, "BR2_DL_DIR=%s" % dldir, - "BR2_JLEVEL=%s" % kwargs['njobs']] \ - + kwargs['make_opts'].split() - sub = subprocess.Popen(cmd, stdout=f, stderr=f) - - # Setup hung build monitoring thread - monitor_thread_hung_build_flag = Event() - monitor_thread_stop_flag = Event() - build_monitor = Thread(target=stop_on_build_hang, - args=(monitor_thread_hung_build_flag, - monitor_thread_stop_flag, - sub, outputdir, log)) - build_monitor.daemon = True - build_monitor.start() - - kwargs['buildpid'][kwargs['instance']] = sub.pid - ret = sub.wait() - kwargs['buildpid'][kwargs['instance']] = 0 - - # If build failed, monitor thread would have exited at this point - if monitor_thread_hung_build_flag.is_set(): - log_write(log, "INFO: build timed out [%d]" % ret) - return -2 - else: - # Stop monitor thread as this build didn't timeout - monitor_thread_stop_flag.set() - # Monitor thread should be exiting around this point - - if ret != 0: - log_write(log, "INFO: build failed [%d]" % ret) - return -1 - - cmd = ["make", "O=%s" % outputdir, "-C", srcdir, - "BR2_DL_DIR=%s" % dldir, "legal-info"] \ - + kwargs['make_opts'].split() - ret = subprocess.call(cmd, stdout=f, stderr=f) - if ret != 0: - log_write(log, "INFO: build failed during legal-info") - return -1 - log_write(log, "INFO: build successful") - return 0 - -def do_reproducible_build(**kwargs): - """Run the builds for reproducibility testing - - Build twice with the same configuration. Calls do_build() to - perform the actual build. - """ + devnull = open(os.devnull, "w") + + args = [os.path.join(srcdir, "utils/genrandconfig"), + "-o", outputdir, "-b", srcdir] - idir = "instance-%d" % kwargs['instance'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - log = kwargs['log'] + toolchains_csv = kwargs['toolchains_csv'] + if toolchains_csv: + if not os.path.isabs(toolchains_csv): + toolchains_csv = os.path.join(srcdir, toolchains_csv) + args.extend(["--toolchains-csv", toolchains_csv]) - # Start the first build - log_write(log, "INFO: Reproducible Build Test, starting build 1") - ret = do_build(**kwargs) - if ret != 0: - log_write(log, "INFO: build 1 failed, skipping build 2") + ret = subprocess.call(args, stdout=devnull, stderr=log) return ret - # First build has been built, move files and start build 2 - os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1")) + def stop_on_build_hang(self, monitor_thread_hung_build_flag, + monitor_thread_stop_flag, sub_proc, + outputdir, log): + build_time_logfile = os.path.join(outputdir, "build/build-time.log") + while True: + if monitor_thread_stop_flag.is_set(): + return + if os.path.exists(build_time_logfile): + mtime = datetime.datetime.fromtimestamp(os.stat(build_time_logfile).st_mtime) + + if mtime < datetime.datetime.now() - datetime.timedelta(minutes=HUNG_BUILD_TIMEOUT): + if sub_proc.poll() is None: + monitor_thread_hung_build_flag.set() # Used by do_build() to determine build hang + log_write(log, "INFO: build hung") + sub_proc.kill() + break + monitor_thread_stop_flag.wait(30) + + def check_reproducibility(self, **kwargs): + """Check reproducibility of builds + + Use diffoscope on the built images, if diffoscope is not + installed, fallback to cmp + """ - # Clean up build 1 - f = open(os.path.join(outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) + log = kwargs['log'] + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.join(idir, "output") + srcdir = os.path.join(idir, "buildroot") + reproducible_results = os.path.join(outputdir, "results", "reproducible_results") + # Using only tar images for now + build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") + build_2_image = os.path.join(outputdir, "images", "rootfs.tar") + + with open(reproducible_results, 'w') as diff: + if kwargs['sysinfo'].has("diffoscope"): + # Prefix to point diffoscope towards cross-tools + prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) + # Remove TARGET_CROSS= and \n from the string + prefix = prefix[13:-1] + log_write(log, "INFO: running diffoscope on images") + subprocess.call(["diffoscope", build_1_image, build_2_image, + "--tool-prefix-binutils", prefix], stdout=diff, stderr=log) + else: + log_write(log, "INFO: diffoscope not installed, falling back to cmp") + subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=log) - # Start the second build - log_write(log, "INFO: Reproducible Build Test, starting build 2") - ret = do_build(**kwargs) - if ret != 0: - log_write(log, "INFO: build 2 failed") - return ret + if os.stat(reproducible_results).st_size > 0: + log_write(log, "INFO: Build is non-reproducible.") + return -1 - # Assuming both have built successfully - ret = check_reproducibility(**kwargs) - return ret + # rootfs images match byte-for-byte -> reproducible image + log_write(log, "INFO: Build is reproducible!") + return 0 -def send_results(result, **kwargs): - """Prepare and store/send tarball with results + def do_build(self, **kwargs): + """Run the build itself""" - This function prepares the tarball with the results, and either - submits them to the official server (if the appropriate credentials - are available) or stores them locally as tarballs. - """ + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] + nice = kwargs['nice'] - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - resultdir = os.path.join(outputdir, "results") - - shutil.copyfile(os.path.join(outputdir, ".config"), - os.path.join(resultdir, "config")) - shutil.copyfile(os.path.join(outputdir, "defconfig"), - os.path.join(resultdir, "defconfig")) - shutil.copyfile(os.path.join(outputdir, "branch"), - os.path.join(resultdir, "branch")) - - def copy_if_exists(directory, src, dst=None): - if os.path.exists(os.path.join(outputdir, directory, src)): - shutil.copyfile(os.path.join(outputdir, directory, src), - os.path.join(resultdir, src if dst is None else dst)) - - copy_if_exists("build", "build-time.log") - copy_if_exists("build", "packages-file-list.txt") - copy_if_exists("build", "packages-file-list-host.txt") - copy_if_exists("build", "packages-file-list-staging.txt") - copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv") - - subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ - os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) - - # Return True if the result should be rejected, False otherwise - def reject_results(): - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() - - # Reject results where qemu-user refused to build - regexp = re.compile(r'^package/qemu/qemu.mk:.*Refusing to build qemu-user') - for line in lastlines: - if regexp.match(line): - return True - - return False - - if reject_results(): - return - - def get_failure_reason(): - # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() - - regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') - for line in lastlines: - m = regexp.search(line) - if m: - return m.group(1).rsplit('-', 1) - - # not found - return None + # We need the absolute path to use with O=, because the relative + # path to the output directory here is not relative to the + # Buildroot sources, but to the location of the autobuilder + # script. + dldir = os.path.abspath(os.path.join(idir, "dl")) + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") + f = open(os.path.join(outputdir, "logfile"), "w+") + log_write(log, "INFO: build started") + + cmd = ["nice", "-n", str(nice), + "make", "O=%s" % outputdir, + "-C", srcdir, "BR2_DL_DIR=%s" % dldir, + "BR2_JLEVEL=%s" % kwargs['njobs']] \ + + kwargs['make_opts'].split() + sub = subprocess.Popen(cmd, stdout=f, stderr=f) + + # Setup hung build monitoring thread + monitor_thread_hung_build_flag = Event() + monitor_thread_stop_flag = Event() + build_monitor = Thread(target=self.stop_on_build_hang, + args=(monitor_thread_hung_build_flag, + monitor_thread_stop_flag, + sub, outputdir, log)) + build_monitor.daemon = True + build_monitor.start() + + kwargs['buildpid'][kwargs['instance']] = sub.pid + ret = sub.wait() + kwargs['buildpid'][kwargs['instance']] = 0 + + # If build failed, monitor thread would have exited at this point + if monitor_thread_hung_build_flag.is_set(): + log_write(log, "INFO: build timed out [%d]" % ret) + return -2 + else: + # Stop monitor thread as this build didn't timeout + monitor_thread_stop_flag.set() + # Monitor thread should be exiting around this point - def extract_end_log(resultfile): - """Save the last part of the build log, starting from the failed package""" + if ret != 0: + log_write(log, "INFO: build failed [%d]" % ret) + return -1 - def extract_last_500_lines(): - subprocess.call(["tail -500 %s > %s" % \ - (os.path.join(outputdir, "logfile"), resultfile)], - shell=True) + cmd = ["make", "O=%s" % outputdir, "-C", srcdir, + "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + + kwargs['make_opts'].split() + ret = subprocess.call(cmd, stdout=f, stderr=f) + if ret != 0: + log_write(log, "INFO: build failed during legal-info") + return -1 + log_write(log, "INFO: build successful") + return 0 - reason = get_failure_reason() - if not reason: - extract_last_500_lines() - else: - f = open(os.path.join(outputdir, "logfile"), 'r') - mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) - mf.seek(0) - # Search for first action on the failed package - offset = mf.find(encode_str('>>> %s' % ' '.join(reason))) - if offset != -1: - with open(resultfile, "w") as endlog: - endlog.write(decode_bytes(mf[offset:])) - else: - # not found, use last 500 lines as fallback - extract_last_500_lines() + def do_reproducible_build(self, **kwargs): + """Run the builds for reproducibility testing - mf.close() - f.close() + Build twice with the same configuration. Calls do_build() to + perform the actual build. + """ - extract_end_log(os.path.join(resultdir, "build-end.log")) + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") + log = kwargs['log'] - def copy_config_log_files(): - """Recursively copy any config.log files from the failing package""" + # Start the first build + log_write(log, "INFO: Reproducible Build Test, starting build 1") + ret = self.do_build(**kwargs) + if ret != 0: + log_write(log, "INFO: build 1 failed, skipping build 2") + return ret - reason = get_failure_reason() - if not reason: - return + # First build has been built, move files and start build 2 + os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1")) - srcroot = os.path.join(outputdir, "build", '-'.join(reason)) - destroot = os.path.join(resultdir, '-'.join(reason)) - config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', - 'CMakeOutput.log') - - for root, dirs, files in os.walk(srcroot): - dest = os.path.join(destroot, os.path.relpath(root, srcroot)) - - for fname in files: - if fname in config_files: - if not os.path.exists(dest): - os.makedirs(dest) - shutil.copy(os.path.join(root, fname), os.path.join(dest, fname)) - - copy_config_log_files() - - resultf = open(os.path.join(resultdir, "status"), "w+") - if result == 0: - resultf.write("OK") - elif result == -1: - resultf.write("NOK") - elif result == -2: - resultf.write("TIMEOUT") - resultf.close() - - with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: - submitterf.write(kwargs['submitter']) - - # Yes, shutil.make_archive() would be nice, but it doesn't exist - # in Python 2.6. - ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) - if ret != 0: - log_write(log, "ERROR: could not make results tarball") - sys.exit(1) + # Clean up build 1 + f = open(os.path.join(outputdir, "logfile"), "w+") + subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) - if kwargs['upload']: - # Submit results. Yes, Python has some HTTP libraries, but - # none of the ones that are part of the standard library can - # upload a file without writing dozens of lines of code. - ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), - "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), - "-F", "uploadsubmit=1", - kwargs['http_url']], - stdout=log, stderr=log) + # Start the second build + log_write(log, "INFO: Reproducible Build Test, starting build 2") + ret = self.do_build(**kwargs) if ret != 0: - log_write(log, "INFO: results could not be submitted, %d" % ret) - else: - log_write(log, "INFO: results were submitted successfully") - else: - # No http login/password, keep tarballs locally - with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: - sha1 = hashlib.sha1(f.read()).hexdigest() - resultfilename = "instance-%d-%s.tar.bz2" % (kwargs['instance'], sha1) - os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) - log_write(log, "INFO: results saved as %s" % resultfilename) - -def run_instance(**kwargs): - """Main per-instance loop - - Prepare the build, generate a configuration, run the build, and submit the - results. - """ + log_write(log, "INFO: build 2 failed") + return ret - idir = "instance-%d" % kwargs['instance'] + # Assuming both have built successfully + ret = self.check_reproducibility(**kwargs) + return ret - # If it doesn't exist, create the instance directory - if not os.path.exists(idir): - os.mkdir(idir) + def send_results(self, result, **kwargs): + """Prepare and store/send tarball with results - if kwargs['debug']: - kwargs['log'] = sys.stdout - else: - kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") - log_write(kwargs['log'], "INFO: instance started") + This function prepares the tarball with the results, and either + submits them to the official server (if the appropriate credentials + are available) or stores them locally as tarballs. + """ - while True: - check_version() + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] - ret = prepare_build(**kwargs) - if ret != 0: - continue + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") + resultdir = os.path.join(outputdir, "results") + + shutil.copyfile(os.path.join(outputdir, ".config"), + os.path.join(resultdir, "config")) + shutil.copyfile(os.path.join(outputdir, "defconfig"), + os.path.join(resultdir, "defconfig")) + shutil.copyfile(os.path.join(outputdir, "branch"), + os.path.join(resultdir, "branch")) + + def copy_if_exists(directory, src, dst=None): + if os.path.exists(os.path.join(outputdir, directory, src)): + shutil.copyfile(os.path.join(outputdir, directory, src), + os.path.join(resultdir, src if dst is None else dst)) + + copy_if_exists("build", "build-time.log") + copy_if_exists("build", "packages-file-list.txt") + copy_if_exists("build", "packages-file-list-host.txt") + copy_if_exists("build", "packages-file-list-staging.txt") + copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv") + + subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ + os.path.join(resultdir, "gitid")], + shell=True, cwd=srcdir) + + # Return True if the result should be rejected, False otherwise + def reject_results(): + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() + + # Reject results where qemu-user refused to build + regexp = re.compile(r'^package/qemu/qemu.mk:.*Refusing to build qemu-user') + for line in lastlines: + if regexp.match(line): + return True + + return False + + if reject_results(): + return + + def get_failure_reason(): + # Output is a tuple (package, version), or None. + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() + + regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') + for line in lastlines: + m = regexp.search(line) + if m: + return m.group(1).rsplit('-', 1) + + # not found + return None + + def extract_end_log(resultfile): + """Save the last part of the build log, starting from the failed package""" - resultdir = os.path.join(idir, "output", "results") - os.mkdir(resultdir) + def extract_last_500_lines(): + subprocess.call(["tail -500 %s > %s" % \ + (os.path.join(outputdir, "logfile"), resultfile)], + shell=True) - ret = gen_config(**kwargs) + reason = get_failure_reason() + if not reason: + extract_last_500_lines() + else: + f = open(os.path.join(outputdir, "logfile"), 'r') + mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + mf.seek(0) + # Search for first action on the failed package + offset = mf.find(encode_str('>>> %s' % ' '.join(reason))) + if offset != -1: + with open(resultfile, "w") as endlog: + endlog.write(decode_bytes(mf[offset:])) + else: + # not found, use last 500 lines as fallback + extract_last_500_lines() + + mf.close() + f.close() + + extract_end_log(os.path.join(resultdir, "build-end.log")) + + def copy_config_log_files(): + """Recursively copy any config.log files from the failing package""" + + reason = get_failure_reason() + if not reason: + return + + srcroot = os.path.join(outputdir, "build", '-'.join(reason)) + destroot = os.path.join(resultdir, '-'.join(reason)) + config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', + 'CMakeOutput.log') + + for root, dirs, files in os.walk(srcroot): + dest = os.path.join(destroot, os.path.relpath(root, srcroot)) + + for fname in files: + if fname in config_files: + if not os.path.exists(dest): + os.makedirs(dest) + shutil.copy(os.path.join(root, fname), os.path.join(dest, fname)) + + copy_config_log_files() + + resultf = open(os.path.join(resultdir, "status"), "w+") + if result == 0: + resultf.write("OK") + elif result == -1: + resultf.write("NOK") + elif result == -2: + resultf.write("TIMEOUT") + resultf.close() + + with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: + submitterf.write(kwargs['submitter']) + + # Yes, shutil.make_archive() would be nice, but it doesn't exist + # in Python 2.6. + ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], + cwd=outputdir, stdout=log, stderr=log) if ret != 0: - log_write(kwargs['log'], "WARN: failed to generate configuration") - continue + log_write(log, "ERROR: could not make results tarball") + sys.exit(1) + + if kwargs['upload']: + # Submit results. Yes, Python has some HTTP libraries, but + # none of the ones that are part of the standard library can + # upload a file without writing dozens of lines of code. + ret = subprocess.call(["curl", "-u", + "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "-H", "Expect:", + "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadsubmit=1", + kwargs['http_url']], + stdout=log, stderr=log) + if ret != 0: + log_write(log, "INFO: results could not be submitted, %d" % ret) + else: + log_write(log, "INFO: results were submitted successfully") + else: + # No http login/password, keep tarballs locally + with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: + sha1 = hashlib.sha1(f.read()).hexdigest() + resultfilename = "instance-%d-%s.tar.bz2" % (kwargs['instance'], sha1) + os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) + log_write(log, "INFO: results saved as %s" % resultfilename) + + def run_instance(self, **kwargs): + """Main per-instance loop + + Prepare the build, generate a configuration, run the build, and submit the + results. + """ - # Check if the build test is supposed to be a reproducible test - outputdir = os.path.abspath(os.path.join(idir, "output")) - with open(os.path.join(outputdir, ".config"), "r") as fconf: - reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() - if reproducible: - ret = do_reproducible_build(**kwargs) + idir = "instance-%d" % kwargs['instance'] + + # If it doesn't exist, create the instance directory + if not os.path.exists(idir): + os.mkdir(idir) + + if kwargs['debug']: + kwargs['log'] = sys.stdout else: - ret = do_build(**kwargs) + kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") + log_write(kwargs['log'], "INFO: instance started") + + while True: + check_version() + + ret = self.prepare_build(**kwargs) + if ret != 0: + continue + + resultdir = os.path.join(idir, "output", "results") + os.mkdir(resultdir) + + ret = self.gen_config(**kwargs) + if ret != 0: + log_write(kwargs['log'], "WARN: failed to generate configuration") + continue + + # Check if the build test is supposed to be a reproducible test + outputdir = os.path.abspath(os.path.join(idir, "output")) + with open(os.path.join(outputdir, ".config"), "r") as fconf: + reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() + if reproducible: + ret = self.do_reproducible_build(**kwargs) + else: + ret = self.do_build(**kwargs) - send_results(ret, **kwargs) + self.send_results(ret, **kwargs) # args / config file merging inspired by: # https://github.com/docopt/docopt/blob/master/examples/config_file_example.py @@ -839,7 +840,8 @@ def main(): buildpid = multiprocessing.Array('i', int(args['--ninstances'])) processes = [] for i in range(0, int(args['--ninstances'])): - p = multiprocessing.Process(target=run_instance, kwargs=dict( + builder = Builder() + p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( instance = i, njobs = args['--njobs'], sysinfo = sysinfo, From patchwork Sun Jul 7 05:28:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128551 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="iFzh3LKo"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFS0xDBz9sCJ for ; Sun, 7 Jul 2019 15:29:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id BB44685BCC; Sun, 7 Jul 2019 05:29:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9PJ0XklzqkAx; Sun, 7 Jul 2019 05:29:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 2DF2785BBD; Sun, 7 Jul 2019 05:29:05 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 7A28D1BF5DD for ; Sun, 7 Jul 2019 05:29:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 72B4920422 for ; Sun, 7 Jul 2019 05:29:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uG9jj1VvL4wx for ; Sun, 7 Jul 2019 05:29:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f195.google.com (mail-pl1-f195.google.com [209.85.214.195]) by silver.osuosl.org (Postfix) with ESMTPS id E06B320414 for ; Sun, 7 Jul 2019 05:29:01 +0000 (UTC) Received: by mail-pl1-f195.google.com with SMTP id c14so6523884plo.0 for ; Sat, 06 Jul 2019 22:29:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=prAlPHMoWWUegR1oxXvixZESE5Qm3UdiM9wp5LmpHZE=; b=iFzh3LKoTrvHoBRnJbLJcnwmtPzpIZDjhkGEc3Ksy/9dgA9kQ8+fO/op5vAefhDiEW m4WcQd9eO6vU//lx8wLEfkj6Vu1bCWenO422NUDA+MrzanU1oV6haT0TTI+YvSDvJOdh kFx/yuhHvKUake8V46/UgJPmBBIyDetd19zHYuZXWbjGxD6LPH4TslShFAjfS8WgDN/N Y11SalAervj+WaD1vqxFCOZIqMht3cFkobrJtyY5ZsiHfOyOmXYdXaYFdqS1W81KpmHJ fpfS+qiTB1wXdzQgmqmMZWdhDNgP/qfVrz5xLEKaN3lDYxnFoMVZrOcyoKPYVdmdQnqW UN8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=prAlPHMoWWUegR1oxXvixZESE5Qm3UdiM9wp5LmpHZE=; b=tYtftwl7c6LhlW8foo3ETUdPqf5uJPUiYj7EV56zDeohTWdKOLqykrehbzi04/PC+B X3VHMrfwBpmhSY1vEgItGnlcfKkmQLoMdOT36/tsnNKSRlEVXuveVMZyHViz3uyjWfyl +rATNm/IyfghcM7i1Nv1VWMGJ3wytGt2VU8YGOvvRwmvqfBHNxYvIxjKixudtHR5eGNc WEbJPETXgUdHicl6f/8e9dA0M/FzOTQIUYIM7A2o55how5fxwv93FB3QLL9SGx50BlBO ADWniusuFNPEkSmUPYpoA23gUTcDyZMXlSejamFMUHJ2Bw+rXl5l7WDOBqBrqOHR4uu4 eBxg== X-Gm-Message-State: APjAAAWqPhF8lq82kToJ+ZGrtBwxPpbFN/h2M6iJR/JhKgRcjo6tj0vL nb37R9pepTSuNKJttrc5L8ab85ZXde7vkA== X-Google-Smtp-Source: APXvYqxq9H0VNPVOZOOYaXUVUzDZSnVjz0V+mIwHNt9LJWW+t0fZORlIcRvM5VYfSoLAMB2rViSYCw== X-Received: by 2002:a17:902:f095:: with SMTP id go21mr15333645plb.58.1562477340934; Sat, 06 Jul 2019 22:29:00 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.28.58 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:00 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:03 +0530 Message-Id: <20190707052831.9469-3-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 03/31] autobuild-run: move njobs from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 9946ddc..4d730ce 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -271,8 +271,9 @@ class SystemInfo: return not missing_requirements class Builder: - def __init__(self, instance): + def __init__(self, instance, njobs): self.instance = instance + self.njobs = njobs def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -455,7 +456,7 @@ class Builder: cmd = ["nice", "-n", str(nice), "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, - "BR2_JLEVEL=%s" % kwargs['njobs']] \ + "BR2_JLEVEL=%s" % self.njobs] \ + kwargs['make_opts'].split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) @@ -843,9 +844,10 @@ def main(): buildpid = multiprocessing.Array('i', int(args['--ninstances'])) processes = [] for i in range(0, int(args['--ninstances'])): - builder = Builder(instance = i) + builder = Builder( + instance = i, + njobs = args['--njobs']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - njobs = args['--njobs'], sysinfo = sysinfo, http_url = args['--http-url'], http_login = args['--http-login'], From patchwork Sun Jul 7 05:28:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128556 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="kNWuFwj7"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFd5754z9sCJ for ; Sun, 7 Jul 2019 15:29:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id BF675204E7; Sun, 7 Jul 2019 05:29:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VXMxKbnHFEiJ; Sun, 7 Jul 2019 05:29:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 778D920422; Sun, 7 Jul 2019 05:29:06 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id C2DB71BF5DD for ; Sun, 7 Jul 2019 05:29:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id BFC3587082 for ; Sun, 7 Jul 2019 05:29:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0a7LPDQAkokB for ; Sun, 7 Jul 2019 05:29:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 1D8E28702D for ; Sun, 7 Jul 2019 05:29:04 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id i18so6017840pgl.11 for ; Sat, 06 Jul 2019 22:29:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Sje947sqdPy118AFUppRM1f9qpvrxudkNpt1r5kPj3E=; b=kNWuFwj7G5+CsIrnstpTxCY8cIJM7Cky8oxt9+u+FNfUd+M9Z0XcOegL5jrfBKL0ts pR9P4CIGSa4ycmeM8oLuSxP8Xr0YMcKyQu9g+q7Z1mvrx6zJPDBwRh5DHFRkdyZDs8e7 ANMQrWGwdsLnvATwySopKcqCf+qlHukqfSz6vVdOguky66Dvi5cLChgcLauw+r1/5GJJ wUJmsVbN3cRmboaSfUZelogdTeIAeMSuv043t9t27t6b49HYocFjdA4/KD7HHbZ+dMjS 87E56sKESBvjRVAvPG5+cI7zXiMmVv11sAAvoz3kGTGao+ATkUf7RPBVdPF+ToK0DY79 RQwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Sje947sqdPy118AFUppRM1f9qpvrxudkNpt1r5kPj3E=; b=ro3BN+gxtJ3zF2RBrT1TVeL7LCec664/tyXJX0fZNNf8UGVAu2FswA0eqycySbh+XH TqI1j6aTPu0PvrdtZE/swIG96JltKtfmDyF26rUtVhh3jBfq417RR5TekrHvBGfHL6d4 mCkmtSFQPrJmGR3wEb/B8zPum/eVgvqMBhuLZcfaHOk4oige7Uo81ptdbcYq5TwUKGtS YALLssSJ1fff79Jps6OrFqur8Iniip5jmEc1kdqxk7D2mcrMxfv6xbMMqsZ4abgc5XcK K7iaDrgjeYGqJ/0UV7cGCnC2xvakdlBuVpuXbEEr2kEhLid/Uztt6mBLgq0hSQJMMYp0 FUYg== X-Gm-Message-State: APjAAAUu4ylaOL8vZaY1T8U1YzMq/TySJrIxpCr1waPscPkgChQLabdS t2zSzNbb2d7qv0rR2mHOsU9PAeMnsygnFA== X-Google-Smtp-Source: APXvYqxSw2RuxnYa8qbKVP0SYE9m2bW61Y1x8cPepvWa3nh3CZ2V6PicqXEvht/2ruyg0DloujiIpQ== X-Received: by 2002:a17:90a:37e9:: with SMTP id v96mr15066758pjb.10.1562477343319; Sat, 06 Jul 2019 22:29:03 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.01 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:02 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:04 +0530 Message-Id: <20190707052831.9469-4-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 04/31] autobuild-run: move sysinfo from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 4d730ce..ca2dd83 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -271,9 +271,10 @@ class SystemInfo: return not missing_requirements class Builder: - def __init__(self, instance, njobs): + def __init__(self, instance, njobs, sysinfo): self.instance = instance self.njobs = njobs + self.sysinfo = sysinfo def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -416,7 +417,7 @@ class Builder: build_2_image = os.path.join(outputdir, "images", "rootfs.tar") with open(reproducible_results, 'w') as diff: - if kwargs['sysinfo'].has("diffoscope"): + if self.sysinfo.has("diffoscope"): # Prefix to point diffoscope towards cross-tools prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string @@ -846,9 +847,9 @@ def main(): for i in range(0, int(args['--ninstances'])): builder = Builder( instance = i, - njobs = args['--njobs']) + njobs = args['--njobs'], + sysinfo = sysinfo) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - sysinfo = sysinfo, http_url = args['--http-url'], http_login = args['--http-login'], http_password = args['--http-password'], From patchwork Sun Jul 7 05:28:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128557 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="J3TUcuL9"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFg0wntz9s8m for ; Sun, 7 Jul 2019 15:29:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id A515985BB0; Sun, 7 Jul 2019 05:29:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1J7gY7DEqucz; Sun, 7 Jul 2019 05:29:24 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 5D0E985D26; Sun, 7 Jul 2019 05:29:24 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id BDB0A1BF5DD for ; Sun, 7 Jul 2019 05:29:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id B15FB204CB for ; Sun, 7 Jul 2019 05:29:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vPDlw6GRjc7c for ; Sun, 7 Jul 2019 05:29:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by silver.osuosl.org (Postfix) with ESMTPS id 84AB220424 for ; Sun, 7 Jul 2019 05:29:06 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id u17so5542927pgi.6 for ; Sat, 06 Jul 2019 22:29:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P9sJ6e5bXM0uINj4uQIubtTSWTPFMDjlaGi829MtpzI=; b=J3TUcuL9srRqV5sRB30H8i1MVv9Lwpodpvm4tkkeKtiuC7zYYHLj1U1lujz68OlTP7 9T0l9Lots7ZJmKKl7WtNjcwVPJs/BLsglbzR1nTJdSOyXzTOR8juV98sNCQlndoFBDbt zyU22HFzSB5QiahlLvTYtpm+t0IZV5LOTAd3hi9+WUSVReDs0DRdH1FLQNUVw4DzRy1Q sGP/+ceYh1BupBDGlIxqPKx873NcuifNIiVg0rLmBzDFKJ7bhjeANXeN17CzhJdMOoXv po8Z5i0eMlqohIkGKOZ2jGNp2Rw78cN/IYyc8SZoBC2KcldP/aQhbQNLNpcMkJxg5lrv 3ZZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P9sJ6e5bXM0uINj4uQIubtTSWTPFMDjlaGi829MtpzI=; b=sxhleKguIXKBzPkMVQwuauJZe/BCt3g7ZouTxWBP/7zeVsV6eN9cQVMBm3dOySbtHh Xg8Kf0zjcxG+Zr0KiBxOKwFcbAXflYAD513Iy4Kznwz7t/c8oKVmv+cF9q3J+Wa+HNud TcyKP/1R+t6T1PbnEzRmPf0AuhtEbDSESDCipUODqffMgHc+MG8Nay+n2+aJQ2VU5G4k dnUqNLfKDRZjuqHrqgCxMbQMS6S3LeSn5aSRXepqFk2uJiQBX1R+C03xbIXrw0zX9jVo 7E4jn76S4VYdXuEm0t+nofuAz31ETJ0zxz9/vO4nWPj0hoiYZgMI/Z0uCwiJBl0QOMPW 7rgw== X-Gm-Message-State: APjAAAXw+MvoEiONAbyBx4vJ6BFH99vwQDCWCkFQ0VQmUHjFTlV5nHXi zLr5A/RON79d4wtZJ3MUZqN5Rm4WGLQLIQ== X-Google-Smtp-Source: APXvYqyfahmqHOZjmWNpS9HOg91/KU4TNQY3yIQ2Y2r1BMee2wIjl5DTIlyJpQa/MrcV9S7AqRzv4Q== X-Received: by 2002:a65:5a42:: with SMTP id z2mr14759265pgs.421.1562477345672; Sat, 06 Jul 2019 22:29:05 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.03 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:05 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:05 +0530 Message-Id: <20190707052831.9469-5-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 05/31] autobuild-run: move http variables from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor arguments - Fix whitespace issues --- scripts/autobuild-run | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index ca2dd83..9d16ce5 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -271,10 +271,14 @@ class SystemInfo: return not missing_requirements class Builder: - def __init__(self, instance, njobs, sysinfo): + def __init__(self, instance, njobs, sysinfo, + http_url, http_login, http_password): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo + self.http_url = http_url + self.http_login = http_login + self.http_password = http_password def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -681,11 +685,11 @@ class Builder: # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "%s:%s" % (self.http_login, self.http_password), "-H", "Expect:", "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", - kwargs['http_url']], + self.http_url], stdout=log, stderr=log) if ret != 0: log_write(log, "INFO: results could not be submitted, %d" % ret) @@ -848,11 +852,11 @@ def main(): builder = Builder( instance = i, njobs = args['--njobs'], - sysinfo = sysinfo) + sysinfo = sysinfo, + http_url = args['--http-url'], + http_login = args['--http-login'], + http_password = args['--http-password']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - http_url = args['--http-url'], - http_login = args['--http-login'], - http_password = args['--http-password'], submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), From patchwork Sun Jul 7 05:28:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128554 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nB4jbpg9"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFW2WKWz9sN4 for ; Sun, 7 Jul 2019 15:29:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5224F84B08; Sun, 7 Jul 2019 05:29:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AARLh15b3fAa; Sun, 7 Jul 2019 05:29:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 30FAF844D3; Sun, 7 Jul 2019 05:29:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id CB20D1BF5DD for ; Sun, 7 Jul 2019 05:29:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 71AE085E12 for ; Sun, 7 Jul 2019 05:29:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ak7V1qoi6PKd for ; Sun, 7 Jul 2019 05:29:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by whitealder.osuosl.org (Postfix) with ESMTPS id 1A2E385BB0 for ; Sun, 7 Jul 2019 05:29:09 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id r1so5985068pfq.12 for ; Sat, 06 Jul 2019 22:29:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l+UJWpbEhZ0RPpaZ6Mt7yq58a9kGFHhrKIbZi1l5Blc=; b=nB4jbpg9Q+zZIHM4+w++n7/d0QPjia5wVrbp/J8wrUlL1IZtDCGAifu8JBY+hLXh2G 4g0jRInowXl9ImtZ6s2Qr2oQPTfEM4H9jNO9j94MXpts+L3YfJBizoLk7hgMynWBY9ze yEFbqvA/D8PJrCT7SKj7bqRxhYq0WbZlw3lOMcyk1RLSwBwlqyKSF7nnDYiEByUl4ZDo xE/80Bn+P4fZOojeCQjYccCvdNGEu2O7hjaNOdPpcOwOHP0UpfbOpRCEYOy+HdqUq6K1 r3/5K+KT1CikjEhT4nknUOMzWX4hbFQ6V15osLezCduLGduR6pf0MKkeefvAD/H/15sZ ivVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l+UJWpbEhZ0RPpaZ6Mt7yq58a9kGFHhrKIbZi1l5Blc=; b=G46BW9NRJYrQYqpLw15WoUnZNNt0c/zZj3j+bi1qlQ0SibIieJqi1X8NTwFMRB27F3 Z5N9/IJ34fpeD6d+muT7nwII3c2mn+pzjVI5Mn5gxHwf1OR6ATRAIihMJ5s58mzmOAbp Si1CCXe2IIfrodYjZj5dFIkqKb+O5PQxzqa3fnoc46ZdWGRhu7FdyXZNthwjozcUGbGV CxrL1XK/cc94z5UIPSKAji5NQUlkmWPWdzGZNm58/ec/krgeS8v01RIyiimQLt2mQOZr /GWc1+2HoCIeW8Kuz241MMdPuejBI9oTBYaGF2S6wukVVFGTdPkRnY4g/MZdPSwiRWZF GjHQ== X-Gm-Message-State: APjAAAV9NnRIBLhCKwRfic0Zns8zKrwosNnLPOUI7gO2ejOtrq6USDWx 1MeAyTmuk6qUu+56yfa67o40gFsj1rnwzQ== X-Google-Smtp-Source: APXvYqwNLflxB9gz98RBs0jQKu90cW19QSLcGak9rO0Is4QXH8D69tbT/YhUgFz6eXpNkR6yU8HnMg== X-Received: by 2002:a63:5a4b:: with SMTP id k11mr14495829pgm.143.1562477348108; Sat, 06 Jul 2019 22:29:08 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.06 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:07 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:06 +0530 Message-Id: <20190707052831.9469-6-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 06/31] autobuild-run: move submitter from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 9d16ce5..426078c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -272,13 +272,15 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, - http_url, http_login, http_password): + http_url, http_login, http_password, + submitter): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo self.http_url = http_url self.http_login = http_login self.http_password = http_password + self.submitter = submitter def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -670,7 +672,7 @@ class Builder: resultf.close() with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: - submitterf.write(kwargs['submitter']) + submitterf.write(self.submitter) # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. @@ -855,9 +857,9 @@ def main(): sysinfo = sysinfo, http_url = args['--http-url'], http_login = args['--http-login'], - http_password = args['--http-password']) + http_password = args['--http-password'], + submitter = args['--submitter']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], From patchwork Sun Jul 7 05:28:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128552 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="prY8fBCp"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFT6xMRz9s8m for ; Sun, 7 Jul 2019 15:29:17 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 223B485E14; Sun, 7 Jul 2019 05:29:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y+dCaKDOUdq3; Sun, 7 Jul 2019 05:29:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 0B6F485C10; Sun, 7 Jul 2019 05:29:15 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id E31A11BF956 for ; Sun, 7 Jul 2019 05:29:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id C8D8A85C10 for ; Sun, 7 Jul 2019 05:29:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kq0LBZ-Vd6ZF for ; Sun, 7 Jul 2019 05:29:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by whitealder.osuosl.org (Postfix) with ESMTPS id 475D485EF5 for ; Sun, 7 Jul 2019 05:29:11 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id b3so3220553plr.4 for ; Sat, 06 Jul 2019 22:29:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9hHjyKhUYI9YMqEyOMdWZ8nDZvWj3hjtKrBhVCUUuGE=; b=prY8fBCpyFwsVuYe8iZLJO1lYZRcEdJ/OqXlnecXUsDTO+O9spJGgQ/58fN/8Kqvco So48GP8TuxHhdihX9lhePvxkMkmOHilw3UYsF5+WYAUOYwydzJbW0nYxbnudvccGKFXe RMBuspsS0D0H8CYN4QRxuUfE8NPRWGV8ymxjtPeguqaKLZBpAnDYlKxl8vZbLl1xMmcq We7BUHRfi7L5xuQt38KLOAMv9jDqEvMZ9Z052mBZ+SNRMvWhDhdlE1jASxEQJ4Zk4ZvO u/1Q0O91tIDzlqg9Z0Jcq/sJXEnWpZuj8Haa2FZLUysdPx4NSDo1RQ0IqFhc+HYZ1H8E vr4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9hHjyKhUYI9YMqEyOMdWZ8nDZvWj3hjtKrBhVCUUuGE=; b=SnXvbA2CxpdMv2a1b8W/O8gbLCBgmrxHnrZmZPs2Oqrf5GxuTxWNnU2CDEPsgmb1Wf g1Za+aFL1DnR45h7gyX8fPVbPi/AP7hHT/fI7n1+G54Tdx3bOQ8IVprFgwwB4pP4AUfo wkIus02CfMHgv/z7YgdWlQjdMA64Fua9v4p3/XU/VTI/CFtYW0gZ+7BsF/Q8XHx/RsAt v91+X03Usr75lMFezvobGTzfZOp+I6Faq0J7xqI32FuF+XFBwR41pWVKijUwM3zuWn1T xQmGQXNefxL1Anarw/KrhxssmEv7CgkwfvlhSyLb4lrDzSMTF2gUH0WMk/xv5AKo3R1b LMVA== X-Gm-Message-State: APjAAAVDRzj9pciI3zI9hSIS2xV2HlRSXwbcY96wKrcOQuCWwDXp4AYu Jgo4hu05DpjCo1/P2UOCOw8XO9ooO73VqA== X-Google-Smtp-Source: APXvYqxzzShF3h2qOvh87bNxPtEcef0aRZtSGWwoVg2Z2lQAcz4OllPd5vKWnrT/PQBNYz+wSGZ99Q== X-Received: by 2002:a17:902:2808:: with SMTP id e8mr14450119plb.317.1562477350529; Sat, 06 Jul 2019 22:29:10 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.08 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:10 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:07 +0530 Message-Id: <20190707052831.9469-7-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 07/31] autobuild-run: move make_opts from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - New patch, split accidental squash of niceness move and make_opts move --- scripts/autobuild-run | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 426078c..c2d137c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,7 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter): + submitter, make_opts): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -281,6 +281,7 @@ class Builder: self.http_login = http_login self.http_password = http_password self.submitter = submitter + self.make_opts = make_opts def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -464,7 +465,7 @@ class Builder: "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % self.njobs] \ - + kwargs['make_opts'].split() + + self.make_opts.split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) # Setup hung build monitoring thread @@ -496,7 +497,7 @@ class Builder: cmd = ["make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "legal-info"] \ - + kwargs['make_opts'].split() + + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) if ret != 0: log_write(log, "INFO: build failed during legal-info") @@ -858,9 +859,9 @@ def main(): http_url = args['--http-url'], http_login = args['--http-login'], http_password = args['--http-password'], - submitter = args['--submitter']) + submitter = args['--submitter'], + make_opts = (args['--make-opts'] or '')) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], From patchwork Sun Jul 7 05:28:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128561 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="lnqXcAu9"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFm6Kmvz9sNj for ; Sun, 7 Jul 2019 15:29:32 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id E2B15204AE; Sun, 7 Jul 2019 05:29:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BU1k9ND8y3ju; Sun, 7 Jul 2019 05:29:22 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 3028B204DE; Sun, 7 Jul 2019 05:29:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 3D7B21BF5DD for ; Sun, 7 Jul 2019 05:29:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 3B1F3870DF for ; Sun, 7 Jul 2019 05:29:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DJ-fcoGrvXJZ for ; Sun, 7 Jul 2019 05:29:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by hemlock.osuosl.org (Postfix) with ESMTPS id A79388702D for ; Sun, 7 Jul 2019 05:29:13 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id s27so6034880pgl.2 for ; Sat, 06 Jul 2019 22:29:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TDaMfyGks+XpP3BmK5qQOdRgXsusVv8Th6xg2XSN6jk=; b=lnqXcAu937ouE0+4VhvvmYZ3HQuCl0b47AFWz77OL7nDBfKnINu7cRmunYzYp9o/Xj DSu8QzCMj2BFH4qwtqYZMqnrOvi3dmVupbZ10zvUxD0uq1dFdhHkkU8f4Isev80QU1ek JtLfvG+TfYlO8X9ELvvlLThpVz7CJ66rYuDeTuw1RvycoFMARtOVwsGmgHrS7PBOT4J6 7yrNGEUuGqX2Q+by+HGmV33CoQ4ICpEVemSzZrDK/ngvyEOEtKvWFa8+0vrrAM4Jr2/5 ueNk88FLmm1b411sii/VIiPY2xZwUP+ySf6mAfKT8EA7GjUFN4TgTDfeMkbqfIpguXNS Chzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TDaMfyGks+XpP3BmK5qQOdRgXsusVv8Th6xg2XSN6jk=; b=aKr1aJ3P1naK0f6gd4mLeuoNHW8jtCHrJDBJR+Y3hVVld7Cv+MjFoFDIWIpRj+k2pR 32Hc6rmYrq+qesEaQSRD3bATtu8WAeiEyBNH7EmKRSKnbWFGQv2VRzUmCa2Ct0ViXhFp 4Kn/RLGnw9zOV4CnGxszdrcCktlWiTl/aOJ/f/Wz9SYZjE/146FifFcsEJWsP7gOsL8T scelOlAqwTTnSNW6Q7AoMRPR31Gb/wDLKuvKr96nWQnaGaqgS1vGeAq5fUG186/tajwL v37areuvFI4uYYmGp5uSeHNX9L1sw2mWVpikiqd872DZyKlUP+SztbfEDmyFJcUd5c4t BKrA== X-Gm-Message-State: APjAAAV/FROWPyW7sKhUQASpPuYU8G11YjAT2NFG7SKYLl3TfKREMNyI 9JRXGdMp7Gzyk3JWxp06QEnKcBBBocGOww== X-Google-Smtp-Source: APXvYqysSSnl8S5mLJdf0g3AK0KTMjUo+lJTPykOSFDm+OQDtwp4HFuoBgeaJCq62kcsuHv7sAJNMg== X-Received: by 2002:a17:90a:cb87:: with SMTP id a7mr15840615pju.130.1562477352886; Sat, 06 Jul 2019 22:29:12 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.10 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:12 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:08 +0530 Message-Id: <20190707052831.9469-8-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 08/31] autobuild-run: move niceness from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument - Fix accidentally moving make_opts in the same patch --- scripts/autobuild-run | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index c2d137c..87d0bd8 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,7 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter, make_opts): + submitter, make_opts, nice): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -282,6 +282,7 @@ class Builder: self.http_password = http_password self.submitter = submitter self.make_opts = make_opts + self.nice = nice def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -449,7 +450,6 @@ class Builder: idir = "instance-%d" % self.instance log = kwargs['log'] - nice = kwargs['nice'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the @@ -461,7 +461,7 @@ class Builder: f = open(os.path.join(outputdir, "logfile"), "w+") log_write(log, "INFO: build started") - cmd = ["nice", "-n", str(nice), + cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % self.njobs] \ @@ -860,9 +860,9 @@ def main(): http_login = args['--http-login'], http_password = args['--http-password'], submitter = args['--submitter'], - make_opts = (args['--make-opts'] or '')) + make_opts = (args['--make-opts'] or ''), + nice = (args['--nice'] or 0)) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], upload = upload, From patchwork Sun Jul 7 05:28:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128563 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="spRQbwYb"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFy2tRkz9s8m for ; Sun, 7 Jul 2019 15:29:42 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id CBB7D20472; Sun, 7 Jul 2019 05:29:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CsOaTt7gIc1O; Sun, 7 Jul 2019 05:29:34 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id BB230204D7; Sun, 7 Jul 2019 05:29:33 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id C075B1BF5DD for ; Sun, 7 Jul 2019 05:29:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A65E1204DE for ; Sun, 7 Jul 2019 05:29:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QIxJKTHabc-8 for ; Sun, 7 Jul 2019 05:29:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by silver.osuosl.org (Postfix) with ESMTPS id 1D12E204DB for ; Sun, 7 Jul 2019 05:29:16 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id c14so6524015plo.0 for ; Sat, 06 Jul 2019 22:29:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5EftJHixxuhoHpi4E0UF9jhCseR78k9fvs4xIvsHehc=; b=spRQbwYbqWfMLF9q0kP5K4C5ZVaC31v1vKWo7e4l1RpCDpQF3en7jin6El6ywQbVTd j6v2lbUes+gthplFC89wlSkI423JMn82F4nMPEEkg8AY12x6fS04h9T1KUgzzsyE3/FQ WwIFjpIFjrV6ov0iMKeqRkhpOjvcfOFavyhWxnbBGCC/uF7FDMoKwhLucbDBdTdBxvkm kT4n+QJk7Z+5XbGZqpZnp+K8GwBiE7MU4qJV9yltPDLS1FNN8ZxoNm2mk66+45PTWN8G tMlTLG/X3H7u+2xQGch4F1/bSTzjHSvejlj9wcRUnyK5lk1cZAW5X2WN9PZFrMlZ7Wn5 3SuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5EftJHixxuhoHpi4E0UF9jhCseR78k9fvs4xIvsHehc=; b=o0/AqQk3sKNa144KHw4wYyI2PCtCYbG62STnGuNDRlRlMAG7S76JXcqlZebdTFixyl mC9Q/oNto95vCtQq8TDyAk4p6qufTIl2iw8lsYd7Jcef8smxJZzdayJ+Cr9u6EjZrOeI Tbrm/xUMuKuQpSuSA9LsQh8D2j1zpW3bojeFh8WsVznmjxfrWIa+N3N795f9x9nrt7HX 1FVekotvwDxXEOnyb8Ouf9MaN3FJjljxfTVg1dYkak1l4HXiN4V9a9i7oJD0ocGVrJ1M JB3dNwGszOcx493vjPYhrZ329pknXOtVoLJEXia3Y7rUEGpy/omIm1HTAFpHgqycuCUV HC7Q== X-Gm-Message-State: APjAAAXwouSuxMn958D9bu9cCOvFznWArzDlr3n5dBXDaP6HQu4/tniM q40m9Nn2+Bll/03XjzVhzKv9rLHPpvXfGA== X-Google-Smtp-Source: APXvYqyKFqNMsYMlTQN5E15i+PnHiORuF4wmftTeI7kZdJPtL5DMVDlyLSEK40EIc9XLGV0uYFqVVQ== X-Received: by 2002:a17:902:fe14:: with SMTP id g20mr14223815plj.54.1562477355328; Sat, 06 Jul 2019 22:29:15 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.13 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:14 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:09 +0530 Message-Id: <20190707052831.9469-9-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 09/31] autobuild-run: move toolchains_csv from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 87d0bd8..3bd10f7 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,7 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter, make_opts, nice): + submitter, make_opts, nice, toolchains_csv): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -283,6 +283,7 @@ class Builder: self.submitter = submitter self.make_opts = make_opts self.nice = nice + self.toolchains_csv = toolchains_csv def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -381,7 +382,7 @@ class Builder: args = [os.path.join(srcdir, "utils/genrandconfig"), "-o", outputdir, "-b", srcdir] - toolchains_csv = kwargs['toolchains_csv'] + toolchains_csv = self.toolchains_csv if toolchains_csv: if not os.path.isabs(toolchains_csv): toolchains_csv = os.path.join(srcdir, toolchains_csv) @@ -861,9 +862,9 @@ def main(): http_password = args['--http-password'], submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), - nice = (args['--nice'] or 0)) + nice = (args['--nice'] or 0), + toolchains_csv = args['--toolchains-csv']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], upload = upload, buildpid = buildpid, From patchwork Sun Jul 7 05:28:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128562 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OPYS0IXT"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFp6yjzz9s8m for ; Sun, 7 Jul 2019 15:29:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 4949B204EC; Sun, 7 Jul 2019 05:29:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ECfjZ1lKQvDD; Sun, 7 Jul 2019 05:29:26 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id D405C204E3; Sun, 7 Jul 2019 05:29:21 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 89D9D1BF956 for ; Sun, 7 Jul 2019 05:29:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8723D84506 for ; Sun, 7 Jul 2019 05:29:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p03KWuSUPsxm for ; Sun, 7 Jul 2019 05:29:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 8FF10844D3 for ; Sun, 7 Jul 2019 05:29:18 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id i8so6003794pgm.13 for ; Sat, 06 Jul 2019 22:29:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5o6PCskCymzBNT0CypYL0tOvHLU++nQ7F15CNgcXojs=; b=OPYS0IXTunM7BpfTOIkN4QK/F20vRTu4jg8vMCgP4YoT1akDztZK+/QRhGxFPwL6aA 3JM2A6BDHGmXA2RO7hFW3HngUBydncS22BbQ9wxf7gBusNoNkWb+ZTLEjzF3pG56BdLj 5UYkyaVhtjfdilsPBRLPASjD0/gRtm/DL2iiTsr8RFmJwifs6HRUEfzmHfxdWPQenVxE Jn8NKsOYw3jjpBBGltxyfEiTzTIPvln3s/nC+xmhaBRdTkobEfeQA4ZQYOWyNTtS0/sR emotzN0TWUdrZhIRD5Ab5C4vA5xkkl7ILGAG1ROgXKNZnGUupvzw00sgOUqZDmb3nkNx kMzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5o6PCskCymzBNT0CypYL0tOvHLU++nQ7F15CNgcXojs=; b=CQ2TLV/8lxwXvTQpX0QoO6SrTl6St50NVVatK7ayd4sKodrX2yR1sgqb16PN34jiRV fYESVCkdU/hdE7yhfFG39o4VX7TsoByjFVkCLIDb9Yl3CZsQuC+/KSg/jDGlEhyH96D3 HED8i2G1682N2glbQXrnKViFVKQA/5UpWegqIMRl4cJcIcQE6UDJUN1YCW1KHC5pYRo1 qb47ek6Q/gZ6kYUldNSCr4duAe5kX3IiDOK3OsNR559OnvggNkjbElzaW8nIbIpSCOZC 9NRWpV5eORFoRCrCjDxQ/4sGB7hHdtZ/Sg4ahRl/25f+OqI4e2hIt9kzmT4N6CLui4xt i8iA== X-Gm-Message-State: APjAAAXbVhfJLYdQya+xoKy/xIHVYCGo8Vz7R34kp1GQ7EDwdTdwl29e m5Rblfj5tubYymAI6qXNBQP1VzPpWxLmoQ== X-Google-Smtp-Source: APXvYqyg8nRjXcMeoXyBFIFH2atwz7rM0Xkvv9AddtyWGk2vHLVMtoq8+nylChd5tWvAvhwWTG2qPA== X-Received: by 2002:a65:498c:: with SMTP id r12mr14022315pgs.27.1562477357728; Sat, 06 Jul 2019 22:29:17 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.15 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:17 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:10 +0530 Message-Id: <20190707052831.9469-10-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 10/31] autobuild-run: move repo from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 3bd10f7..e64f6db 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,8 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter, make_opts, nice, toolchains_csv): + submitter, make_opts, nice, toolchains_csv, + repo): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -284,6 +285,7 @@ class Builder: self.make_opts = make_opts self.nice = nice self.toolchains_csv = toolchains_csv + self.repo = repo def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -336,7 +338,7 @@ class Builder: # didn't exist already. srcdir = os.path.join(idir, "buildroot") if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", kwargs['repo'], srcdir], + ret = subprocess.call(["git", "clone", self.repo, srcdir], stdout=log, stderr=log) if ret != 0: log_write(log, "ERROR: could not clone Buildroot sources") @@ -863,9 +865,9 @@ def main(): submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), - toolchains_csv = args['--toolchains-csv']) + toolchains_csv = args['--toolchains-csv'], + repo = args['--repo']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - repo = args['--repo'], upload = upload, buildpid = buildpid, debug = args['--debug'] From patchwork Sun Jul 7 05:28:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128555 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="uqXXo4Kt"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFd3Ks8z9s8m for ; Sun, 7 Jul 2019 15:29:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id F348C844D3; Sun, 7 Jul 2019 05:29:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Fvoi45MP5v0W; Sun, 7 Jul 2019 05:29:23 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6348F84506; Sun, 7 Jul 2019 05:29:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 8C5551BF5DD for ; Sun, 7 Jul 2019 05:29:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 893A184506 for ; Sun, 7 Jul 2019 05:29:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3Mf065DGaoot for ; Sun, 7 Jul 2019 05:29:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 173D4844D3 for ; Sun, 7 Jul 2019 05:29:21 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id z75so6029851pgz.5 for ; Sat, 06 Jul 2019 22:29:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GZxQocoqcj49NL44N8bS+rVXaJUZSla4XKb59boib3w=; b=uqXXo4Kt8GtceKuBxaub6pC9Duyap0d5VAftiRqL5Wak5NPatKbt3jvptYrjuoPvcl 1diIFyNMEhh5a7lcgOsRrMznk0ft9MBH9qyfOUEg5i63GrEiyFOabfI9msTokX7ASB8j q6fHx/zuFv2uDle5GEydutCRmawMwueRxNNQbTtTKNYcNcOky5+EWmdvinfw+wEk/Scz ku8Cfole/p4zfEaz+5aUNsudgnuJS+JMJALje4f2aZK55XJAakKVaNos+OteJRwdz/Un FW/HHy1+hK36fL4s6855VZF+z+SncbjTTT1cD6D8xLYKBdC58Ah9DLbIyEbW4BfpCX5G oHwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GZxQocoqcj49NL44N8bS+rVXaJUZSla4XKb59boib3w=; b=ASym8P65qcEBC+NXxufB1uKAMm5a2CmCjyMFXqkCFcq5KC726gDdZZ/5SVJRwwfZon kxAn8vpsfiJ3vMMJoxUPNqrpbqIhprbBkP8uwZ5Bw99cULTe6mC3Df9k1QlYOFgJGd78 QamQEaaGDKKVi7mSDSQ0T3y2SQCvSNJHjM/mWzdjl7kjRG64PgJt1tXG0g2wL6TbVb9w Z5aYaFPloMyd7k9NLc1jaZgDHqelKJRs258KJeaiEspOl4gk2vk7cY2Hirz3z8ouThah LIOTc3Dxhz7mPBALp9Z/zpxs3hQCeMVtKtqxwRKLJ7H3Rz3L872G3ZSVPisRwCBLJ0uY CYMg== X-Gm-Message-State: APjAAAUmYWN0rRNssG1BG3w/yqm1x9Qzgx3En29uap8FSQ5EGNnoVQ9L MbnqnY/fAVn/cpupi6CFmS2YD47nUziLJA== X-Google-Smtp-Source: APXvYqxzssHoOvrdR8V3qoXmH8LX7JoSZ1pwktFSA3bftUY5lRRmEQvsc+hZivRYa+rWPkoHcekgog== X-Received: by 2002:a63:69c1:: with SMTP id e184mr13947894pgc.198.1562477360242; Sat, 06 Jul 2019 22:29:20 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.18 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:19 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:11 +0530 Message-Id: <20190707052831.9469-11-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 11/31] autobuild-run: move upload variable from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index e64f6db..e217117 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -274,7 +274,7 @@ class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, submitter, make_opts, nice, toolchains_csv, - repo): + repo, upload): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -286,6 +286,7 @@ class Builder: self.nice = nice self.toolchains_csv = toolchains_csv self.repo = repo + self.upload = upload def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -686,7 +687,7 @@ class Builder: log_write(log, "ERROR: could not make results tarball") sys.exit(1) - if kwargs['upload']: + if self.upload: # Submit results. Yes, Python has some HTTP libraries, but # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. @@ -866,9 +867,9 @@ def main(): make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], - repo = args['--repo']) + repo = args['--repo'], + upload = upload) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - upload = upload, buildpid = buildpid, debug = args['--debug'] )) From patchwork Sun Jul 7 05:28:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128560 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aywlNyBT"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFm2jmYz9sN4 for ; Sun, 7 Jul 2019 15:29:32 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B25A785D26; Sun, 7 Jul 2019 05:29:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BnZCGtqmo-B1; Sun, 7 Jul 2019 05:29:26 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 2D7B085FEF; Sun, 7 Jul 2019 05:29:26 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 360C01BF5DD for ; Sun, 7 Jul 2019 05:29:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 3110F85F6D for ; Sun, 7 Jul 2019 05:29:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oIUVAyhWgM+s for ; Sun, 7 Jul 2019 05:29:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id A76E285CA1 for ; Sun, 7 Jul 2019 05:29:23 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id p10so6035319pgn.1 for ; Sat, 06 Jul 2019 22:29:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MHod2FHMLaBwSOLqciL+ZW/QXgkguLYbMOvAlz5xHWE=; b=aywlNyBTSlcTU0WF0g8mxA5fMapikZIG5Dsy8ThWMdwxe5yf3TRCBvrSVZg1OxmE4V jNd5xjL9beegWFA5vWhBPoXzAquWzsJBaczLOPe/DuAYD29XPRSbWhXyHboIkUNCsl9f c0N49LEvn+omn7zD/IpJC5zZ41egbdGlG1t72eSkgwYKA25xl787zFDxSul3zFUropw+ k2cb2FQMV0QmBsuG9RH2R5dAGTt+qInbzEdFtyqJrKJGdcmvk2K/yNAkY/DPgMarRHwp UAH6OSoFbVo73VpIBj4zDCQnQ9eDYT8C6I1XBGirHcyTv0dBBLA2W1N+6MvGoaG8VweX m3uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MHod2FHMLaBwSOLqciL+ZW/QXgkguLYbMOvAlz5xHWE=; b=CyYu74yq0TDtYpXuCI3A8sQH2ILpRFhcydp0USDwkZqm5ZDKyVBDfUXecg4kFAIkpB 5jOjbbW2K3HrKzL30yBCqw+j8MpPSyd6Mnw7RKG+7SLfzgOQ7NchmZk6/9kjDnTwwegV dY87Fcw12zYT1tgQ1UrA9qjUIy8Pg0o56JfDVDBuu7GSHNeOKUF+FZOXDOEnYg6pwCM+ qBji1YB+c+RLoY8Pay0D8gfBKrBx4o1am0KwfB9sVETBGbY4ISIMCMa6pGeGKn+B5f4Q XS33MvoKzHNGXxKXVxKSR19t+ic3a5r0ZSvDQvik6+IyfsmShzVVJaikdZMbdIrhasiV ggaA== X-Gm-Message-State: APjAAAVP+GDfdIi0jY/W2rjSs+XSCGk359uqNAkKtmrOFBybpC6m/WpB etfRrkr8JiTmGYp+8H8pvVUcpbuYtVXpIw== X-Google-Smtp-Source: APXvYqyRnUsbL0ITma9J0olA8DtWwzd8mQ5IQ7rU88Zk/A2qwjOE68H4TqVjSMaqEC/07g0zGPOpkQ== X-Received: by 2002:a63:fd0d:: with SMTP id d13mr14840566pgh.423.1562477362724; Sat, 06 Jul 2019 22:29:22 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.20 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:22 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:12 +0530 Message-Id: <20190707052831.9469-12-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 12/31] autobuild-run: move buildpid from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index e217117..05de063 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -274,7 +274,7 @@ class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, submitter, make_opts, nice, toolchains_csv, - repo, upload): + repo, upload, buildpid): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -287,6 +287,7 @@ class Builder: self.toolchains_csv = toolchains_csv self.repo = repo self.upload = upload + self.buildpid = buildpid def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -482,9 +483,9 @@ class Builder: build_monitor.daemon = True build_monitor.start() - kwargs['buildpid'][self.instance] = sub.pid + self.buildpid[self.instance] = sub.pid ret = sub.wait() - kwargs['buildpid'][self.instance] = 0 + self.buildpid[self.instance] = 0 # If build failed, monitor thread would have exited at this point if monitor_thread_hung_build_flag.is_set(): @@ -868,9 +869,9 @@ def main(): nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], - upload = upload) + upload = upload, + buildpid = buildpid) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - buildpid = buildpid, debug = args['--debug'] )) p.start() From patchwork Sun Jul 7 05:28:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128558 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DYQGBs6f"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFm15SDz9s8m for ; Sun, 7 Jul 2019 15:29:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5243987100; Sun, 7 Jul 2019 05:29:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AxBbBDKjuR5v; Sun, 7 Jul 2019 05:29:27 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id C36D687082; Sun, 7 Jul 2019 05:29:27 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 398E41BF5DD for ; Sun, 7 Jul 2019 05:29:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 321B185535 for ; Sun, 7 Jul 2019 05:29:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id L8WlmUFB_fdh for ; Sun, 7 Jul 2019 05:29:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by fraxinus.osuosl.org (Postfix) with ESMTPS id D602E84506 for ; Sun, 7 Jul 2019 05:29:25 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id 9so6532583ple.5 for ; Sat, 06 Jul 2019 22:29:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vFEF8xNwPY/nI4pGo1F9VQO3SGe4ZjXobS9tnyaybLM=; b=DYQGBs6f2m7owLTjA2mmtefu0gHf3UikkzQJmJZvtSSUJ1H1L5LCFIBTTb17+QxgrH rSHnTOrDB+OYasy+RV31upmUAU3MRCMQdIFUUtEadw9VAl8lvRflkgVsKpXXeH/JpjUK bbpBLLtWpLWIoj/Z2ESo+vU6n2cbWPZXgg1B9YinPaqbZ3U8Zlh8wYNlIRhz1tPdEpnP 1We7D/tmprS54awDeGx4dwIFgpegtuYKSFlSkKbAZxSXcgBY2yO1YSNGzotfpMYtuW9W 3AHYQDcA9jqnXVLgUh4wDf4WdSs18EKolFPdgpaQMtBzlkA5l6JZvx7TbXXKrvyvqak5 1jWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vFEF8xNwPY/nI4pGo1F9VQO3SGe4ZjXobS9tnyaybLM=; b=j9zM7REELKpATLt7OhH7tt2U+AMG/52NdzAp3vYG0zbr/VV7J+ota6aYWBuBlvr/uL PfMytFCzlXenvyyU8uiYKY+rmnIMJNZPPrMYjfQ3CPubG33MVjVrLLrt/5RXUjtbPo9Z BrNlT+q5BMrLrdA3Vu1IusH/L9PzOJ9hZcACQ//29tovhts0IkY6v/isHPkHfjLtmPP8 0Xvr0drNGQPhyctWDQLCh/yR91MguqbKyOOw1XKoWtJCA3ZmTQUeMU8TbzF9PRlsKmLy U8ctJKkSl+DAIurIxGe3JgAp0uNvIAgbxP6nkmo7wd4jApG+yeU31Bfsi5aJEu9LAv8u a2xg== X-Gm-Message-State: APjAAAVA3mtYxdT8q3WXf3wgmuEZp3x+wj3jK0d9LNXKqkWpFaOirIfk d6VR6vrC2rDpKYVzeuHxawYlm7io5EJxaA== X-Google-Smtp-Source: APXvYqx4uMjkZiHiMVS73aVBV/l4UyuvAOu2v255fwtBeC+vNYUVWPf5zRyMv1gb38/QdXNTo0Oi9w== X-Received: by 2002:a17:902:9a06:: with SMTP id v6mr14674643plp.71.1562477365099; Sat, 06 Jul 2019 22:29:25 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.23 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:24 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:13 +0530 Message-Id: <20190707052831.9469-13-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 13/31] autobuild-run: move debug from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 05de063..923652c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -274,7 +274,7 @@ class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, submitter, make_opts, nice, toolchains_csv, - repo, upload, buildpid): + repo, upload, buildpid, debug): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -288,6 +288,7 @@ class Builder: self.repo = repo self.upload = upload self.buildpid = buildpid + self.debug = debug def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -378,7 +379,7 @@ class Builder: log_write(log, "INFO: generate the configuration") - if kwargs['debug']: + if self.debug: devnull = log else: devnull = open(os.devnull, "w") @@ -724,7 +725,7 @@ class Builder: if not os.path.exists(idir): os.mkdir(idir) - if kwargs['debug']: + if self.debug: kwargs['log'] = sys.stdout else: kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") @@ -870,10 +871,9 @@ def main(): toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], upload = upload, - buildpid = buildpid) - p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - debug = args['--debug'] - )) + buildpid = buildpid, + debug = args['--debug']) + p = multiprocessing.Process(target=builder.run_instance) p.start() processes.append(p) From patchwork Sun Jul 7 05:28:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128559 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="YLOqCW2X"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFm1962z9sCJ for ; Sun, 7 Jul 2019 15:29:32 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B2EF88558F; Sun, 7 Jul 2019 05:29:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8cH9OWLm1a7m; Sun, 7 Jul 2019 05:29:29 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id DB20C850DC; Sun, 7 Jul 2019 05:29:29 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 3FBC61BF956 for ; Sun, 7 Jul 2019 05:29:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 35CF8850DC for ; Sun, 7 Jul 2019 05:29:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id U3L45hrs3gi5 for ; Sun, 7 Jul 2019 05:29:28 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 489D884506 for ; Sun, 7 Jul 2019 05:29:28 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id c73so1342702pfb.13 for ; Sat, 06 Jul 2019 22:29:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JULdsXfxnFfDm4CpNdKO1WAk6JndMs66hs4Oh5v+SLM=; b=YLOqCW2XhBfPZ127df0xqg1mh3v/djOLbd08ly3ipdyZ40KcEhYKl3c3zYOE5oWkA7 wqHBKiurdM7zh9g+Vg0QupC7Jjc9yyRAn5fAh52zd+RINU6wNss6yGLKfozq82y2LGaO zjd9t+XbE2r0/zviOsfeyLuoCxeG1hWcTdWCSiLsvKBwJby0fHjBNaS9wRTlF63hV3Tj 5fgSww1FTGh6dE1Q4+8uOtgsqaJAByGUyZft6NH4ULdXqle7kFz2RkgYMZiA8Ci/O/29 r41qViLFZlK9mIEK/w41gzXjP6rYtPT9MK5uJ+0AbTBh2EGrAX2rHfT6adfFKkBiMV3S VNzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JULdsXfxnFfDm4CpNdKO1WAk6JndMs66hs4Oh5v+SLM=; b=pTYXRWWsB5hgoRfwk8NsrnBPJkb+lM6KXaaoSznQ3uh5IxEOVYiaOTFeT3KsvZLEEK deN2TpLgzKXk0WObdqCEMtOGVrg2JY1Gp9af8iDpeYNheYmAn/pe0VMxEXJgsMy+/PUm eMoXpRf5FvTmdoyJnhOLMcOmfq7wY90qmRqwb7sRfPqTlMXXulYk/eUKiSKmq1Wl4mCy zrgLk+DuBLtr0Iv8XuICeyIT3SUW7dxbELletwcE194/OKdk1HmvNMiyByJ8ZK+2rnma ymA9xnVPCK4hN6h59LRjwNyrYCeVnZxKwdXM+kP9dmAZLBvOSqRQqM/MoYbRLDpPqQIg LGuA== X-Gm-Message-State: APjAAAVek57p5s5x6BArRQCG1tDG6bTTtZOY8TVlxSj4oHNR31glXtn4 ZLoW4u8OAnoHMCeNkGoUm2K5w4IryLRwsw== X-Google-Smtp-Source: APXvYqwCLJPLztFwU6XcWv4NBZRMOLRS3X29cGhR3jyhI0N1XEHXT3dK7gxVtZdC1fuR1nkBp33nJA== X-Received: by 2002:a17:90a:bd8c:: with SMTP id z12mr15438528pjr.60.1562477367427; Sat, 06 Jul 2019 22:29:27 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.25 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:27 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:14 +0530 Message-Id: <20190707052831.9469-14-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 14/31] autobuild-run: define instance directory as a part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" We need idir as a class member before moving log out of kwargs. Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 49 +++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 923652c..d1ecbb3 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -290,6 +290,9 @@ class Builder: self.buildpid = buildpid self.debug = debug + # frequently needed directories + self.idir = "instance-%d" % self.instance + def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -298,13 +301,12 @@ class Builder: code, and cleaning up remaining stuff from previous builds. """ - idir = "instance-%d" % self.instance log = kwargs['log'] log_write(log, "INFO: preparing a new build") # Create the download directory if it doesn't exist - dldir = os.path.join(idir, "dl") + dldir = os.path.join(self.idir, "dl") if not os.path.exists(dldir): os.mkdir(dldir) @@ -339,7 +341,7 @@ class Builder: # Clone Buildroot. This only happens if the source directory # didn't exist already. - srcdir = os.path.join(idir, "buildroot") + srcdir = os.path.join(self.idir, "buildroot") if not os.path.exists(srcdir): ret = subprocess.call(["git", "clone", self.repo, srcdir], stdout=log, stderr=log) @@ -360,7 +362,7 @@ class Builder: return -1 # Create an empty output directory. We remove it first, in case a previous build was aborted. - outputdir = os.path.join(idir, "output") + outputdir = os.path.join(self.idir, "output") if os.path.exists(outputdir): # shutil.rmtree doesn't remove write-protected files subprocess.call(["rm", "-rf", outputdir]) @@ -372,10 +374,9 @@ class Builder: def gen_config(self, **kwargs): """Generate a new random configuration.""" - idir = "instance-%d" % self.instance log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") log_write(log, "INFO: generate the configuration") @@ -422,9 +423,8 @@ class Builder: """ log = kwargs['log'] - idir = "instance-%d" % self.instance - outputdir = os.path.join(idir, "output") - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.join(self.idir, "output") + srcdir = os.path.join(self.idir, "buildroot") reproducible_results = os.path.join(outputdir, "results", "reproducible_results") # Using only tar images for now build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") @@ -454,16 +454,15 @@ class Builder: def do_build(self, **kwargs): """Run the build itself""" - idir = "instance-%d" % self.instance log = kwargs['log'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the # Buildroot sources, but to the location of the autobuilder # script. - dldir = os.path.abspath(os.path.join(idir, "dl")) - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + dldir = os.path.abspath(os.path.join(self.idir, "dl")) + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") f = open(os.path.join(outputdir, "logfile"), "w+") log_write(log, "INFO: build started") @@ -518,9 +517,8 @@ class Builder: perform the actual build. """ - idir = "instance-%d" % self.instance - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") log = kwargs['log'] # Start the first build @@ -556,11 +554,10 @@ class Builder: are available) or stores them locally as tarballs. """ - idir = "instance-%d" % self.instance log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") resultdir = os.path.join(outputdir, "results") shutil.copyfile(os.path.join(outputdir, ".config"), @@ -719,16 +716,14 @@ class Builder: results. """ - idir = "instance-%d" % self.instance - # If it doesn't exist, create the instance directory - if not os.path.exists(idir): - os.mkdir(idir) + if not os.path.exists(self.idir): + os.mkdir(self.idir) if self.debug: kwargs['log'] = sys.stdout else: - kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") + kwargs['log'] = open(os.path.join(self.idir, "instance.log"), "a+") log_write(kwargs['log'], "INFO: instance started") while True: @@ -738,7 +733,7 @@ class Builder: if ret != 0: continue - resultdir = os.path.join(idir, "output", "results") + resultdir = os.path.join(self.idir, "output", "results") os.mkdir(resultdir) ret = self.gen_config(**kwargs) @@ -747,7 +742,7 @@ class Builder: continue # Check if the build test is supposed to be a reproducible test - outputdir = os.path.abspath(os.path.join(idir, "output")) + outputdir = os.path.abspath(os.path.join(self.idir, "output")) with open(os.path.join(outputdir, ".config"), "r") as fconf: reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() if reproducible: From patchwork Sun Jul 7 05:28:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128569 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="oeqvTg27"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGC5wjRz9s8m for ; Sun, 7 Jul 2019 15:29:55 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id F39EB204E3; Sun, 7 Jul 2019 05:29:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fIMAlSUPnGJb; Sun, 7 Jul 2019 05:29:39 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 53043204F1; Sun, 7 Jul 2019 05:29:36 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 0E1801BF5DD for ; Sun, 7 Jul 2019 05:29:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 03CAC8560B for ; Sun, 7 Jul 2019 05:29:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ODeH9HAzKSpu for ; Sun, 7 Jul 2019 05:29:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id A0B6084506 for ; Sun, 7 Jul 2019 05:29:30 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id z75so6029922pgz.5 for ; Sat, 06 Jul 2019 22:29:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n8IeSIJyIy9uECnxTJSeftrSeRw3fFR3eyDIRrQ8ZNc=; b=oeqvTg270rrY7rA3WgMnx9g874g1dhEu1EcPoKyU03Fo0Ad/CnI1tv1DfnPF4fKETV bXteNsngsoT4HjOmhDxzxdMfUm87i7mx0PyqBvsmicAFxL3jpsYRoJ467Ql7wzW/dk+v p1HxFQ9sB9cbVtttuT075F0V5KZYk2eJCWRhs8FJvnBZ412kFzh8g+4UV4fbyStSpJsd iQNZAUdAWjXPR1SyxKJMEWd7uIhToZiGBXxH0fFcLeiL7Bqeq0Q1bs08u6snrAaUrGqH nyZxcIgArWxvGZsMI/To6+yl8/e2+DX1NI3fT3qNLx7KkLbwkoDGuEUKr+YCcjMsxpOv /+UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n8IeSIJyIy9uECnxTJSeftrSeRw3fFR3eyDIRrQ8ZNc=; b=jqQ7J8SSTiyttLTYdcLtwxFxpIcMnXchVOjCtXrVlggR9nq7AaTjwZ/9f7W8vJEm80 mut2j10T+CY4VD+kBZvgsZDni0vzLvbyX03tJogHt4yEo/TOvQm0TD5bdbFP/YH6VKw/ seU50DMcu6x60zcA6mLzbV3bZWdQCn6uO0KKf4R5vN3pYdhgsqgzdkRKSsyfC5xVLcLA H4MpOiFZr5nZczP5cEc5YjEmwKu7XvgO5vVqUrLGkGZk2+heORDsXgTVmq2biVeTWDc8 hcuPGVpDtPaYAWSBBMHTyrcRX7l//7uZuNyLr20gNgqIrmR2wUiTAJqjV5QSBRlBzdio 4klg== X-Gm-Message-State: APjAAAXASO8995C1oi/kyBCmaLUj2BwJHUeI/neiGicrh/Dwii7rdtXe VoC9EsTeWTb+KbC+efQxpGLGMM+6IBhhgg== X-Google-Smtp-Source: APXvYqySXQ1ZmXoGGIqvFW2QY8mZEg1O+rzy5tUdfaaEt4jvXxWtarNjK07zUz2/jy6sw+at6iulwA== X-Received: by 2002:a17:90a:22c6:: with SMTP id s64mr15617949pjc.5.1562477369800; Sat, 06 Jul 2019 22:29:29 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.27 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:29 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:15 +0530 Message-Id: <20190707052831.9469-15-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 15/31] autobuild-run: move log variable to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Fix whitespace errors --- scripts/autobuild-run | 94 ++++++++++++++++++++----------------------- 1 file changed, 43 insertions(+), 51 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index d1ecbb3..cb00745 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -293,6 +293,11 @@ class Builder: # frequently needed directories self.idir = "instance-%d" % self.instance + if self.debug: + self.log = sys.stdout + else: + self.log = open(os.path.join(self.idir, "instance.log"), "a+") + def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -301,9 +306,7 @@ class Builder: code, and cleaning up remaining stuff from previous builds. """ - log = kwargs['log'] - - log_write(log, "INFO: preparing a new build") + log_write(self.log, "INFO: preparing a new build") # Create the download directory if it doesn't exist dldir = os.path.join(self.idir, "dl") @@ -332,33 +335,33 @@ class Builder: if not flist: break f = flist[randint(0, len(flist) - 1)] - log_write(log, "INFO: removing %s from downloads" % + log_write(self.log, "INFO: removing %s from downloads" % os.path.relpath(f, dldir)) os.remove(f) branch = get_branch() - log_write(log, "INFO: testing branch '%s'" % branch) + log_write(self.log, "INFO: testing branch '%s'" % branch) # Clone Buildroot. This only happens if the source directory # didn't exist already. srcdir = os.path.join(self.idir, "buildroot") if not os.path.exists(srcdir): ret = subprocess.call(["git", "clone", self.repo, srcdir], - stdout=log, stderr=log) + stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not clone Buildroot sources") + log_write(self.log, "ERROR: could not clone Buildroot sources") return -1 # Update the Buildroot sources. abssrcdir = os.path.abspath(srcdir) - ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=log, stderr=log) + ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not fetch Buildroot sources") + log_write(self.log, "ERROR: could not fetch Buildroot sources") return -1 - ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=log, stderr=log) + ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not check out Buildroot sources") + log_write(self.log, "ERROR: could not check out Buildroot sources") return -1 # Create an empty output directory. We remove it first, in case a previous build was aborted. @@ -374,14 +377,13 @@ class Builder: def gen_config(self, **kwargs): """Generate a new random configuration.""" - log = kwargs['log'] outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") - log_write(log, "INFO: generate the configuration") + log_write(self.log, "INFO: generate the configuration") if self.debug: - devnull = log + devnull = self.log else: devnull = open(os.devnull, "w") @@ -394,12 +396,12 @@ class Builder: toolchains_csv = os.path.join(srcdir, toolchains_csv) args.extend(["--toolchains-csv", toolchains_csv]) - ret = subprocess.call(args, stdout=devnull, stderr=log) + ret = subprocess.call(args, stdout=devnull, stderr=self.log) return ret def stop_on_build_hang(self, monitor_thread_hung_build_flag, monitor_thread_stop_flag, sub_proc, - outputdir, log): + outputdir): build_time_logfile = os.path.join(outputdir, "build/build-time.log") while True: if monitor_thread_stop_flag.is_set(): @@ -410,7 +412,7 @@ class Builder: if mtime < datetime.datetime.now() - datetime.timedelta(minutes=HUNG_BUILD_TIMEOUT): if sub_proc.poll() is None: monitor_thread_hung_build_flag.set() # Used by do_build() to determine build hang - log_write(log, "INFO: build hung") + log_write(self.log, "INFO: build hung") sub_proc.kill() break monitor_thread_stop_flag.wait(30) @@ -422,7 +424,6 @@ class Builder: installed, fallback to cmp """ - log = kwargs['log'] outputdir = os.path.join(self.idir, "output") srcdir = os.path.join(self.idir, "buildroot") reproducible_results = os.path.join(outputdir, "results", "reproducible_results") @@ -436,26 +437,24 @@ class Builder: prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string prefix = prefix[13:-1] - log_write(log, "INFO: running diffoscope on images") + log_write(self.log, "INFO: running diffoscope on images") subprocess.call(["diffoscope", build_1_image, build_2_image, - "--tool-prefix-binutils", prefix], stdout=diff, stderr=log) + "--tool-prefix-binutils", prefix], stdout=diff, stderr=self.log) else: - log_write(log, "INFO: diffoscope not installed, falling back to cmp") - subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=log) + log_write(self.log, "INFO: diffoscope not installed, falling back to cmp") + subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=self.log) if os.stat(reproducible_results).st_size > 0: - log_write(log, "INFO: Build is non-reproducible.") + log_write(self.log, "INFO: Build is non-reproducible.") return -1 # rootfs images match byte-for-byte -> reproducible image - log_write(log, "INFO: Build is reproducible!") + log_write(self.log, "INFO: Build is reproducible!") return 0 def do_build(self, **kwargs): """Run the build itself""" - log = kwargs['log'] - # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the # Buildroot sources, but to the location of the autobuilder @@ -464,7 +463,7 @@ class Builder: outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") f = open(os.path.join(outputdir, "logfile"), "w+") - log_write(log, "INFO: build started") + log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, @@ -479,7 +478,7 @@ class Builder: build_monitor = Thread(target=self.stop_on_build_hang, args=(monitor_thread_hung_build_flag, monitor_thread_stop_flag, - sub, outputdir, log)) + sub, outputdir)) build_monitor.daemon = True build_monitor.start() @@ -489,7 +488,7 @@ class Builder: # If build failed, monitor thread would have exited at this point if monitor_thread_hung_build_flag.is_set(): - log_write(log, "INFO: build timed out [%d]" % ret) + log_write(self.log, "INFO: build timed out [%d]" % ret) return -2 else: # Stop monitor thread as this build didn't timeout @@ -497,7 +496,7 @@ class Builder: # Monitor thread should be exiting around this point if ret != 0: - log_write(log, "INFO: build failed [%d]" % ret) + log_write(self.log, "INFO: build failed [%d]" % ret) return -1 cmd = ["make", "O=%s" % outputdir, "-C", srcdir, @@ -505,9 +504,9 @@ class Builder: + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) if ret != 0: - log_write(log, "INFO: build failed during legal-info") + log_write(self.log, "INFO: build failed during legal-info") return -1 - log_write(log, "INFO: build successful") + log_write(self.log, "INFO: build successful") return 0 def do_reproducible_build(self, **kwargs): @@ -519,13 +518,12 @@ class Builder: outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") - log = kwargs['log'] # Start the first build - log_write(log, "INFO: Reproducible Build Test, starting build 1") + log_write(self.log, "INFO: Reproducible Build Test, starting build 1") ret = self.do_build(**kwargs) if ret != 0: - log_write(log, "INFO: build 1 failed, skipping build 2") + log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret # First build has been built, move files and start build 2 @@ -536,10 +534,10 @@ class Builder: subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) # Start the second build - log_write(log, "INFO: Reproducible Build Test, starting build 2") + log_write(self.log, "INFO: Reproducible Build Test, starting build 2") ret = self.do_build(**kwargs) if ret != 0: - log_write(log, "INFO: build 2 failed") + log_write(self.log, "INFO: build 2 failed") return ret # Assuming both have built successfully @@ -554,8 +552,6 @@ class Builder: are available) or stores them locally as tarballs. """ - log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") resultdir = os.path.join(outputdir, "results") @@ -681,9 +677,9 @@ class Builder: # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) + cwd=outputdir, stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not make results tarball") + log_write(self.log, "ERROR: could not make results tarball") sys.exit(1) if self.upload: @@ -696,18 +692,18 @@ class Builder: "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", self.http_url], - stdout=log, stderr=log) + stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "INFO: results could not be submitted, %d" % ret) + log_write(self.log, "INFO: results could not be submitted, %d" % ret) else: - log_write(log, "INFO: results were submitted successfully") + log_write(self.log, "INFO: results were submitted successfully") else: # No http login/password, keep tarballs locally with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: sha1 = hashlib.sha1(f.read()).hexdigest() resultfilename = "instance-%d-%s.tar.bz2" % (self.instance, sha1) os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) - log_write(log, "INFO: results saved as %s" % resultfilename) + log_write(self.log, "INFO: results saved as %s" % resultfilename) def run_instance(self, **kwargs): """Main per-instance loop @@ -720,11 +716,7 @@ class Builder: if not os.path.exists(self.idir): os.mkdir(self.idir) - if self.debug: - kwargs['log'] = sys.stdout - else: - kwargs['log'] = open(os.path.join(self.idir, "instance.log"), "a+") - log_write(kwargs['log'], "INFO: instance started") + log_write(self.log, "INFO: instance started") while True: check_version() @@ -738,7 +730,7 @@ class Builder: ret = self.gen_config(**kwargs) if ret != 0: - log_write(kwargs['log'], "WARN: failed to generate configuration") + log_write(self.log, "WARN: failed to generate configuration") continue # Check if the build test is supposed to be a reproducible test From patchwork Sun Jul 7 05:28:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128564 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="c6sRrYEm"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHFz6KSBz9s8m for ; Sun, 7 Jul 2019 15:29:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 5C58086004; Sun, 7 Jul 2019 05:29:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id X-j+TF-AbQbS; Sun, 7 Jul 2019 05:29:37 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id C490E85CA1; Sun, 7 Jul 2019 05:29:37 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 8D4371BF5DD for ; Sun, 7 Jul 2019 05:29:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8140884506 for ; Sun, 7 Jul 2019 05:29:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IUn6jAl9GDNj for ; Sun, 7 Jul 2019 05:29:32 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by fraxinus.osuosl.org (Postfix) with ESMTPS id E770E854FC for ; Sun, 7 Jul 2019 05:29:32 +0000 (UTC) Received: by mail-pg1-f196.google.com with SMTP id l21so2023926pgm.3 for ; Sat, 06 Jul 2019 22:29:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Pm3yZoBcuycz57fAm4ovvVjemf6JsU9dWM0c92PGE/0=; b=c6sRrYEmh4ZSYZwTChC+dCaod14SXn0JUPDsiUIFQKLLKg+qbntTW+VQKNafnTjMjD tUppv+dBUJrTccuP7PKEhwEFkbIyx4e8Or/p3yZr88iWJVLRegCMDDIeNoHFI/Uk9pmt Y/GNEXYKhr50JFRxharfTk/zis0HS8S8J3zhwxs3SDtlSCecCUBNAVpEs1VMvA+1QUlc +f4xfZgS9eQ2UYEoS+s68R/9WiF43J16v+Gy2GMuAGyVZXbbeOhZKj74olduND4joO5c y8tZHI9QEj5EZjEpmkVGQojydMq1OT3Psp7x6EECvGNgTWo23SqzUhJv44MSGfRqvnPM Jm2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Pm3yZoBcuycz57fAm4ovvVjemf6JsU9dWM0c92PGE/0=; b=Na2sdIaiXFK/GDqXwfaDdfK6uT+PTrRKWQ15jo+C1GMmLljyrBXz/z4Ao/hLshUMcJ Y5p3stKtWv0H8LReYBGpbmOGcJRS+QE1zGJXe3EzR5UPIAN9KTN5pODvET+QjYMEHBTP LbqfPWURwg0vpO0HV3qR53WbnGbrIclx8m5JchlGi27/+h/NvwyysYcGRzlE5eRAJdTX GuFFVRx4SB3VbC8orsYwMKomSPuzpxYSG2WmLnS6zICN26TWsqL/3xBxz6X/zQDkxa7W iWbd1EGTwYScm0Visy94jiYYUYn3cro5VCBggGlbAku8iGqTa5XxM3wUDQW+kuOHdH3A cPig== X-Gm-Message-State: APjAAAVLG3AsJZRzSBPQIBUwPP9yhJuef6lsFjHurbfOE1YiQegzwaZV bb2AYfI2/16DC+DJlRzwQRV65yzNSmAsgg== X-Google-Smtp-Source: APXvYqy7vnb5gzIwV0lIAOWZQ/Y4yoAB2VdbMAZ9HAFjam9xKIZG/lnwZtteICV5sZNe7HTNAFDMuw== X-Received: by 2002:a17:90a:8a91:: with SMTP id x17mr15400586pjn.95.1562477372108; Sat, 06 Jul 2019 22:29:32 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.30 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:31 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:16 +0530 Message-Id: <20190707052831.9469-16-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 16/31] autobuild-run: remove kwargs argument from function calls and definitions X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index cb00745..adc8795 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -298,7 +298,7 @@ class Builder: else: self.log = open(os.path.join(self.idir, "instance.log"), "a+") - def prepare_build(self, **kwargs): + def prepare_build(self): """Prepare for the next build of the specified instance This function prepares the build by making sure all the needed @@ -375,7 +375,7 @@ class Builder: return 0 - def gen_config(self, **kwargs): + def gen_config(self): """Generate a new random configuration.""" outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") @@ -417,7 +417,7 @@ class Builder: break monitor_thread_stop_flag.wait(30) - def check_reproducibility(self, **kwargs): + def check_reproducibility(self): """Check reproducibility of builds Use diffoscope on the built images, if diffoscope is not @@ -452,7 +452,7 @@ class Builder: log_write(self.log, "INFO: Build is reproducible!") return 0 - def do_build(self, **kwargs): + def do_build(self): """Run the build itself""" # We need the absolute path to use with O=, because the relative @@ -509,7 +509,7 @@ class Builder: log_write(self.log, "INFO: build successful") return 0 - def do_reproducible_build(self, **kwargs): + def do_reproducible_build(self): """Run the builds for reproducibility testing Build twice with the same configuration. Calls do_build() to @@ -521,7 +521,7 @@ class Builder: # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") - ret = self.do_build(**kwargs) + ret = self.do_build() if ret != 0: log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret @@ -535,16 +535,16 @@ class Builder: # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") - ret = self.do_build(**kwargs) + ret = self.do_build() if ret != 0: log_write(self.log, "INFO: build 2 failed") return ret # Assuming both have built successfully - ret = self.check_reproducibility(**kwargs) + ret = self.check_reproducibility() return ret - def send_results(self, result, **kwargs): + def send_results(self, result): """Prepare and store/send tarball with results This function prepares the tarball with the results, and either @@ -705,7 +705,7 @@ class Builder: os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) log_write(self.log, "INFO: results saved as %s" % resultfilename) - def run_instance(self, **kwargs): + def run_instance(self): """Main per-instance loop Prepare the build, generate a configuration, run the build, and submit the @@ -721,14 +721,14 @@ class Builder: while True: check_version() - ret = self.prepare_build(**kwargs) + ret = self.prepare_build() if ret != 0: continue resultdir = os.path.join(self.idir, "output", "results") os.mkdir(resultdir) - ret = self.gen_config(**kwargs) + ret = self.gen_config() if ret != 0: log_write(self.log, "WARN: failed to generate configuration") continue @@ -738,11 +738,11 @@ class Builder: with open(os.path.join(outputdir, ".config"), "r") as fconf: reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() if reproducible: - ret = self.do_reproducible_build(**kwargs) + ret = self.do_reproducible_build() else: - ret = self.do_build(**kwargs) + ret = self.do_build() - self.send_results(ret, **kwargs) + self.send_results(ret) # args / config file merging inspired by: # https://github.com/docopt/docopt/blob/master/examples/config_file_example.py From patchwork Sun Jul 7 05:28:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128566 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="IkMmUbzY"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHG40LKmz9s8m for ; Sun, 7 Jul 2019 15:29:48 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 88488812E6; Sun, 7 Jul 2019 05:29:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0aA5Ja0nwvah; Sun, 7 Jul 2019 05:29:40 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id A078F85F6D; Sun, 7 Jul 2019 05:29:38 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 02F7F1BF5DD for ; Sun, 7 Jul 2019 05:29:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id F3FD785F6D for ; Sun, 7 Jul 2019 05:29:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tvM+GpwEFnSv for ; Sun, 7 Jul 2019 05:29:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by whitealder.osuosl.org (Postfix) with ESMTPS id 483F985BCC for ; Sun, 7 Jul 2019 05:29:35 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id t16so5993741pfe.11 for ; Sat, 06 Jul 2019 22:29:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EgOLzDsviE0OUUW1xbY5ID+MsgXqQYItfBVJHkjMr8Q=; b=IkMmUbzYF1hu3MmePBIjSZTj4sM3mFuYMetgZr1y2Hl7/yRxNon9AYSMhkts5eb88T uDFMG3l1wqgjz1HOa/M6mj2ii32XmoipHT6x/uTiPBPc2955/X7NHetVNIqnnr6apMuF qJBe2rWjl11k9fxGU30Y6Ge5MqAPo4PdlHQv2qOYMOeMZ+vk1w9/8PcJWQvylgSRi3Wa K3tDUIQSxIKtz1Ag9hkneb494/u4U1zk+GbVxQ8xECzQZghptVwXRngWCs0CZ/7hMJDn CcnvXZkJ4Cpp6DubCWN8354I88e0dkGiBZMS2RNMsexjruZessHd6OEKezpaUE+FNGfH oUIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EgOLzDsviE0OUUW1xbY5ID+MsgXqQYItfBVJHkjMr8Q=; b=AIpWz1v3AIH8DggUpZDlaWuVYuSV6ymzQh+RrRnJyVSR12pQieTTZTmg/1I4DAuxKE 3nGhyYR7nufA9/mWsG9Skrzxl2Gea+fEtuu6o3LvFT7Y/aYLITYkDDUr6l+XyikMNttp mSoFKi0JtFQDA5DTKNgMQHVvtcwgtgw3kD/e7iwasnTh7O5731SzHdzZ/HJmnYqXLkMM lExn3S51odEjUlErBxT5grlMv3s5RhKGeQ3bodA80E471FuhwwIwvY1D4ApT18BeSyoc vPM4FUj05WDIaN9q9eIbyawMCAj4qVp2jNY5oOwcYVG8rhs3EAbiIo3Cui0gglY3P8KZ GJuw== X-Gm-Message-State: APjAAAU6FF9CmXyS4e4egVuBHMcT1iJlkYABl0WdOkwNb/VX6PmP4c6R Ursl5bTiScniSxeunRSM+/9WjlfjlHai4w== X-Google-Smtp-Source: APXvYqxu98QiydJ7lpvd7tREXbnkKubdsjSDuMIQgrAitdyZz8x1Kc2qh/RU0THhQoy3HUDNYERMRw== X-Received: by 2002:a17:90a:109:: with SMTP id b9mr14660934pjb.112.1562477374443; Sat, 06 Jul 2019 22:29:34 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.32 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:34 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:17 +0530 Message-Id: <20190707052831.9469-17-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 17/31] autobuild-run: define source directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Fix whitespace issue --- scripts/autobuild-run | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index adc8795..741d316 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -292,6 +292,7 @@ class Builder: # frequently needed directories self.idir = "instance-%d" % self.instance + self.srcdir = os.path.join(self.idir, "buildroot") if self.debug: self.log = sys.stdout @@ -344,16 +345,15 @@ class Builder: # Clone Buildroot. This only happens if the source directory # didn't exist already. - srcdir = os.path.join(self.idir, "buildroot") - if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", self.repo, srcdir], + if not os.path.exists(self.srcdir): + ret = subprocess.call(["git", "clone", self.repo, self.srcdir], stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not clone Buildroot sources") return -1 # Update the Buildroot sources. - abssrcdir = os.path.abspath(srcdir) + abssrcdir = os.path.abspath(self.srcdir) ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not fetch Buildroot sources") @@ -378,7 +378,6 @@ class Builder: def gen_config(self): """Generate a new random configuration.""" outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") log_write(self.log, "INFO: generate the configuration") @@ -387,13 +386,13 @@ class Builder: else: devnull = open(os.devnull, "w") - args = [os.path.join(srcdir, "utils/genrandconfig"), - "-o", outputdir, "-b", srcdir] + args = [os.path.join(self.srcdir, "utils/genrandconfig"), + "-o", outputdir, "-b", self.srcdir] toolchains_csv = self.toolchains_csv if toolchains_csv: if not os.path.isabs(toolchains_csv): - toolchains_csv = os.path.join(srcdir, toolchains_csv) + toolchains_csv = os.path.join(self.srcdir, toolchains_csv) args.extend(["--toolchains-csv", toolchains_csv]) ret = subprocess.call(args, stdout=devnull, stderr=self.log) @@ -425,7 +424,6 @@ class Builder: """ outputdir = os.path.join(self.idir, "output") - srcdir = os.path.join(self.idir, "buildroot") reproducible_results = os.path.join(outputdir, "results", "reproducible_results") # Using only tar images for now build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") @@ -434,7 +432,7 @@ class Builder: with open(reproducible_results, 'w') as diff: if self.sysinfo.has("diffoscope"): # Prefix to point diffoscope towards cross-tools - prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) + prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string prefix = prefix[13:-1] log_write(self.log, "INFO: running diffoscope on images") @@ -461,13 +459,12 @@ class Builder: # script. dldir = os.path.abspath(os.path.join(self.idir, "dl")) outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") f = open(os.path.join(outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, - "-C", srcdir, "BR2_DL_DIR=%s" % dldir, + "-C", self.srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) @@ -499,7 +496,7 @@ class Builder: log_write(self.log, "INFO: build failed [%d]" % ret) return -1 - cmd = ["make", "O=%s" % outputdir, "-C", srcdir, + cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) @@ -517,7 +514,6 @@ class Builder: """ outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") @@ -531,7 +527,7 @@ class Builder: # Clean up build 1 f = open(os.path.join(outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) + subprocess.call(["make", "O=%s" % outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") @@ -553,7 +549,6 @@ class Builder: """ outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") resultdir = os.path.join(outputdir, "results") shutil.copyfile(os.path.join(outputdir, ".config"), @@ -576,7 +571,7 @@ class Builder: subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) + shell=True, cwd=self.srcdir) # Return True if the result should be rejected, False otherwise def reject_results(): From patchwork Sun Jul 7 05:28:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128573 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OGXSMg3b"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGP1sbxz9sCJ for ; Sun, 7 Jul 2019 15:30:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 7FD9E20414; Sun, 7 Jul 2019 05:30:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qNyMnCXEy7dU; Sun, 7 Jul 2019 05:29:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 10D7020430; Sun, 7 Jul 2019 05:29:50 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id D7C841BF956 for ; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id CE0B220430 for ; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9+NQfWBoFXMN for ; Sun, 7 Jul 2019 05:29:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by silver.osuosl.org (Postfix) with ESMTPS id 99035204EA for ; Sun, 7 Jul 2019 05:29:37 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id k8so6541752plt.3 for ; Sat, 06 Jul 2019 22:29:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kG2d14hMokqPCcnbXXf0rNQtRalwe8OWSiiUmlDWwBE=; b=OGXSMg3biSIpCDbMBoZjIrTYFT6vOb6mYBlDOK4o4B5+/pJm8q+Zp+azhm1Qi4hEnC TWr+Fmh3OiJywDsVSnYgzr2VAWEFrRvqbHzq33oyTiDdQqqZTmugnRs5/gDAVexFKxyy FnEn9j2ARr3qJyzuqja+HeSQpgfplhLwk1NMTWEegDMdTBBLnRPZ3dQkI+hzbQGnIKIP tSwMC+eJ+07IDnocPs6Y/ZjTzR0hsAbK3UV7bL40HteS79bLz3joFRJvWodT28dcHBLE EqWzW9j44My8jBBbFoPt20QA3od0oMPdO5B00OP7CW4QY2oKqyUmjh5Af+n9K7Jt27EQ Xg9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kG2d14hMokqPCcnbXXf0rNQtRalwe8OWSiiUmlDWwBE=; b=jJqjqJVbRyTVzjxQ3gPfiXATWiqhrqQaCL6EYY9mEcttxiz181o/wTHSQNk5GxWmRh lVeUGlRLGa5DD72ZChTZznJVaTXRDxwDis42V9/KvY2NABsrXiPaEM7NCOBaUgPMRc1X HywI/D1wwv5Jly2XNZe3zbcbj2PLQANdW6+ejvGsLhrYsPOAUIyRT77MwUTYmHw9DvHK nOWfe1Ykig2/eBCoGdey3D+U5BI51CTJAGz+G3nu5c+Hj4Bcdw0lihqY7KJpCTl9794B 3HnZW97YyUD04uGqTyk4s7jpfZ1k54t3TsdOKnBAxBcqujNsxWUMqKLE4bKTkhRyx1D1 pHsw== X-Gm-Message-State: APjAAAU46qWudZUaRpRRiGsK6OSr8RD5tYHNKEJJFyhD7PikBCvoJyOV hEVZdA6h1Z5cB512200gAy6FX6fZzO6KMw== X-Google-Smtp-Source: APXvYqxTjotApUJB8EAKjTY8x7Z8tsHchrE9oauoSAoUpUlw5uRj+uxm83bHE9HwMPibk6m/VJi+yg== X-Received: by 2002:a17:902:8207:: with SMTP id x7mr15019559pln.63.1562477376886; Sat, 06 Jul 2019 22:29:36 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.34 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:36 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:18 +0530 Message-Id: <20190707052831.9469-18-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 18/31] autobuild-run: define download directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 741d316..84b4dee 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -293,6 +293,7 @@ class Builder: # frequently needed directories self.idir = "instance-%d" % self.instance self.srcdir = os.path.join(self.idir, "buildroot") + self.dldir = os.path.join(self.idir, "dl") if self.debug: self.log = sys.stdout @@ -310,9 +311,8 @@ class Builder: log_write(self.log, "INFO: preparing a new build") # Create the download directory if it doesn't exist - dldir = os.path.join(self.idir, "dl") - if not os.path.exists(dldir): - os.mkdir(dldir) + if not os.path.exists(self.dldir): + os.mkdir(self.dldir) # recursively find files under root def find_files(root): @@ -332,12 +332,12 @@ class Builder: # regularly re-download files to check that their upstream # location is still correct. for i in range(0, 5): - flist = list(find_files(dldir)) + flist = list(find_files(self.dldir)) if not flist: break f = flist[randint(0, len(flist) - 1)] log_write(self.log, "INFO: removing %s from downloads" % - os.path.relpath(f, dldir)) + os.path.relpath(f, self.dldir)) os.remove(f) branch = get_branch() @@ -457,14 +457,13 @@ class Builder: # path to the output directory here is not relative to the # Buildroot sources, but to the location of the autobuilder # script. - dldir = os.path.abspath(os.path.join(self.idir, "dl")) outputdir = os.path.abspath(os.path.join(self.idir, "output")) f = open(os.path.join(outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, - "-C", self.srcdir, "BR2_DL_DIR=%s" % dldir, + "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) @@ -497,7 +496,7 @@ class Builder: return -1 cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, - "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + "BR2_DL_DIR=%s" % self.dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) if ret != 0: From patchwork Sun Jul 7 05:28:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128570 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="QQVmbMBb"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGD3LBmz9sN4 for ; Sun, 7 Jul 2019 15:29:56 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id DB06C859BE; Sun, 7 Jul 2019 05:29:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vgj25EHWoQW9; Sun, 7 Jul 2019 05:29:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id BDB92854C0; Sun, 7 Jul 2019 05:29:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 506FD1BF5DD for ; Sun, 7 Jul 2019 05:29:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 472C985567 for ; Sun, 7 Jul 2019 05:29:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DKDdpZBTI5E6 for ; Sun, 7 Jul 2019 05:29:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id 0DBCF8610A for ; Sun, 7 Jul 2019 05:29:40 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id m9so2856731pls.8 for ; Sat, 06 Jul 2019 22:29:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=M/8yMvcSoGbZLtxyQ9WZwNGCADcA1gBYLfy5FsNVw7k=; b=QQVmbMBba+aKVeSyAO1uh1jrrL/SdWORhF0qfOUPHlK0XH2s6K7cw5CxGWU32z3IQZ WW/W3k8SkzmOFyV43E65nETqva8izLgdqHCXOtzPpan0S9gVqKnGiFbhiQC/BC6KNbLv is1yoX8JeccH+VyjwzaH9ZgDNyX9uC03xI8Ali7fV2rZJ1KlagvajPY7RD7geTPQhpTw M98tAqavhV2x95xJ6HMEl92WggUUxghPbgaGe/VRLRLG7EMHpei5SqpYg5Q1dYrZXtsi oxp7suHAbD+6FbgVfsiqqLVC3r/GIfPhcflfsmHXKL+Q7fCd6L5ciIEZbNYA4GHYsKwM lcdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M/8yMvcSoGbZLtxyQ9WZwNGCADcA1gBYLfy5FsNVw7k=; b=SFpt/yFc6skf7wlO6spSxyHS7e37NvJ2HrCJmSS3pwkyCy+v2HPuTE20YfbL2Ss2nV BGqJPQ/1zgsVATk2CgDDcjSuT5t5ET1wYjo8rd1GQL1RM/TO3ts9Dx9MYXUnPoe1Zgcu OoQ1KDnZQmuDVHBe0iqdvTgfBV9EmZBcV6th2zrmQeL0xwWhHWpRanp9GAP3n+eQubIs EeRGLpgdXOcEhmBrqtl5bDpVAvxEwyHCt8nHXHEIO9Hk3hQqS9kRiF75sLzgc2c5u2kw CBSfwXibfraJoRSE4OWToAO8xIQryMTiasnPt8VVM4OGnJ9P2H89HO2SOQWEjJ+175mT 3VYA== X-Gm-Message-State: APjAAAWUNy7AoHa7rTU2g+L1szbdu9jjjCHVJYIyi1hDdXN6zCob6MIt rpk7tfkosiAYgfepvwvb7B4UmnWl7tuW/w== X-Google-Smtp-Source: APXvYqylSj3GkCYIttH33GpXkznZv4j8A6Niij6lRqxsqqn9wIMb9r4lSrnmMh3kRb6XfJtVcWT9ew== X-Received: by 2002:a17:902:f81:: with SMTP id 1mr14825483plz.191.1562477379244; Sat, 06 Jul 2019 22:29:39 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.37 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:38 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:19 +0530 Message-Id: <20190707052831.9469-19-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 19/31] autobuild-run: define output directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Absolute path for output dir is required when using outputdir with system commands like make or tail. It was necessary because the relative path would be relative to the autobuilder script rather than the buildroot source directory. It is OK to use an absolute path everywhere because an absolute path is always correct. Signed-off-by: Atharva Lele --- Changes v2 -> v3: - Use outputdir as argument in stop_on_build_hang(), required for later changes (different output dir for reproducibility testing) Changes v1 -> v2: - Fixed inaccuracies in commit message - Fixed whitespace issue --- scripts/autobuild-run | 81 ++++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 44 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 84b4dee..e225211 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -294,6 +294,11 @@ class Builder: self.idir = "instance-%d" % self.instance self.srcdir = os.path.join(self.idir, "buildroot") self.dldir = os.path.join(self.idir, "dl") + # We need the absolute path to use with O=, because the relative + # path to the output directory here is not relative to the + # Buildroot sources, but to the location of the autobuilder + # script. + self.outputdir = os.path.abspath(os.path.join(self.idir, "output")) if self.debug: self.log = sys.stdout @@ -365,19 +370,17 @@ class Builder: return -1 # Create an empty output directory. We remove it first, in case a previous build was aborted. - outputdir = os.path.join(self.idir, "output") - if os.path.exists(outputdir): + if os.path.exists(self.outputdir): # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", outputdir]) - os.mkdir(outputdir) - with open(os.path.join(outputdir, "branch"), "w") as branchf: + subprocess.call(["rm", "-rf", self.outputdir]) + os.mkdir(self.outputdir) + with open(os.path.join(self.outputdir, "branch"), "w") as branchf: branchf.write(branch) return 0 def gen_config(self): """Generate a new random configuration.""" - outputdir = os.path.abspath(os.path.join(self.idir, "output")) log_write(self.log, "INFO: generate the configuration") @@ -387,7 +390,7 @@ class Builder: devnull = open(os.devnull, "w") args = [os.path.join(self.srcdir, "utils/genrandconfig"), - "-o", outputdir, "-b", self.srcdir] + "-o", self.outputdir, "-b", self.srcdir] toolchains_csv = self.toolchains_csv if toolchains_csv: @@ -423,16 +426,15 @@ class Builder: installed, fallback to cmp """ - outputdir = os.path.join(self.idir, "output") - reproducible_results = os.path.join(outputdir, "results", "reproducible_results") + reproducible_results = os.path.join(self.outputdir, "results", "reproducible_results") # Using only tar images for now - build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") - build_2_image = os.path.join(outputdir, "images", "rootfs.tar") + build_1_image = os.path.join(self.outputdir, "images-1", "rootfs.tar") + build_2_image = os.path.join(self.outputdir, "images", "rootfs.tar") with open(reproducible_results, 'w') as diff: if self.sysinfo.has("diffoscope"): # Prefix to point diffoscope towards cross-tools - prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) + prefix = subprocess.check_output(["make", "O=%s" % self.outputdir, "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string prefix = prefix[13:-1] log_write(self.log, "INFO: running diffoscope on images") @@ -453,16 +455,11 @@ class Builder: def do_build(self): """Run the build itself""" - # We need the absolute path to use with O=, because the relative - # path to the output directory here is not relative to the - # Buildroot sources, but to the location of the autobuilder - # script. - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - f = open(os.path.join(outputdir, "logfile"), "w+") + f = open(os.path.join(self.outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), - "make", "O=%s" % outputdir, + "make", "O=%s" % self.outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() @@ -474,7 +471,7 @@ class Builder: build_monitor = Thread(target=self.stop_on_build_hang, args=(monitor_thread_hung_build_flag, monitor_thread_stop_flag, - sub, outputdir)) + sub, self.outputdir)) build_monitor.daemon = True build_monitor.start() @@ -495,7 +492,7 @@ class Builder: log_write(self.log, "INFO: build failed [%d]" % ret) return -1 - cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, + cmd = ["make", "O=%s" % self.outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) @@ -512,8 +509,6 @@ class Builder: perform the actual build. """ - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") ret = self.do_build() @@ -522,11 +517,11 @@ class Builder: return ret # First build has been built, move files and start build 2 - os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1")) + os.rename(os.path.join(self.outputdir, "images"), os.path.join(self.outputdir, "images-1")) # Clean up build 1 - f = open(os.path.join(outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) + f = open(os.path.join(self.outputdir, "logfile"), "w+") + subprocess.call(["make", "O=%s" % self.outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") @@ -547,19 +542,18 @@ class Builder: are available) or stores them locally as tarballs. """ - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - resultdir = os.path.join(outputdir, "results") + resultdir = os.path.join(self.outputdir, "results") - shutil.copyfile(os.path.join(outputdir, ".config"), + shutil.copyfile(os.path.join(self.outputdir, ".config"), os.path.join(resultdir, "config")) - shutil.copyfile(os.path.join(outputdir, "defconfig"), + shutil.copyfile(os.path.join(self.outputdir, "defconfig"), os.path.join(resultdir, "defconfig")) - shutil.copyfile(os.path.join(outputdir, "branch"), + shutil.copyfile(os.path.join(self.outputdir, "branch"), os.path.join(resultdir, "branch")) def copy_if_exists(directory, src, dst=None): - if os.path.exists(os.path.join(outputdir, directory, src)): - shutil.copyfile(os.path.join(outputdir, directory, src), + if os.path.exists(os.path.join(self.outputdir, directory, src)): + shutil.copyfile(os.path.join(self.outputdir, directory, src), os.path.join(resultdir, src if dst is None else dst)) copy_if_exists("build", "build-time.log") @@ -575,7 +569,7 @@ class Builder: # Return True if the result should be rejected, False otherwise def reject_results(): lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], stdout=subprocess.PIPE).communicate()[0]).splitlines() # Reject results where qemu-user refused to build @@ -592,7 +586,7 @@ class Builder: def get_failure_reason(): # Output is a tuple (package, version), or None. lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], stdout=subprocess.PIPE).communicate()[0]).splitlines() regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') @@ -609,14 +603,14 @@ class Builder: def extract_last_500_lines(): subprocess.call(["tail -500 %s > %s" % \ - (os.path.join(outputdir, "logfile"), resultfile)], + (os.path.join(self.outputdir, "logfile"), resultfile)], shell=True) reason = get_failure_reason() if not reason: extract_last_500_lines() else: - f = open(os.path.join(outputdir, "logfile"), 'r') + f = open(os.path.join(self.outputdir, "logfile"), 'r') mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) mf.seek(0) # Search for first action on the failed package @@ -640,7 +634,7 @@ class Builder: if not reason: return - srcroot = os.path.join(outputdir, "build", '-'.join(reason)) + srcroot = os.path.join(self.outputdir, "build", '-'.join(reason)) destroot = os.path.join(resultdir, '-'.join(reason)) config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', 'CMakeOutput.log') @@ -671,7 +665,7 @@ class Builder: # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=self.log, stderr=self.log) + cwd=self.outputdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not make results tarball") sys.exit(1) @@ -683,7 +677,7 @@ class Builder: ret = subprocess.call(["curl", "-u", "%s:%s" % (self.http_login, self.http_password), "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadedfile=@%s" % os.path.join(self.outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", self.http_url], stdout=self.log, stderr=self.log) @@ -693,10 +687,10 @@ class Builder: log_write(self.log, "INFO: results were submitted successfully") else: # No http login/password, keep tarballs locally - with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: + with open(os.path.join(self.outputdir, "results.tar.bz2"), 'rb') as f: sha1 = hashlib.sha1(f.read()).hexdigest() resultfilename = "instance-%d-%s.tar.bz2" % (self.instance, sha1) - os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) + os.rename(os.path.join(self.outputdir, "results.tar.bz2"), resultfilename) log_write(self.log, "INFO: results saved as %s" % resultfilename) def run_instance(self): @@ -728,8 +722,7 @@ class Builder: continue # Check if the build test is supposed to be a reproducible test - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - with open(os.path.join(outputdir, ".config"), "r") as fconf: + with open(os.path.join(self.outputdir, ".config"), "r") as fconf: reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() if reproducible: ret = self.do_reproducible_build() From patchwork Sun Jul 7 05:28:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128565 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="CvsjmXc7"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHG23V16z9s8m for ; Sun, 7 Jul 2019 15:29:46 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id EFB21854FC; Sun, 7 Jul 2019 05:29:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ccgifAOO3gVa; Sun, 7 Jul 2019 05:29:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5FD6D84B08; Sun, 7 Jul 2019 05:29:44 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id DE6131BF5DD for ; Sun, 7 Jul 2019 05:29:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id DB4FD854CC for ; Sun, 7 Jul 2019 05:29:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id u78FQMZiQWUx for ; Sun, 7 Jul 2019 05:29:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f194.google.com (mail-pl1-f194.google.com [209.85.214.194]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 59FB084506 for ; Sun, 7 Jul 2019 05:29:42 +0000 (UTC) Received: by mail-pl1-f194.google.com with SMTP id b3so3220873plr.4 for ; Sat, 06 Jul 2019 22:29:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h3MHZ4Z7iYwb1SSYjW/0XYW9hLEZvlPXuLL3ZF+QVs4=; b=CvsjmXc7gG9SWz2RxNzs64iyj8EXhJwvMl2A2ADjyrwVyDX2kGPi1Q9ZBaUQmh21Ce 9eAw1igsOmMRZvuA9Ye3zFb9nYvlt0VAFrMmybyhD/qWnYD1k7G7HHZHSa59KhFn7fYF sE1Gduxscekmyc+3hdgvSE67s27CohHUwXuFnYt02/L+OiwUZXj/ZdGLkG57FecnBG4K yv3ilFaFpDj4o7O0iUZ4UFfZa0RtpE30FJ2EXDTvFe2m/iWFsfFdbs2bTr7GyVKSKe4c PBPpE4jhGauuLCwFaNl5dW4/JR+xIXMCmp71g7mYj+O2tNR5AVPm1f8z2b2Q4ARPW1xx mhdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h3MHZ4Z7iYwb1SSYjW/0XYW9hLEZvlPXuLL3ZF+QVs4=; b=lQLeXDZ9rRpUP0XWGpPNFB2MZmv7A8/YPAzTOl1bd5ecJl0itbNa1wpPZHWeN3UPOe mRrhtRr0Lb45pO3bh9w+ET7mARrHfla23zS4E2WeauuwVa8rxKE7ao9K7PXGFZWwCHhq hZTGZRmKbk/UnuJ+Ph2UrAu0qKtB+R7QExCB8RFDG22iDFOK4hVBFCcnAuJZ3JhhPYCT GvJdqUJsDE7a8rNgPcpBYAmzORXj853sQ67JPalpiEwxGV+KenspbVYgBRHoi2uhoItq EzMf5pTpgGwIbTwnJisgIMLv80Oy8BBBgnIbYIW9t6P3DU4MfJvNUn7Jb4+0Bl8ZbcW1 s1CA== X-Gm-Message-State: APjAAAVIWjWmbzPOAqXXiEAL0brtdo/GFco5IBMREkjnq4Pvjg8GrzLF EvDTKV7DCOkYNIr2vda1dREvwUrLsshkXQ== X-Google-Smtp-Source: APXvYqyMwWxKLiJNOcpnQrn+NZbUkjAC3e66zOKmmt243dqkQwHIijO8OR5ubYt4aHuWt2wSl04rXQ== X-Received: by 2002:a17:902:8c98:: with SMTP id t24mr15429564plo.320.1562477381577; Sat, 06 Jul 2019 22:29:41 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.39 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:41 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:20 +0530 Message-Id: <20190707052831.9469-20-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 20/31] autobuild-run: define results directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index e225211..8a0bec7 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -299,6 +299,7 @@ class Builder: # Buildroot sources, but to the location of the autobuilder # script. self.outputdir = os.path.abspath(os.path.join(self.idir, "output")) + self.resultdir = os.path.join(self.outputdir, "results") if self.debug: self.log = sys.stdout @@ -542,19 +543,17 @@ class Builder: are available) or stores them locally as tarballs. """ - resultdir = os.path.join(self.outputdir, "results") - shutil.copyfile(os.path.join(self.outputdir, ".config"), - os.path.join(resultdir, "config")) + os.path.join(self.resultdir, "config")) shutil.copyfile(os.path.join(self.outputdir, "defconfig"), - os.path.join(resultdir, "defconfig")) + os.path.join(self.resultdir, "defconfig")) shutil.copyfile(os.path.join(self.outputdir, "branch"), - os.path.join(resultdir, "branch")) + os.path.join(self.resultdir, "branch")) def copy_if_exists(directory, src, dst=None): if os.path.exists(os.path.join(self.outputdir, directory, src)): shutil.copyfile(os.path.join(self.outputdir, directory, src), - os.path.join(resultdir, src if dst is None else dst)) + os.path.join(self.resultdir, src if dst is None else dst)) copy_if_exists("build", "build-time.log") copy_if_exists("build", "packages-file-list.txt") @@ -563,7 +562,7 @@ class Builder: copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv") subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ - os.path.join(resultdir, "gitid")], + os.path.join(self.resultdir, "gitid")], shell=True, cwd=self.srcdir) # Return True if the result should be rejected, False otherwise @@ -625,7 +624,7 @@ class Builder: mf.close() f.close() - extract_end_log(os.path.join(resultdir, "build-end.log")) + extract_end_log(os.path.join(self.resultdir, "build-end.log")) def copy_config_log_files(): """Recursively copy any config.log files from the failing package""" @@ -635,7 +634,7 @@ class Builder: return srcroot = os.path.join(self.outputdir, "build", '-'.join(reason)) - destroot = os.path.join(resultdir, '-'.join(reason)) + destroot = os.path.join(self.resultdir, '-'.join(reason)) config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', 'CMakeOutput.log') @@ -650,7 +649,7 @@ class Builder: copy_config_log_files() - resultf = open(os.path.join(resultdir, "status"), "w+") + resultf = open(os.path.join(self.resultdir, "status"), "w+") if result == 0: resultf.write("OK") elif result == -1: @@ -659,7 +658,7 @@ class Builder: resultf.write("TIMEOUT") resultf.close() - with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: + with open(os.path.join(self.resultdir, "submitter"), "w+") as submitterf: submitterf.write(self.submitter) # Yes, shutil.make_archive() would be nice, but it doesn't exist @@ -713,8 +712,7 @@ class Builder: if ret != 0: continue - resultdir = os.path.join(self.idir, "output", "results") - os.mkdir(resultdir) + os.mkdir(self.resultdir) ret = self.gen_config() if ret != 0: From patchwork Sun Jul 7 05:28:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128568 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Ldvj62at"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGC6jtWz9sCJ for ; Sun, 7 Jul 2019 15:29:55 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 59BA220480; Sun, 7 Jul 2019 05:29:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UxLE0DNFM1Pr; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 42D62203C3; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 6FAA91BF5DD for ; Sun, 7 Jul 2019 05:29:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 6BFAC87082 for ; Sun, 7 Jul 2019 05:29:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3lprWkrMiDhk for ; Sun, 7 Jul 2019 05:29:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by hemlock.osuosl.org (Postfix) with ESMTPS id B5BCC8702D for ; Sun, 7 Jul 2019 05:29:44 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id m4so6039833pgk.0 for ; Sat, 06 Jul 2019 22:29:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=o++Q7ln/GUoIHHganxrehuyBNxr07J6LxIzsYLfcSCA=; b=Ldvj62at1c97si/m9Hr5nIebN1aDV4adBbgqLb3yeUsTtZZ6R5b+lWdFi3TTZpBnB0 qv1PP9ibp8g+hr7bdyqZZeZ6f+Ex2YuvDYpmfQ3p/hcP7w7Bgmki9yzHfdZRQmncvOwT diTFM16LywLBCPL3Pxb/yIwYiwBFOGZMej5ypkKysFb+8rqmoO0Ojg+LL75KlRx9/o6/ Kk+mbb20JImSIJPmMApfnevdxkldXcX9DP3sSLCqk1EzFUDjQhk9d9gDPdhVLklf5lCN Ml6w9urBrWOZ/VtaWcLwaTNUQNtjDwnSrgBW7aojUs9pvKWgmAPcaLBayXVycDL0FjFT 9KVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o++Q7ln/GUoIHHganxrehuyBNxr07J6LxIzsYLfcSCA=; b=ggSQCOCr0EVlEu+drIGkovK0GVODYaJNPRL2eLGYR1iqVMk1/bvazY5FCuVUn0Kfkw 7eIK0K+No8f7K3K89HkIP/IGcO+NCRn2iy3XTyK1mr61kzBoa8hfECPa1TAbwbpk3V4P UJ9TT02jkgFiWm7hl6az2+B90MCO88dAMIM2+ZZw7Ip1j6Se8o5k1BOlqqjqLwvCvs5Q enq/2MtokshEpxyqfnd5FDuAnYkswmHlU2KoxT94PkwsEyV2Ho6rt57z11/H1T1tnzuE 236AX1pThlJm3d/J9XCa1qtwhejHbYyeiPKyj/WrhMWhoBVQHb+USEk91H85bZhwtPTQ wQfA== X-Gm-Message-State: APjAAAWpd2jQjkpQP97nBzIspw5sayP+iHLGKJTI05RDXe+cGXIeoyee LRhtJTJLg7daZ+3ggS0DmXy+6esdmEBiLw== X-Google-Smtp-Source: APXvYqy7UcCbjf5ROHKrgKm3LtEfGHjvO9JhCZelumr+Mos8TeNDh+9a8ZxXITd3sD+ZVZ6SqGHbQQ== X-Received: by 2002:a17:90a:346c:: with SMTP id o99mr15127622pjb.20.1562477383970; Sat, 06 Jul 2019 22:29:43 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.41 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:43 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:21 +0530 Message-Id: <20190707052831.9469-21-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 21/31] autobuild-run: move check_version() to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Upcoming patch from Thomas needs check_version() to use http_url. As that is inside Builder class and check_version() is only used inside Builder class, move it inside the class. Also remove the redundant call to check_version() from main(). We already call check_version() in run_instance(). Also the fact that we cannot call check_version() in main() since the builder class object is not defined at that point. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - New patch --- scripts/autobuild-run | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 8a0bec7..391c1dc 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -178,13 +178,6 @@ def log_write(logf, msg): logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg)) logf.flush() -def check_version(): - with urlopen_closing('http://autobuild.buildroot.org/version') as r: - version = int(decode_bytes(r.readline()).strip()) - if version > VERSION: - print("ERROR: script version too old, please upgrade.") - sys.exit(1) - def get_branch(): """Returns the branch that should be built. It fetches a CSV file from autobuild.buildroot.org that provides the list of branches to test @@ -306,6 +299,13 @@ class Builder: else: self.log = open(os.path.join(self.idir, "instance.log"), "a+") + def check_version(self): + with urlopen_closing('http://autobuild.buildroot.org/version') as r: + version = int(decode_bytes(r.readline()).strip()) + if version > VERSION: + print("ERROR: script version too old, please upgrade.") + sys.exit(1) + def prepare_build(self): """Prepare for the next build of the specified instance @@ -706,7 +706,7 @@ class Builder: log_write(self.log, "INFO: instance started") while True: - check_version() + self.check_version() ret = self.prepare_build() if ret != 0: @@ -765,7 +765,6 @@ def main(): # showing error messages in another language. os.environ['LC_ALL'] = 'C' - check_version() sysinfo = SystemInfo() args = docopt.docopt(doc, version=VERSION) From patchwork Sun Jul 7 05:28:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128567 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="uEiSpjKa"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHG71qd3z9s8m for ; Sun, 7 Jul 2019 15:29:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id B7E7987108; Sun, 7 Jul 2019 05:29:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O45NFBRq9U58; Sun, 7 Jul 2019 05:29:49 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id ED15E87082; Sun, 7 Jul 2019 05:29:48 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id CED5F1BF5DD for ; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C857184AF4 for ; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8KM7QPfGSDYS for ; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 4165984506 for ; Sun, 7 Jul 2019 05:29:47 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id j2so6006862pfe.6 for ; Sat, 06 Jul 2019 22:29:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jv7WWoFIpG1b6byU6+O93J/WQpVnC24lvDCFz3bL/og=; b=uEiSpjKaw0YF/F+XntmyhI+MydZ7EaXGFxzCnk9cLhDCn0nRFOwT13+hz07e0XNttl Oo4PnoiA1eafguOry79eB7wF6AKdmhddmp92snFhExOXajpGL3/K+OEGJVJOKQVb6qNx Msv+8FQJPFfREgwB5MUFdBraO88A/X0AcrZX6otD7mtH4t33NB4yJ4Msum/aSX4m78D1 Nmr+bIcjk1WzEzFNnWs8CgfbMkxXOSxqTKKiW8QAWKgK4VMflid8XU0KA7u7e+jlOTVw sFB8rJ254PmffWYpU+FwopHf4tL341mGDaOfJCOa/clivlkbjUhtmywroJFxiztYdXxA cEIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jv7WWoFIpG1b6byU6+O93J/WQpVnC24lvDCFz3bL/og=; b=r74T2g8iwfxup6ORoXHYVbsLrwQN5Vj27TUPTsIKMkcx7WOGddzQ/3gjB/DGVYCchV uKkm1So1pWhFlwbRufPM7I+be2po8sXlP0eOYy1/zW9HaTW74hYycxwYKwAOWhpYky0B dsRGlKbeRwvvh9ucgJm4nXkQomTciO0yTxFuz4ZTqK5QpmeUozyCEHVJmSlaM69HmJB/ v6iEhT+U03QJ4S4TCwbiQ4iUV3WSTUnDFPLai+J2efjkFXLJ7tkUDnF9KcxZG4/jFHYk ZIkx7LEyTlnuUKzrtxfAenkX0OwNjkt1wb79QlvxsyZZlESConzq6DQShoeN8OIgj718 mwNg== X-Gm-Message-State: APjAAAXeVNI0gbHViUmDlWBV+EWyv8NtSRp+uP6nWUo14Ibu+SOiWvAI HNX95zWUl4qX4ihzh1kJqP9KLjycpjvcIA== X-Google-Smtp-Source: APXvYqyPcrBRoSJIYLpsF+RmxfGaHB7REIsRJ+oc54bNrBzyXSq2g77qDT2Cs+A1jsLoIiCKD7/T2Q== X-Received: by 2002:a63:b555:: with SMTP id u21mr14691594pgo.222.1562477386378; Sat, 06 Jul 2019 22:29:46 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.44 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:45 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:22 +0530 Message-Id: <20190707052831.9469-22-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 22/31] autobuild-run: move get_branch() to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" get_branch() requires http_url and is only called inside Builder class. Moving it to builder class makes http_url available to it and makes the Builder class cohesive. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - New patch --- scripts/autobuild-run | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 391c1dc..ccfa9c4 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -178,23 +178,6 @@ def log_write(logf, msg): logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg)) logf.flush() -def get_branch(): - """Returns the branch that should be built. It fetches a CSV file from - autobuild.buildroot.org that provides the list of branches to test - (first field) and their weight (second field). We build a list of - branches, where the branch name is repeated N times, with N being - the weight of the branch. We then pick a random branch in this - list. This way, branches with a higher weight are more likely to - be selected. - """ - with urlopen_closing('http://autobuild.buildroot.org/branches') as r: - csv_branches = r.readlines() - branches = [] - for branch in csv.reader(csv_branches): - branches += [branch[0]] * int(branch[1]) - - return branches[randint(0, len(branches) - 1)] - class SystemInfo: DEFAULT_NEEDED_PROGS = ["make", "git", "gcc"] DEFAULT_OPTIONAL_PROGS = ["bzr", "diffoscope", "java", "javac", "jar"] @@ -306,6 +289,23 @@ class Builder: print("ERROR: script version too old, please upgrade.") sys.exit(1) + def get_branch(self): + """Returns the branch that should be built. It fetches a CSV file from + autobuild.buildroot.org that provides the list of branches to test + (first field) and their weight (second field). We build a list of + branches, where the branch name is repeated N times, with N being + the weight of the branch. We then pick a random branch in this + list. This way, branches with a higher weight are more likely to + be selected. + """ + with urlopen_closing('http://autobuild.buildroot.org/branches') as r: + csv_branches = r.readlines() + branches = [] + for branch in csv.reader(csv_branches): + branches += [branch[0]] * int(branch[1]) + + return branches[randint(0, len(branches) - 1)] + def prepare_build(self): """Prepare for the next build of the specified instance @@ -346,7 +346,7 @@ class Builder: os.path.relpath(f, self.dldir)) os.remove(f) - branch = get_branch() + branch = self.get_branch() log_write(self.log, "INFO: testing branch '%s'" % branch) # Clone Buildroot. This only happens if the source directory From patchwork Sun Jul 7 05:28:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128571 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="lYCTL7Xj"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGJ6ysjz9s8m for ; Sun, 7 Jul 2019 15:30:00 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 7E9EF85725; Sun, 7 Jul 2019 05:29:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SQhdRQpGoCcW; Sun, 7 Jul 2019 05:29:55 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 97F088575A; Sun, 7 Jul 2019 05:29:52 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 7A3201BF5DD for ; Sun, 7 Jul 2019 05:29:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7720284AF4 for ; Sun, 7 Jul 2019 05:29:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KsTV2JHYo4WH for ; Sun, 7 Jul 2019 05:29:49 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id A40D484506 for ; Sun, 7 Jul 2019 05:29:49 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id t14so3438497plr.11 for ; Sat, 06 Jul 2019 22:29:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Qa9i9ZHSAtB4trGEnxaO3r8utSZeromPfcjGLjBeKJk=; b=lYCTL7XjMA+e3e2mwS3aDvw+MC/L0TeeOTX/lIPiCv2CMq42jC/5Uxlay5eBTM/gNY RzLuGe/1gdTXJ+c4Xx/vT4VWEFgWvkKobtr9x24/yiXkVCFr/t0NwbgWihzCCWjyGk6o po6ALW8RvXkccAxhj0N40gajvzbg6EPpJdzakgy/Sq71bDBSw2C9hS+KUewwc6Oy9XL2 6LWoggYVmlOiAvGFcmdwrFeQDnE7wn4yuWeIf+X2UCf68YOJJCcqmS7A+fGWyy4A6s/R IXaZOd1nt+RUZqp4gsNgPkscnx5gLKLKtozltaiZYFt5d4Kg/jMT1R2ADcLKWd5vku2X jtaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Qa9i9ZHSAtB4trGEnxaO3r8utSZeromPfcjGLjBeKJk=; b=k3BFlKQ87gP6LVk2ej/zIEkNncCpo+cV9lt7iPi0Yit5l78PvLMckN/EZf/zPH/xuH rAxuC7+onqyXiZ8TUG2D6yTEO77i/gwZ3l2oMvwDDA/MBfhPQq4zenw+RHKf4aDgMPOb vOG/x5GVeTJp9gDE4OExC1nZqG0GW0BN3kC8NtGjlSy4Hok7hqsxdntkcVSjb3gdpOsr jGUgt0+402VCy/OhyxxIfVP1n3c2HL8B0h0ACU5EwmfqTRrYLWlXtD9ysb1h1jsYoRfL c0RLhGg/TjA00mFTeO13X2E58wx2nGOUqFIOQzxLGe1MEpYNtptGG/8k4kUAU8PhahdI 1qXA== X-Gm-Message-State: APjAAAWAG/2slkJbdQQNKHOw6NAsOr71U80lrAHy3mB4zzzDt9J7+Oj2 eMR7bHMkNfBj6/8DK9iyjVcIid158I0cOw== X-Google-Smtp-Source: APXvYqzg0v6NEzqnyfv7KwmP7Jne/3CoFZjN+wWQS6F9dBXZb2Fc8bu0gBYKa0YtxN0szsHnLu6pvw== X-Received: by 2002:a17:902:e282:: with SMTP id cf2mr15220418plb.301.1562477388895; Sat, 06 Jul 2019 22:29:48 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.46 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:48 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:23 +0530 Message-Id: <20190707052831.9469-23-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 23/31] scripts/autobuild-run: make the HTTP URL really configurable X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas Petazzoni The --http-url option allowed to customize the URL at which build results are submitted. However, there were two other places in the script where autobuild.buildroot.org was hardcoded: when checking the script version, and when getting the list of branches to build. This commit changes the --http-url to be the base URL of the autobuild server, and it is used everywhere instead of hardcoding autobuild.buildroot.org. Note: users of autobuild-run that were passing a custom --http-url option such as http://foo.com/submit/ should change it to just http://foo.com/ Signed-off-by: Thomas Petazzoni [Atharva: Adapted Thomas' patches to work with Builder class] Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Incorporate Thomas' patch into Builder class series - Move patch for appropriate order of patches --- scripts/autobuild-run | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index ccfa9c4..678bac5 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -66,7 +66,7 @@ defaults = { '--make-opts': '', '--nice': 0, '--pid-file': '/tmp/buildroot-autobuild.pid', - '--http-url': 'http://autobuild.buildroot.org/submit/', + '--http-url': 'http://autobuild.buildroot.org/', '--toolchains-csv': 'support/config-fragments/autobuild/toolchain-configs.csv', '--repo': 'https://github.com/buildroot/buildroot.git', } @@ -145,6 +145,7 @@ from distutils.version import StrictVersion import platform from threading import Thread, Event import datetime +import urlparse if sys.hexversion >= 0x3000000: import configparser @@ -283,7 +284,7 @@ class Builder: self.log = open(os.path.join(self.idir, "instance.log"), "a+") def check_version(self): - with urlopen_closing('http://autobuild.buildroot.org/version') as r: + with urlopen_closing(urlparse.urljoin(self.http_url, 'version')) as r: version = int(decode_bytes(r.readline()).strip()) if version > VERSION: print("ERROR: script version too old, please upgrade.") @@ -291,14 +292,14 @@ class Builder: def get_branch(self): """Returns the branch that should be built. It fetches a CSV file from - autobuild.buildroot.org that provides the list of branches to test + the autobuild server that provides the list of branches to test (first field) and their weight (second field). We build a list of branches, where the branch name is repeated N times, with N being the weight of the branch. We then pick a random branch in this list. This way, branches with a higher weight are more likely to be selected. """ - with urlopen_closing('http://autobuild.buildroot.org/branches') as r: + with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r: csv_branches = r.readlines() branches = [] for branch in csv.reader(csv_branches): @@ -678,7 +679,7 @@ class Builder: "-H", "Expect:", "-F", "uploadedfile=@%s" % os.path.join(self.outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", - self.http_url], + urlparse.urljoin(self.http_url, 'submit/')], stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "INFO: results could not be submitted, %d" % ret) From patchwork Sun Jul 7 05:28:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128577 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="GEPo/iYj"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGf57JSz9s8m for ; Sun, 7 Jul 2019 15:30:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 3E2FF84B35; Sun, 7 Jul 2019 05:30:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Qj0Yk+NZK2ZT; Sun, 7 Jul 2019 05:30:13 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 657EB84848; Sun, 7 Jul 2019 05:30:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 9E9801BF5DD for ; Sun, 7 Jul 2019 05:30:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 9BA698702D for ; Sun, 7 Jul 2019 05:30:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JHQ7z2OMVjsL for ; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by hemlock.osuosl.org (Postfix) with ESMTPS id 0D9328733C for ; Sun, 7 Jul 2019 05:29:52 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id q10so5999086pff.9 for ; Sat, 06 Jul 2019 22:29:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lNHBTMFIGwb9H7hNx4sCqafyG+kux1lxRqUDS1oIKtQ=; b=GEPo/iYjXlDpcYNZD4Iw+8sff9BwGRS4+p6J0c/xD1UCeM2CQsmEOWXfG7XRe1MAQN lbHPMb4ciADCsPdqGaU1KYUQPuwOF6LFkkg7PgSAcGyXdDcDPFc00m20LUkYz0tCb1Ps 40tkvk4jzoqD0W1BPpdci6pv8EOHHxQgYV7HXs1IKdhJJCog3guhai9+lVyhMuS0lKxV lZWJQIMMon+d9hJ9J3kgZTR/YtCMjnlP/gW9/B4yCwhRxz/oFvWHf/SEmnncEFfXGQAt p/xlZYeUPzyEa0GEZeNWCSvDzKXt2nQUX8I2UGzDqEi4wCJsHosbUE2y7ec6q4VSsmnQ Leeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lNHBTMFIGwb9H7hNx4sCqafyG+kux1lxRqUDS1oIKtQ=; b=S/Ts68dqUPP0/sxp1bkX/LOqNUkQGmOiDhW0vrAQ4/CUBgksfcAno48EZDxfW845jD 4Fmrca11DzVW24U4d5gRNojGHOCeIigMrgUrQcKKIw6zXFgBHuMSgz+0Eq2FViXd9SLw Kvl13fiSQb3bqFRbJlaZcVSr6zg/R+tP8KdYFsRZQBbkgdDczXRliy5mmzURnunB3/Yf audrTtPEfke+9YjgqqAPTI3aEQq0h4U38yNPR8LAG6b+BrhFPzdhcH3JvqZa51b+t1W5 dFS/nZcgNaidn1r/BjfJAGBr4nx/0emRU+FX+3OyE0PxHXyGRINgtp2gKMk1KBRIU5uS Kqdg== X-Gm-Message-State: APjAAAWJ/RexKBPAt5p8tAu+igJANzWCnRpj1obxmIsr0n8r5MM9eYMR sTQPngrUGeIcx4vqSirt0ZWickTM4sVR0g== X-Google-Smtp-Source: APXvYqzy7D03+jYkUdA0QOOApYO/g3l88teDxv/s+fpiJo01OuHhHnAv5xgNgtgnrGeAiafItfogSg== X-Received: by 2002:a63:494d:: with SMTP id y13mr14680071pgk.109.1562477391230; Sat, 06 Jul 2019 22:29:51 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.49 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:50 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:24 +0530 Message-Id: <20190707052831.9469-24-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 24/31] autobuild-run: create reason file on build failures X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" When a build fails, we calculate the reason of failure on the server side as well as client side. To solve this redundancy we only calculate on the client side and submit a file which contains the reason of failure. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes from reason-file-v2 series: - Move to builder-class series --- scripts/autobuild-run | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 678bac5..8f60e89 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -598,6 +598,11 @@ class Builder: # not found return None + reason = get_failure_reason() + if reason: + with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf: + reasonf.write("-".join(reason)) + def extract_end_log(resultfile): """Save the last part of the build log, starting from the failed package""" @@ -606,7 +611,6 @@ class Builder: (os.path.join(self.outputdir, "logfile"), resultfile)], shell=True) - reason = get_failure_reason() if not reason: extract_last_500_lines() else: @@ -630,7 +634,6 @@ class Builder: def copy_config_log_files(): """Recursively copy any config.log files from the failing package""" - reason = get_failure_reason() if not reason: return From patchwork Sun Jul 7 05:28:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128576 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="RnvbbuqK"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGc291cz9s8m for ; Sun, 7 Jul 2019 15:30:16 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 6F62D204DB; Sun, 7 Jul 2019 05:30:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dKs69FFRxaWC; Sun, 7 Jul 2019 05:30:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id EAECB204E6; Sun, 7 Jul 2019 05:30:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id DABD11BF5DD for ; Sun, 7 Jul 2019 05:30:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id D4C1F87134 for ; Sun, 7 Jul 2019 05:30:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e0u-QZ5L+pZz for ; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 8A708870DF for ; Sun, 7 Jul 2019 05:29:54 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id u14so4818230pfn.2 for ; Sat, 06 Jul 2019 22:29:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R+yJFqU0o4zbl5h0iz0rZDCWyfvdbmc17da+SKsfKFU=; b=RnvbbuqKep77oTYF0oYZBKqPhnFnUukEjn7XeclFJ72Kq1Yuk0Vx+Y8r72u3uMEydY AxbpMwrsayzoO/oBcHqYxqYF3f07CPHqQxnq+1GBnLhVrwppLRoyCPngk8nzgKqziBXB KPMqen3f39rLcM+comQK829sX+sxHGYpjGYzqaRtKQUnnQyGL2s4SSDUDdzPjym52C8m yrf5tTiEpmJ6Ir3TCdMQwyq3YU2usnESBO8YTMhiIvuy/lh+x39SjTEpdO3FH9NNHmDA YJm4HvqbhOG/W7BbkQAzayfbPYyJbLqaFyXp5SK9MJ6nDkpPZeQuJqm5IRG0dv+7Bq6h UZ7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R+yJFqU0o4zbl5h0iz0rZDCWyfvdbmc17da+SKsfKFU=; b=tpm6mw6Zok0AU0TIO98HYJRQZ2WOrz/to905CRONE7x+b9pTTqBuIRJlu7qyGPMPBs tVmKKFblaWv7UGhqNHWRaNQKlOerXCZMHIAwIIwmqVnZN3c8qhfwVlOHcMcjdnzMZD/T L24bZV/6UcOTvcC/d/cBmk30ceRgxeVQ2cWwVPiocGW/wtIPMINQg3vayemFDYXkZgDh S0OK5YdyjNXbjC1062mhf35s0lgAu3Iah0h7GS9sV4LcjptqyE78ZjetCcvGoX3LapyM v3B1SEHYN8t2kxHoeE+ux0BAYzempJtV/O2B5SoWJnAm63bDj3b6YoPghO5PrdFph4zX GU3Q== X-Gm-Message-State: APjAAAUj9EQ7R6mr5V0BpRO1zww89/pqOrf3gzqDx0ORQaKaYCwFbtty dxjOEUlCEkOC7B8WWhm5Ie2bJVuKVnwjDQ== X-Google-Smtp-Source: APXvYqxb5puZlKZxea39/yzKU+EoEHhZ9FdltY+7uB4dvREnPqy9XW7VyCCI0AW40eSesgY4Fz6FSQ== X-Received: by 2002:a17:90a:37e9:: with SMTP id v96mr15069202pjb.10.1562477393749; Sat, 06 Jul 2019 22:29:53 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.51 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:53 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:25 +0530 Message-Id: <20190707052831.9469-25-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 25/31] autobuild-run: account for reproducibility failures when creating the reason file X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- Changes from reason-file-v2 series: - Move to builder-class series --- scripts/autobuild-run | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 8f60e89..8011ebb 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -584,16 +584,26 @@ class Builder: return def get_failure_reason(): - # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() + # Output is a tuple (package, version), or None in case of package failure + # Output is "reproducible" in case of reproducibility failure + + reproducible_results = os.path.join(self.resultdir, "reproducible_results") + if os.path.exists(reproducible_results): + if os.stat(reproducible_results).st_size > 0: + reason = "reproducible" + return reason + else: + return None + else: + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() - regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') - for line in lastlines: - m = regexp.search(line) - if m: - return m.group(1).rsplit('-', 1) + regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') + for line in lastlines: + m = regexp.search(line) + if m: + return m.group(1).rsplit('-', 1) # not found return None @@ -601,7 +611,10 @@ class Builder: reason = get_failure_reason() if reason: with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf: - reasonf.write("-".join(reason)) + if reason == "reproducible": + reasonf.write(reason) + else: + reasonf.write("-".join(reason)) def extract_end_log(resultfile): """Save the last part of the build log, starting from the failed package""" From patchwork Sun Jul 7 05:28:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128575 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="X/f1LWP2"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGW3Wmwz9sCJ for ; Sun, 7 Jul 2019 15:30:11 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 23AFC8653F; Sun, 7 Jul 2019 05:30:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rQrecSkh5xE5; Sun, 7 Jul 2019 05:30:08 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 1252E83DD9; Sun, 7 Jul 2019 05:30:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 648B81BF5DD for ; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 61BB820414 for ; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HDqRzp0sJbAx for ; Sun, 7 Jul 2019 05:29:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by silver.osuosl.org (Postfix) with ESMTPS id D9766204E7 for ; Sun, 7 Jul 2019 05:29:56 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id m9so2856894pls.8 for ; Sat, 06 Jul 2019 22:29:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Zk2FiuLaLs/di/XgoD6jTxFuScCyTPRxSsKeZt9wdRg=; b=X/f1LWP28y5lmgQ02Y8DDZpjTdIFrGYuqZ2N+Cx1WRPEF6aJ+SoC12F2FRspSfooD5 URu5D3dfdUICgHVlLBO08qkUxDM8RbRl2hElFIabsWo1wyNNXNYfdLU7m8Eg/WQuaQUo btNasDk8s7PIFd6kKPe8D5msxVlMs0Ph3H/x6FoWBb2xnZ9sOmYTZcg5FfbjXYBHUniL ltBNp6Z52hZvPOkt1kt9KLS3DXtNbtBZtrTz2YRyrcF3CZfiHK0fyoZz7eLo9CS32fh5 85mFgLNjM8PApbAzYYSR7tl3pzXFVZOkSsaJrQ0ClarWS4dNcdYS/GWjc8YVN7GuyiJP hLig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Zk2FiuLaLs/di/XgoD6jTxFuScCyTPRxSsKeZt9wdRg=; b=ZTdiYSZdIxbNZP4WAumy4DwCFBrGc5G2TtQnk6TGdRIPE8/YD+35F+4l5Nf1wDV0EH +wzTWNXVPFzYx9vA2KXQPKrXMV74mYq2DatRxL2xSLtTiwBpWx6AL8grCuXw2I54HZ5D pIuVF34aPvjtLshxY3gEW8VpzhbVCIbmu0ghiH+E48g2qnItr/GQ86uQuj8hefE2Kcap C//LhH2VSKGRvPl8N37xsyPFPdW2+6cCmEf2A+Jbp+xp2wIXOyl0QV8FUaK9X8G7q7N/ S4Qa2ckpjJgY80+FB5cvu50f3v8/PVT3kahJ6v+nchAWlU7cz10TK2/CNCBXTrFR1qmR 5CzA== X-Gm-Message-State: APjAAAWA3rizbLhoVqIskjeVtAqaf6I3vAOkpaJlWwdwGzoDSBHd9ZgV s4tuF+akVG+CU4s1kgrwjycxUQi0t/fMLw== X-Google-Smtp-Source: APXvYqznPPMMSY/xbQG07/5SiKK2czgJZWeQC4E8TK7o7tFynqA7cq4ZimXXia+z5VPgZh09l3KKbQ== X-Received: by 2002:a17:902:2aab:: with SMTP id j40mr14367148plb.76.1562477396162; Sat, 06 Jul 2019 22:29:56 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.53 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:55 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:26 +0530 Message-Id: <20190707052831.9469-26-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 26/31] web/import.inc.php: support reading failure reason from reason file X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- NOTE: This patch is untested. Changes from reason-file-v2 series: - Move to builder-class series --- web/import.inc.php | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/web/import.inc.php b/web/import.inc.php index a9b368f..6f878e0 100644 --- a/web/import.inc.php +++ b/web/import.inc.php @@ -230,17 +230,21 @@ function import_result($buildid, $filename) if ($status == 0) $reason = "none"; else { - $tmp = Array(); - exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp); - if (trim($tmp[0])) - $reason = $tmp[0]; - else { - exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp); - if (trim($tmp[0])) - $reason = trim($tmp[0]); - else - $reason = "unknown"; - } + if (file_exists($thisbuildfinaldir . "reason")) + $reason = trim(file_get_contents($thisbuildfinaldir . "reason", "r")); + else { + $tmp = Array(); + exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp); + if (trim($tmp[0])) + $reason = $tmp[0]; + else { + exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp); + if (trim($tmp[0])) + $reason = trim($tmp[0]); + else + $reason = "unknown"; + } + } } /* Compress files that are typically too large and infrequently From patchwork Sun Jul 7 05:28:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128572 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AZHYUfrG"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGN3hFGz9s8m for ; Sun, 7 Jul 2019 15:30:04 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 02216844D3; Sun, 7 Jul 2019 05:30:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DGXGXdMALGcO; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6138B84506; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id C13271BF956 for ; Sun, 7 Jul 2019 05:29:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id BB24E84506 for ; Sun, 7 Jul 2019 05:29:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sux47e9gfw6a for ; Sun, 7 Jul 2019 05:29:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 4DAC7844D3 for ; Sun, 7 Jul 2019 05:29:59 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id cl9so6505738plb.10 for ; Sat, 06 Jul 2019 22:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QdIjDL0MK/1MQ0hGChhz+gL159ZMD1GlJvokF5OE3Wg=; b=AZHYUfrGuaBwdlj2kBNaSYPEIBMDfg+68qmQUypTmE1VINUdf1A2J1ppnhB3In9W1S 1g+wtnJJOtukPK7e1DnV+4vDPrJaqdPLbAsQ1NW+I2/5lEJQE4FisazONSF+l+EyhJps KD+v1JXi1C2uARpZgCylInJGh7L0+tOT0fxccWUU3JY8yQmxcoV3RErtPgDtp67Skpsu GdqTeb4plI4bngIAR0M55TC1br9pPNej05r2D2RTMTkZ78FCT7EOXn7jnvPQh6A1bsyE dxhuIf5XUCCR1ziVcFJcY9pnR+H/S7heV8N9BR2roStj27vMJzc5D5wIDbt0bB0x7v0R fZVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QdIjDL0MK/1MQ0hGChhz+gL159ZMD1GlJvokF5OE3Wg=; b=VfgqAsExqPcg3zOV4um/XQkbmWyO8I2S2cwC0hd8Oy2w3dVmFRGEk9qYVNrml+nL6j 6aHGfQKLr+aS6HXyH7B0e6ShVthPS8mnk0lOZb3JLj0MATpcGQjjxXfoF5sGpNSqVqAE MWB+ZQaADlGpaJE00Z/xBo9VIoLfDHbeP4HobLq30xPfUtD9emSXLheCxwC0fkhtv5GV xbuPFOS8F6d+uKEfmcFIR7YepR5wEYjKvgC5amf5xZ6g5BFIzyLvXNJU9NryHSGkTTZv Uz8W1uuvTrt7Odr7Kvza9vo6OpNwEtfBrD+4qC2Vb9p27If/887RLjZplWJgUgYEMdwE d8cg== X-Gm-Message-State: APjAAAVmC2ijoVOldnA6ZAl3GfTjaeqsdSc1gaNWSiVNAMhcTlCKctfn ecqP4NoS3KI1prlcvTK8nRcVH41IcYcQnQ== X-Google-Smtp-Source: APXvYqwoVdh6wyqIWSpWf0+Geo6mwsEqNd3HsO5sTdgBP27AkgJaIfKCMOJ9CkovUxp3C03AqRhiOA== X-Received: by 2002:a17:902:704a:: with SMTP id h10mr14694962plt.337.1562477398593; Sat, 06 Jul 2019 22:29:58 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.56 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:29:58 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:27 +0530 Message-Id: <20190707052831.9469-27-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 27/31] scripts/autobuild-run: support changing repo X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas Petazzoni The current logic in prepare_build() assumes that the "origin" repo never changes. However, if one regularly changes his autobuild-run configuration, switching being repository, this is not true. Currently, it requires manually wiping out the Buildroot clone in every autobuild instance when changing the repository to pull from. So instead, use: git fetch git checkout FETCH_HEAD which will easily allow switching from one repo to the other. Signed-off-by: Thomas Petazzoni [Atharva: Adapted Thomas' patches to work with Builder class] Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Incorporate Thomas' patch into Builder class series --- scripts/autobuild-run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 8011ebb..acd1547 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -361,12 +361,12 @@ class Builder: # Update the Buildroot sources. abssrcdir = os.path.abspath(self.srcdir) - ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=self.log, stderr=self.log) + ret = subprocess.call(["git", "fetch", self.repo, branch], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not fetch Buildroot sources") return -1 - ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=self.log, stderr=self.log) + ret = subprocess.call(["git", "checkout", "FETCH_HEAD"], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not check out Buildroot sources") return -1 From patchwork Sun Jul 7 05:28:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128574 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aRETRdpo"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGV1VjSz9s8m for ; Sun, 7 Jul 2019 15:30:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id AAC48204DD; Sun, 7 Jul 2019 05:30:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QY+bhYiTKVk9; Sun, 7 Jul 2019 05:30:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 64420204F9; Sun, 7 Jul 2019 05:30:05 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 348821BF5DD for ; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 317BC84506 for ; Sun, 7 Jul 2019 05:30:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5VL3q2dAnRzg for ; Sun, 7 Jul 2019 05:30:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id B6334844D3 for ; Sun, 7 Jul 2019 05:30:01 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id l21so2024151pgm.3 for ; Sat, 06 Jul 2019 22:30:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2ITJ4hs8+PTyZDtJh60/4OqpTkhbGKTsH/VMI9ajQqs=; b=aRETRdpovTIHBjiqW/XLGL/pzL6ECfbQPyOQ4RCyz6xF0fdlEotcDgejKMGDShNHPL rS0eVNvr9SaCMUOkiufSJ/k+vZVE/gZ7pvAE6bJ4l1OHngDyrpLox2nkNn90T0dsWFgC 0zjPMtGX6ptuDl789igdkO+nevkJb/jHvXLBQTg33SSAdyzwdGLBx4GdapzzI1woy1O4 diajkw0jq/cHWk/LXPFy9hf3AkNzrLi7hbN26Ed/AZVawNI88HahkLwamEoBkwtwqkwz Yha6KhNDM6qpaA4iPr4YB8RMS4kYzMX59CiwH43ysIeVHAz1XkNU1FJrilp7Gv6yiCCe psUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2ITJ4hs8+PTyZDtJh60/4OqpTkhbGKTsH/VMI9ajQqs=; b=Pf635/8RNkNfPt9MpO1UZKk8G8ulAQGiI9TqZtrej9ojNTP5mrk66OIIoArl/pJdD9 UhKMrErKcjEWdMAGROBGu+e3E6XSIlDGFgKcV1lkges2YScRtq8nK6xWtOBfSVtBB8Aa RQQQNHITqcyL8J6Q+GPILjzAC53awMUl/55DcJsCm3NAye9Vb2mYzfBh2gjGf+8C1jI1 xNUNiHd0swFEdvONV0v9de2I68LCGzjfBZ29pLGF54bsBNf8wUbA10qmH4m9P3riHEPT trauDlJIDeWBg8l+JvcYa/jh4VXMJzwoQMkZN+dPdaV5Glz3UADUxzaioO4Nmo3QZ4cX Jfow== X-Gm-Message-State: APjAAAUZnITo08Vhm5R25Ebv/2tHU0YdCeKgOtaZhyrGDiHimp9VkFOP UayrDJU0R+LoOd0QAMURocax29sSAlXrGQ== X-Google-Smtp-Source: APXvYqycJa9zql6DpUYjE1iERmAXJrSiI+e7SQsksyavM9cpsUh5tWjTKCHrZdnYzOKy3ZpDm/OHXw== X-Received: by 2002:a63:5610:: with SMTP id k16mr14440334pgb.335.1562477400958; Sat, 06 Jul 2019 22:30:00 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.29.58 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:30:00 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:28 +0530 Message-Id: <20190707052831.9469-28-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 28/31] autobuild-run: modify do_build() to accept outputdir as argument X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Pass outputdir to do_build() so in future patches we can pass different output directories to increase variance in case of reproducibility testing. Signed-off-by: Atharva Lele --- scripts/autobuild-run | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index acd1547..522421e 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -454,14 +454,14 @@ class Builder: log_write(self.log, "INFO: Build is reproducible!") return 0 - def do_build(self): + def do_build(self, outputdir): """Run the build itself""" - f = open(os.path.join(self.outputdir, "logfile"), "w+") + f = open(os.path.join(outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), - "make", "O=%s" % self.outputdir, + "make", "O=%s" % outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() @@ -473,7 +473,7 @@ class Builder: build_monitor = Thread(target=self.stop_on_build_hang, args=(monitor_thread_hung_build_flag, monitor_thread_stop_flag, - sub, self.outputdir)) + sub, outputdir)) build_monitor.daemon = True build_monitor.start() @@ -494,7 +494,7 @@ class Builder: log_write(self.log, "INFO: build failed [%d]" % ret) return -1 - cmd = ["make", "O=%s" % self.outputdir, "-C", self.srcdir, + cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) @@ -513,7 +513,7 @@ class Builder: # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") - ret = self.do_build() + ret = self.do_build(self.outputdir) if ret != 0: log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret @@ -527,7 +527,7 @@ class Builder: # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") - ret = self.do_build() + ret = self.do_build(self.outputdir) if ret != 0: log_write(self.log, "INFO: build 2 failed") return ret From patchwork Sun Jul 7 05:28:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128580 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="PZwQfYsT"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGn3qZQz9s8m for ; Sun, 7 Jul 2019 15:30:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 7BE0420503; Sun, 7 Jul 2019 05:30:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id guZpIz7WB3Ka; Sun, 7 Jul 2019 05:30:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 5008720472; Sun, 7 Jul 2019 05:30:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 377EB1BF5DD for ; Sun, 7 Jul 2019 05:30:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3312484AF4 for ; Sun, 7 Jul 2019 05:30:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 13q8sqI7dCgq for ; Sun, 7 Jul 2019 05:30:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 1FE52855CC for ; Sun, 7 Jul 2019 05:30:04 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id i189so5998044pfg.10 for ; Sat, 06 Jul 2019 22:30:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BWPlX7YnSIqy8+SmQJ9cCHVhNdZNa7A6W21lonMrq4o=; b=PZwQfYsT3SdD9G/whvmcCdDS2KxMaUu9Q4FjD/Z50/7oJkVAOQYjvH73vCiIejuSUC 2KSLO0mj4x9aIuRvkkjFzEqNBlxhS2iatQ1WaTuwouAXDBypz/DE71ziYCpchLaRgw9I o6YVQRcMj/hwXgGji8/ndtkCnE/tXNhcdj/fqMm5f7T+ps+7cBsfUfMINCcin9nJMmeR +aszFcA5XjLgqdjbiFt7CvOBTCVg0zlRIluUaaOGFHUg+zw9Nx6s2/WDtctSlvYeK2Jk TVs7fHRoShFKAx2AK+kTCJLUjdTBr6YlVEKuYvj4OLXbOjsMCwFMsO3LZxuyzV5J9EoL 6Asw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BWPlX7YnSIqy8+SmQJ9cCHVhNdZNa7A6W21lonMrq4o=; b=Q83saTwv95YmApmgX6fqNtefDTxu92Z2L2i47Xq8b71JIQimONTZk/shGwQ1kLStgb fqgTLJXLIjizYMQBRr9XIwnq69omUW5+SjYRNjpI/odau/JwXTHwy44JO6uBKoTAzwb2 lu2pE1y7ILceNnQKn6tkRmBer2gKiTy3kxlaVZBiEMos5qAwhc8+DaO8MzR6Mhv9SVxN g+/lEwbxQQSNiMeq2SephVp458RG1FEJMc6pIyNUyVr2IDG8u87whZn4nnU5nSu+5wQ+ pHOsqUX1/VvQy6NhFsZWuFospDm7tuzVFwF7glf0y8E0L8MoeddnWrvB3bUCPlSwj6eq 8Zjg== X-Gm-Message-State: APjAAAUsZJcnLKUgCh3OXSR8FQkr31YRIPdXEpJ1ie7B92c5ZLOXZFmy d1UMf+BJ0VKmmufm77+nEwf3v+AA7JiwWQ== X-Google-Smtp-Source: APXvYqy/es42WkOJPnxkM4woZc734TxpOqbpz9onmDASbOPnSGkpirjjo91GjWVjh7k4oEpaQ9709g== X-Received: by 2002:a17:90a:cb87:: with SMTP id a7mr15843164pju.130.1562477403377; Sat, 06 Jul 2019 22:30:03 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.30.01 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:30:03 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:29 +0530 Message-Id: <20190707052831.9469-29-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 29/31] autobuild-run: define different output directory for reproducible builds X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Having different output directories will increase variance. Subsequent patches will modify other parts of the scripts to use this directory. Signed-off-by: Atharva Lele --- scripts/autobuild-run | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 522421e..84212c6 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -278,6 +278,10 @@ class Builder: self.outputdir = os.path.abspath(os.path.join(self.idir, "output")) self.resultdir = os.path.join(self.outputdir, "results") + # We use a different output directory for more variance in reproducibility + # testing. + self.outputdir_1 = os.path.abspath(os.path.join(self.idir, "output-1")) + if self.debug: self.log = sys.stdout else: From patchwork Sun Jul 7 05:28:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128579 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VUrge/ly"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGl2yQxz9s8m for ; Sun, 7 Jul 2019 15:30:23 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 258D9859C9; Sun, 7 Jul 2019 05:30:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J5xxhqmu0dup; Sun, 7 Jul 2019 05:30:16 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 0DE3D86A07; Sun, 7 Jul 2019 05:30:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 96D001BF5DD for ; Sun, 7 Jul 2019 05:30:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 931878527F for ; Sun, 7 Jul 2019 05:30:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UNn4VrYQuDU8 for ; Sun, 7 Jul 2019 05:30:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by whitealder.osuosl.org (Postfix) with ESMTPS id 8DB83810BD for ; Sun, 7 Jul 2019 05:30:06 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id r1so5985568pfq.12 for ; Sat, 06 Jul 2019 22:30:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yEZqti3IuI/c7LhS6yfZcxv/8QAMnedp/eD4w8Nmf6M=; b=VUrge/lyZkN+JA2TJAvr8JGYDaZ27YE6FFCqSPFf2yd9y7Iff5KQxOGnCJOnnkPmBF b2RETAchDUk3fejv4U3+RUU/ktNXeHf3+mhrQd65lURSJpmdXbSRcUnz1BXuj/po9RQ/ 3KQxZAutH7mruimJK/K5Em3MF0didahdUKwOSjm9ew/dMoJdw+3lgwGWfgTX94q5K5xj Uou+xdS7oF7qpt3UKjOcwpneKgU5hV3+VyxC59/4nflV3V55EazgyAVkmUWgxLohZTMb XPJX/jYJ9cycP38mpOtLunK8VD5QBfFJZ7F/ctBeqi4QhtzcmMrChmUUjijIPWL9Ah20 KUJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yEZqti3IuI/c7LhS6yfZcxv/8QAMnedp/eD4w8Nmf6M=; b=OGLh8F8XbUQ8kUkdheLkR/Ys+/ptFg11s2NAS81ub/goNIc//AGf+1REZ/FklrTw1h w8dwwO6yIMWktFXC27GxcwDLWuN+Aw05aHyDNeid/z8MZ1mxjNbcPE8L8HykQelYPFV5 Ns/IcJPRV+7q+lsau2rL4afYh2xJ9PseEkKw6DO1OU/RU5wGRbvlNB2oX1kNNYEET2+B +tlIIjkyaONAxSdbq9T8lQ+5BlIq9qIFWunIaepMbWJR5mc2sPS0E2V1bTmnWrPt0tWL YXxeVZnMTCxw7uWgJAj87RziKFS68QNBDkl25Hvs/z+jQ1iPIpsc2c3bKuGiiZQ6ETgu ITGA== X-Gm-Message-State: APjAAAWFxXVK+XygSmUBKB0wPGcbkvV3p6Qq6E1gNRMGTgfmxA15Gjt2 AKAnqmsjmWD1El/H49O+TVZv9JryrVSlTw== X-Google-Smtp-Source: APXvYqyeqTOo2bDGVlJr+97ZVoSDNzI4/MPNB6agRj3vDAKSLh+0qv2cTqPJnL/sUlpAA6+YGZ4XVA== X-Received: by 2002:a65:4d4e:: with SMTP id j14mr14306896pgt.50.1562477405710; Sat, 06 Jul 2019 22:30:05 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.30.03 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:30:05 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:30 +0530 Message-Id: <20190707052831.9469-30-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 30/31] autobuild-run: use different output directories for reproducible builds testing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- scripts/autobuild-run | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 84212c6..cdf013c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -434,8 +434,8 @@ class Builder: reproducible_results = os.path.join(self.outputdir, "results", "reproducible_results") # Using only tar images for now - build_1_image = os.path.join(self.outputdir, "images-1", "rootfs.tar") - build_2_image = os.path.join(self.outputdir, "images", "rootfs.tar") + build_1_image = os.path.join(self.outputdir, "images", "rootfs.tar") + build_2_image = os.path.join(self.outputdir_1, "images", "rootfs.tar") with open(reproducible_results, 'w') as diff: if self.sysinfo.has("diffoscope"): @@ -522,16 +522,17 @@ class Builder: log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret - # First build has been built, move files and start build 2 - os.rename(os.path.join(self.outputdir, "images"), os.path.join(self.outputdir, "images-1")) + # Create the second output directory if it does not exist + if not os.path.exists(self.outputdir_1): + os.mkdir(self.outputdir_1) - # Clean up build 1 - f = open(os.path.join(self.outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % self.outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) + # Copy .config to the other output directory + shutil.copyfile(os.path.join(self.outputdir, ".config"), + os.path.join(self.outputdir_1, ".config")) # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") - ret = self.do_build(self.outputdir) + ret = self.do_build(self.outputdir_1) if ret != 0: log_write(self.log, "INFO: build 2 failed") return ret From patchwork Sun Jul 7 05:28:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1128578 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ksoVDY/T"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45hHGh1J6Yz9s8m for ; Sun, 7 Jul 2019 15:30:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id AA96585567; Sun, 7 Jul 2019 05:30:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BfA3pmqlqKmi; Sun, 7 Jul 2019 05:30:17 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 903BB85BBD; Sun, 7 Jul 2019 05:30:17 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 34EAC1BF5DD for ; Sun, 7 Jul 2019 05:30:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1EE1F8243D for ; Sun, 7 Jul 2019 05:30:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7CHBIHqDlbhh for ; Sun, 7 Jul 2019 05:30:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by hemlock.osuosl.org (Postfix) with ESMTPS id 0F809859C9 for ; Sun, 7 Jul 2019 05:30:09 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id t132so6017956pgb.9 for ; Sat, 06 Jul 2019 22:30:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lSfEGvekgujksV7YgU7t3ILeHlMzktE5pGyjyeBgdMQ=; b=ksoVDY/TGPtfW4S0SW5GpzTblgQNndWc6Sui4bM7QNwe9njDdGdTj/7rlHlzM8JLl1 Bsv6Gg5srrF2RGQPIsIqdlImjQKlzl3+Gg36RLTbqStmrEftnnvFzTIcel6jadXWSVcb lhrQMMBIYLL8xwBK/rJnAePVItp08jK6MgoAln5obAPjnX3Mx4EDnZQWg/EWVuttmL3z i1hikkNL1jrvSB6Tc8Vy+NlfdOTbmSxDV8TZmHNIHCVqhiyJEcG2jDOPsAguQXUcdHFf Yr+XShEhsqrsQl8aI4bkfN9wW0jMZlubWgQyujYyYqHuVgCz90l2tDbPGmJCmb5sRYcK aPxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lSfEGvekgujksV7YgU7t3ILeHlMzktE5pGyjyeBgdMQ=; b=tWqxwOEkAWZSGy0f0e4uA4qxD7bZm3uTyABpeeRr6VQux240+dEbFPLDEfDxH220rB u2omc+01180DgxfQZ+ZdSaeB0OEFuIkBao00EiaGI66eqvtbGjiRF6+/JyOen6WxiSYF tia31vgH7+tYV8gFKUXUvLdtjDtHWUcsmVcMkI10b6A8JNf21pHDdlbdaGTKDo+Ci5FJ IVsxFWEHkVAbjwp3TVu4BJA98CoVQTX+Plve0H0oVNEf/O3meCRjg3BU4UkR3WGGXXk/ 31HAbhSbDUeaeE9RUYR+mU3hG5A/n0/PeNhpmArLbQsjwUR8taXc73PlOIwJQ/oYlDX1 jyYg== X-Gm-Message-State: APjAAAU1ReIiSydUnPSdO5Ab007B7xLelLT/Ixi58dhQtC9uscOXXK2N JWlU0BmMgmw3kZczNeLIrPV35q7OSO22Dg== X-Google-Smtp-Source: APXvYqxAEueRIkKXbuR+ex1KN5q7tgjRVdvj+iMG+DnyuaRBRXh9EnHVfjxcy6BqHlMtGS7x8CEj0A== X-Received: by 2002:a63:2148:: with SMTP id s8mr14132494pgm.336.1562477408248; Sat, 06 Jul 2019 22:30:08 -0700 (PDT) Received: from localhost.localdomain ([123.201.194.71]) by smtp.gmail.com with ESMTPSA id j24sm18039214pgg.86.2019.07.06.22.30.05 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Sat, 06 Jul 2019 22:30:07 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Sun, 7 Jul 2019 10:58:31 +0530 Message-Id: <20190707052831.9469-31-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190707052831.9469-1-itsatharva@gmail.com> References: <20190707052831.9469-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 31/31] autobuild-run: make prepare_build() clean the output directory used for reproducibility testing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- scripts/autobuild-run | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index cdf013c..3156e9d 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -380,6 +380,10 @@ class Builder: # shutil.rmtree doesn't remove write-protected files subprocess.call(["rm", "-rf", self.outputdir]) os.mkdir(self.outputdir) + + # If it exists, remove the other output directory used for reproducibility testing + if os.path.exists(self.outputdir_1): + subprocess.call(["rm", "-rf", self.outputdir_1]) with open(os.path.join(self.outputdir, "branch"), "w") as branchf: branchf.write(branch)