From patchwork Thu Aug 1 02:46:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140199 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="B3hZucip"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZSl2SYFz9sNf for ; Thu, 1 Aug 2019 12:47:02 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 00FF087E14; Thu, 1 Aug 2019 02:47:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kOfBAkeBJj0R; Thu, 1 Aug 2019 02:46:58 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id A99BC87E19; Thu, 1 Aug 2019 02:46:58 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 576CA1BF59D for ; Thu, 1 Aug 2019 02:46:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5436C87E19 for ; Thu, 1 Aug 2019 02:46:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FAkF9LmcWW0f for ; Thu, 1 Aug 2019 02:46:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by hemlock.osuosl.org (Postfix) with ESMTPS id 2FEF087E14 for ; Thu, 1 Aug 2019 02:46:55 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id w10so33186783pgj.7 for ; Wed, 31 Jul 2019 19:46:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JWsr5krGaAh5zk5p3jkSRk1l6yA/OJpF2OW6YBQgf8I=; b=B3hZucipCiCEq8sjYvreDlDq9UTCOauE49mZy5IVKy+GCM161a/oXRyXn8D06v0vS9 87o8G7aQmdGlcWBJY9crAnZhMtXYKOkjCsNbbeDWjpyXBUEtotBYingYQKwhzSJq/VUU QK8V4Ne16Hx/J/lMv/Yh4rMUi0TIGdsGJL6n/xwwhwRBqJQIgcuNgChwE+ZKSkYGy1+Z 9/gOYUSSw++cQogxpW4Smu3ttbBB36VqUCK6yqk2UqBWFjVSmAV0AEOUyws/G4hFafjr wmIS3woTHE/vkEo8EiJKkKWgrxRoRYkDER0GqW1xEy4hUBAkxN1KTE6bvTsF7pf2dGPU 629Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JWsr5krGaAh5zk5p3jkSRk1l6yA/OJpF2OW6YBQgf8I=; b=amyOieTY8ZI4b25hKQ5ygXf2J22zixr5D+w5TgFKb5yKxTDlvk6r4yP3BgAqMw3opr qahuQ75MMHi5ADH9PI9AWugbz+lvIo6LjfYWzyWIMPf7WZX8jf8GmVu+iXJoaLxpxGTy u+aXhQLundfKcinY/XSBKBRXDQXuhf4yrv3onun8nQ6DKxA2rt8b5shguoe9GC13wONu 8mmgGtfMekeWiKW0v5SscP9uds2+pnHR5OASH/osK7DtKAA8Ouj4FzvdHmB99TaHkJv1 /fbT9UyKKNpuChxAmn/DBL7kdik5mTEP7FWywrwpgSYvwNdzGcjdQzpNlLuNzOJdqd82 uk4w== X-Gm-Message-State: APjAAAVMIBTeOITny0IGdpu6mqrJIu9crKfjzZjmqUC3W0+lovOGD/mZ 1p5wqdyUpWLriRFmJ/Vq+WS5pz43mcA= X-Google-Smtp-Source: APXvYqwV4FVwB7fHkgdVgQm90SP+w2/Tw7jfezjxbI6vJpb3ofTcZspVFnXMJoUR05n5b69K3bl1Og== X-Received: by 2002:a65:55c7:: with SMTP id k7mr86280965pgs.305.1564627614125; Wed, 31 Jul 2019 19:46:54 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.46.51 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:46:53 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:14 +0530 Message-Id: <20190801024643.11024-2-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 01/30] autobuild-run: introduce Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Various functions in the autobuild-run script use a lot of common data. To make it easier to work with, create a Builder class. For ease of review, this commit only introduces the Builder class but does not actually use it for anything. Subsequent patches will do that. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Fix indentation issues (suggested by Arnout) --- scripts/autobuild-run | 858 +++++++++++++++++++++--------------------- 1 file changed, 430 insertions(+), 428 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 601fb31..6bd6856 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -270,473 +270,474 @@ class SystemInfo: return not missing_requirements -def prepare_build(**kwargs): - """Prepare for the next build of the specified instance +class Builder: + def prepare_build(self, **kwargs): + """Prepare for the next build of the specified instance - This function prepares the build by making sure all the needed - directories are created, cloning or updating the Buildroot source - code, and cleaning up remaining stuff from previous builds. - """ + This function prepares the build by making sure all the needed + directories are created, cloning or updating the Buildroot source + code, and cleaning up remaining stuff from previous builds. + """ - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - - log_write(log, "INFO: preparing a new build") - - # Create the download directory if it doesn't exist - dldir = os.path.join(idir, "dl") - if not os.path.exists(dldir): - os.mkdir(dldir) - - # recursively find files under root - def find_files(root): - for r, d, f in os.walk(root): - # do not remove individual files from git caches. 'git' can - # be either dl//git or dl/git and we want to - # eventually remove tarballs for the git package, so check - # for '.git' instead to match only dl//git/.git . - if '.git' in d: - del d[:] - continue - for i in f: - yield os.path.join(r, i) - - # Remove 5 random files from the download directory. Removing - # random files from the download directory allows to ensure we - # regularly re-download files to check that their upstream - # location is still correct. - for i in range(0, 5): - flist = list(find_files(dldir)) - if not flist: - break - f = flist[randint(0, len(flist) - 1)] - log_write(log, "INFO: removing %s from downloads" % - os.path.relpath(f, dldir)) - os.remove(f) - - branch = get_branch() - log_write(log, "INFO: testing branch '%s'" % branch) - - # Clone Buildroot. This only happens if the source directory - # didn't exist already. - srcdir = os.path.join(idir, "buildroot") - if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", kwargs['repo'], srcdir], - stdout=log, stderr=log) + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] + + log_write(log, "INFO: preparing a new build") + + # Create the download directory if it doesn't exist + dldir = os.path.join(idir, "dl") + if not os.path.exists(dldir): + os.mkdir(dldir) + + # recursively find files under root + def find_files(root): + for r, d, f in os.walk(root): + # do not remove individual files from git caches. 'git' can + # be either dl//git or dl/git and we want to + # eventually remove tarballs for the git package, so check + # for '.git' instead to match only dl//git/.git . + if '.git' in d: + del d[:] + continue + for i in f: + yield os.path.join(r, i) + + # Remove 5 random files from the download directory. Removing + # random files from the download directory allows to ensure we + # regularly re-download files to check that their upstream + # location is still correct. + for i in range(0, 5): + flist = list(find_files(dldir)) + if not flist: + break + f = flist[randint(0, len(flist) - 1)] + log_write(log, "INFO: removing %s from downloads" % + os.path.relpath(f, dldir)) + os.remove(f) + + branch = get_branch() + log_write(log, "INFO: testing branch '%s'" % branch) + + # Clone Buildroot. This only happens if the source directory + # didn't exist already. + srcdir = os.path.join(idir, "buildroot") + if not os.path.exists(srcdir): + ret = subprocess.call(["git", "clone", kwargs['repo'], srcdir], + stdout=log, stderr=log) + if ret != 0: + log_write(log, "ERROR: could not clone Buildroot sources") + return -1 + + # Update the Buildroot sources. + abssrcdir = os.path.abspath(srcdir) + ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=log, stderr=log) if ret != 0: - log_write(log, "ERROR: could not clone Buildroot sources") + log_write(log, "ERROR: could not fetch Buildroot sources") return -1 - # Update the Buildroot sources. - abssrcdir = os.path.abspath(srcdir) - ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=log, stderr=log) - if ret != 0: - log_write(log, "ERROR: could not fetch Buildroot sources") - return -1 - - ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=log, stderr=log) - if ret != 0: - log_write(log, "ERROR: could not check out Buildroot sources") - return -1 - - # Create an empty output directory. We remove it first, in case a previous build was aborted. - outputdir = os.path.join(idir, "output") - if os.path.exists(outputdir): - # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", outputdir]) - os.mkdir(outputdir) - with open(os.path.join(outputdir, "branch"), "w") as branchf: - branchf.write(branch) - - return 0 - -def gen_config(**kwargs): - """Generate a new random configuration.""" - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - - log_write(log, "INFO: generate the configuration") - - if kwargs['debug']: - devnull = log - else: - devnull = open(os.devnull, "w") - - args = [os.path.join(srcdir, "utils/genrandconfig"), - "-o", outputdir, "-b", srcdir] - - toolchains_csv = kwargs['toolchains_csv'] - if toolchains_csv: - if not os.path.isabs(toolchains_csv): - toolchains_csv = os.path.join(srcdir, toolchains_csv) - args.extend(["--toolchains-csv", toolchains_csv]) - - ret = subprocess.call(args, stdout=devnull, stderr=log) - return ret - -def stop_on_build_hang(monitor_thread_hung_build_flag, - monitor_thread_stop_flag, - sub_proc, outputdir, log): - build_time_logfile = os.path.join(outputdir, "build/build-time.log") - while True: - if monitor_thread_stop_flag.is_set(): - return - if os.path.exists(build_time_logfile): - mtime = datetime.datetime.fromtimestamp(os.stat(build_time_logfile).st_mtime) - - if mtime < datetime.datetime.now() - datetime.timedelta(minutes=HUNG_BUILD_TIMEOUT): - if sub_proc.poll() is None: - monitor_thread_hung_build_flag.set() # Used by do_build() to determine build hang - log_write(log, "INFO: build hung") - sub_proc.kill() - break - monitor_thread_stop_flag.wait(30) + ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=log, stderr=log) + if ret != 0: + log_write(log, "ERROR: could not check out Buildroot sources") + return -1 -def check_reproducibility(**kwargs): - """Check reproducibility of builds + # Create an empty output directory. We remove it first, in case a previous build was aborted. + outputdir = os.path.join(idir, "output") + if os.path.exists(outputdir): + # shutil.rmtree doesn't remove write-protected files + subprocess.call(["rm", "-rf", outputdir]) + os.mkdir(outputdir) + with open(os.path.join(outputdir, "branch"), "w") as branchf: + branchf.write(branch) + + return 0 + + def gen_config(self, **kwargs): + """Generate a new random configuration.""" + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") - Use diffoscope on the built images, if diffoscope is not - installed, fallback to cmp - """ + log_write(log, "INFO: generate the configuration") - log = kwargs['log'] - idir = "instance-%d" % kwargs['instance'] - outputdir = os.path.join(idir, "output") - srcdir = os.path.join(idir, "buildroot") - reproducible_results = os.path.join(outputdir, "results", "reproducible_results") - # Using only tar images for now - build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") - build_2_image = os.path.join(outputdir, "images", "rootfs.tar") - - with open(reproducible_results, 'w') as diff: - if kwargs['sysinfo'].has("diffoscope"): - # Prefix to point diffoscope towards cross-tools - prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) - # Remove TARGET_CROSS= and \n from the string - prefix = prefix[13:-1] - log_write(log, "INFO: running diffoscope on images") - subprocess.call(["diffoscope", build_1_image, build_2_image, - "--tool-prefix-binutils", prefix], stdout=diff, stderr=log) + if kwargs['debug']: + devnull = log else: - log_write(log, "INFO: diffoscope not installed, falling back to cmp") - subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=log) - - if os.stat(reproducible_results).st_size > 0: - log_write(log, "INFO: Build is non-reproducible.") - return -1 - - # rootfs images match byte-for-byte -> reproducible image - log_write(log, "INFO: Build is reproducible!") - return 0 - -def do_build(**kwargs): - """Run the build itself""" - - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - nice = kwargs['nice'] - - # We need the absolute path to use with O=, because the relative - # path to the output directory here is not relative to the - # Buildroot sources, but to the location of the autobuilder - # script. - dldir = os.path.abspath(os.path.join(idir, "dl")) - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - f = open(os.path.join(outputdir, "logfile"), "w+") - log_write(log, "INFO: build started") - - cmd = ["nice", "-n", str(nice), - "make", "O=%s" % outputdir, - "-C", srcdir, "BR2_DL_DIR=%s" % dldir, - "BR2_JLEVEL=%s" % kwargs['njobs']] \ - + kwargs['make_opts'].split() - sub = subprocess.Popen(cmd, stdout=f, stderr=f) - - # Setup hung build monitoring thread - monitor_thread_hung_build_flag = Event() - monitor_thread_stop_flag = Event() - build_monitor = Thread(target=stop_on_build_hang, - args=(monitor_thread_hung_build_flag, - monitor_thread_stop_flag, - sub, outputdir, log)) - build_monitor.daemon = True - build_monitor.start() - - kwargs['buildpid'][kwargs['instance']] = sub.pid - ret = sub.wait() - kwargs['buildpid'][kwargs['instance']] = 0 - - # If build failed, monitor thread would have exited at this point - if monitor_thread_hung_build_flag.is_set(): - log_write(log, "INFO: build timed out [%d]" % ret) - return -2 - else: - # Stop monitor thread as this build didn't timeout - monitor_thread_stop_flag.set() - # Monitor thread should be exiting around this point - - if ret != 0: - log_write(log, "INFO: build failed [%d]" % ret) - return -1 - - cmd = ["make", "O=%s" % outputdir, "-C", srcdir, - "BR2_DL_DIR=%s" % dldir, "legal-info"] \ - + kwargs['make_opts'].split() - ret = subprocess.call(cmd, stdout=f, stderr=f) - if ret != 0: - log_write(log, "INFO: build failed during legal-info") - return -1 - log_write(log, "INFO: build successful") - return 0 - -def do_reproducible_build(**kwargs): - """Run the builds for reproducibility testing - - Build twice with the same configuration. Calls do_build() to - perform the actual build. - """ + devnull = open(os.devnull, "w") + + args = [os.path.join(srcdir, "utils/genrandconfig"), + "-o", outputdir, "-b", srcdir] - idir = "instance-%d" % kwargs['instance'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - log = kwargs['log'] + toolchains_csv = kwargs['toolchains_csv'] + if toolchains_csv: + if not os.path.isabs(toolchains_csv): + toolchains_csv = os.path.join(srcdir, toolchains_csv) + args.extend(["--toolchains-csv", toolchains_csv]) - # Start the first build - log_write(log, "INFO: Reproducible Build Test, starting build 1") - ret = do_build(**kwargs) - if ret != 0: - log_write(log, "INFO: build 1 failed, skipping build 2") + ret = subprocess.call(args, stdout=devnull, stderr=log) return ret - # First build has been built, move files and start build 2 - os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1")) + def stop_on_build_hang(self, monitor_thread_hung_build_flag, + monitor_thread_stop_flag, sub_proc, + outputdir, log): + build_time_logfile = os.path.join(outputdir, "build/build-time.log") + while True: + if monitor_thread_stop_flag.is_set(): + return + if os.path.exists(build_time_logfile): + mtime = datetime.datetime.fromtimestamp(os.stat(build_time_logfile).st_mtime) + + if mtime < datetime.datetime.now() - datetime.timedelta(minutes=HUNG_BUILD_TIMEOUT): + if sub_proc.poll() is None: + monitor_thread_hung_build_flag.set() # Used by do_build() to determine build hang + log_write(log, "INFO: build hung") + sub_proc.kill() + break + monitor_thread_stop_flag.wait(30) + + def check_reproducibility(self, **kwargs): + """Check reproducibility of builds + + Use diffoscope on the built images, if diffoscope is not + installed, fallback to cmp + """ - # Clean up build 1 - f = open(os.path.join(outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) + log = kwargs['log'] + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.join(idir, "output") + srcdir = os.path.join(idir, "buildroot") + reproducible_results = os.path.join(outputdir, "results", "reproducible_results") + # Using only tar images for now + build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") + build_2_image = os.path.join(outputdir, "images", "rootfs.tar") + + with open(reproducible_results, 'w') as diff: + if kwargs['sysinfo'].has("diffoscope"): + # Prefix to point diffoscope towards cross-tools + prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) + # Remove TARGET_CROSS= and \n from the string + prefix = prefix[13:-1] + log_write(log, "INFO: running diffoscope on images") + subprocess.call(["diffoscope", build_1_image, build_2_image, + "--tool-prefix-binutils", prefix], stdout=diff, stderr=log) + else: + log_write(log, "INFO: diffoscope not installed, falling back to cmp") + subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=log) - # Start the second build - log_write(log, "INFO: Reproducible Build Test, starting build 2") - ret = do_build(**kwargs) - if ret != 0: - log_write(log, "INFO: build 2 failed") - return ret + if os.stat(reproducible_results).st_size > 0: + log_write(log, "INFO: Build is non-reproducible.") + return -1 - # Assuming both have built successfully - ret = check_reproducibility(**kwargs) - return ret + # rootfs images match byte-for-byte -> reproducible image + log_write(log, "INFO: Build is reproducible!") + return 0 -def send_results(result, **kwargs): - """Prepare and store/send tarball with results + def do_build(self, **kwargs): + """Run the build itself""" - This function prepares the tarball with the results, and either - submits them to the official server (if the appropriate credentials - are available) or stores them locally as tarballs. - """ + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] + nice = kwargs['nice'] - idir = "instance-%d" % kwargs['instance'] - log = kwargs['log'] - - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") - resultdir = os.path.join(outputdir, "results") - - shutil.copyfile(os.path.join(outputdir, ".config"), - os.path.join(resultdir, "config")) - shutil.copyfile(os.path.join(outputdir, "defconfig"), - os.path.join(resultdir, "defconfig")) - shutil.copyfile(os.path.join(outputdir, "branch"), - os.path.join(resultdir, "branch")) - - def copy_if_exists(directory, src, dst=None): - if os.path.exists(os.path.join(outputdir, directory, src)): - shutil.copyfile(os.path.join(outputdir, directory, src), - os.path.join(resultdir, src if dst is None else dst)) - - copy_if_exists("build", "build-time.log") - copy_if_exists("build", "packages-file-list.txt") - copy_if_exists("build", "packages-file-list-host.txt") - copy_if_exists("build", "packages-file-list-staging.txt") - copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv") - - subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ - os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) - - # Return True if the result should be rejected, False otherwise - def reject_results(): - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() - - # Reject results where qemu-user refused to build - regexp = re.compile(r'^package/qemu/qemu.mk:.*Refusing to build qemu-user') - for line in lastlines: - if regexp.match(line): - return True - - return False - - if reject_results(): - return - - def get_failure_reason(): - # Output is a tuple (package, version), or None. - lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], - stdout=subprocess.PIPE).communicate()[0]).splitlines() - - regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') - for line in lastlines: - m = regexp.search(line) - if m: - return m.group(1).rsplit('-', 1) - - # not found - return None + # We need the absolute path to use with O=, because the relative + # path to the output directory here is not relative to the + # Buildroot sources, but to the location of the autobuilder + # script. + dldir = os.path.abspath(os.path.join(idir, "dl")) + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") + f = open(os.path.join(outputdir, "logfile"), "w+") + log_write(log, "INFO: build started") + + cmd = ["nice", "-n", str(nice), + "make", "O=%s" % outputdir, + "-C", srcdir, "BR2_DL_DIR=%s" % dldir, + "BR2_JLEVEL=%s" % kwargs['njobs']] \ + + kwargs['make_opts'].split() + sub = subprocess.Popen(cmd, stdout=f, stderr=f) + + # Setup hung build monitoring thread + monitor_thread_hung_build_flag = Event() + monitor_thread_stop_flag = Event() + build_monitor = Thread(target=self.stop_on_build_hang, + args=(monitor_thread_hung_build_flag, + monitor_thread_stop_flag, + sub, outputdir, log)) + build_monitor.daemon = True + build_monitor.start() + + kwargs['buildpid'][kwargs['instance']] = sub.pid + ret = sub.wait() + kwargs['buildpid'][kwargs['instance']] = 0 + + # If build failed, monitor thread would have exited at this point + if monitor_thread_hung_build_flag.is_set(): + log_write(log, "INFO: build timed out [%d]" % ret) + return -2 + else: + # Stop monitor thread as this build didn't timeout + monitor_thread_stop_flag.set() + # Monitor thread should be exiting around this point - def extract_end_log(resultfile): - """Save the last part of the build log, starting from the failed package""" + if ret != 0: + log_write(log, "INFO: build failed [%d]" % ret) + return -1 - def extract_last_500_lines(): - subprocess.call(["tail -500 %s > %s" % \ - (os.path.join(outputdir, "logfile"), resultfile)], - shell=True) + cmd = ["make", "O=%s" % outputdir, "-C", srcdir, + "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + + kwargs['make_opts'].split() + ret = subprocess.call(cmd, stdout=f, stderr=f) + if ret != 0: + log_write(log, "INFO: build failed during legal-info") + return -1 + log_write(log, "INFO: build successful") + return 0 - reason = get_failure_reason() - if not reason: - extract_last_500_lines() - else: - f = open(os.path.join(outputdir, "logfile"), 'r') - mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) - mf.seek(0) - # Search for first action on the failed package - offset = mf.find(encode_str('>>> %s' % ' '.join(reason))) - if offset != -1: - with open(resultfile, "w") as endlog: - endlog.write(decode_bytes(mf[offset:])) - else: - # not found, use last 500 lines as fallback - extract_last_500_lines() + def do_reproducible_build(self, **kwargs): + """Run the builds for reproducibility testing - mf.close() - f.close() + Build twice with the same configuration. Calls do_build() to + perform the actual build. + """ - extract_end_log(os.path.join(resultdir, "build-end.log")) + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") + log = kwargs['log'] - def copy_config_log_files(): - """Recursively copy any config.log files from the failing package""" + # Start the first build + log_write(log, "INFO: Reproducible Build Test, starting build 1") + ret = self.do_build(**kwargs) + if ret != 0: + log_write(log, "INFO: build 1 failed, skipping build 2") + return ret - reason = get_failure_reason() - if not reason: - return + # First build has been built, move files and start build 2 + os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1")) - srcroot = os.path.join(outputdir, "build", '-'.join(reason)) - destroot = os.path.join(resultdir, '-'.join(reason)) - config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', - 'CMakeOutput.log') - - for root, dirs, files in os.walk(srcroot): - dest = os.path.join(destroot, os.path.relpath(root, srcroot)) - - for fname in files: - if fname in config_files: - if not os.path.exists(dest): - os.makedirs(dest) - shutil.copy(os.path.join(root, fname), os.path.join(dest, fname)) - - copy_config_log_files() - - resultf = open(os.path.join(resultdir, "status"), "w+") - if result == 0: - resultf.write("OK") - elif result == -1: - resultf.write("NOK") - elif result == -2: - resultf.write("TIMEOUT") - resultf.close() - - with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: - submitterf.write(kwargs['submitter']) - - # Yes, shutil.make_archive() would be nice, but it doesn't exist - # in Python 2.6. - ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) - if ret != 0: - log_write(log, "ERROR: could not make results tarball") - sys.exit(1) + # Clean up build 1 + f = open(os.path.join(outputdir, "logfile"), "w+") + subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) - if kwargs['upload']: - # Submit results. Yes, Python has some HTTP libraries, but - # none of the ones that are part of the standard library can - # upload a file without writing dozens of lines of code. - ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), - "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), - "-F", "uploadsubmit=1", - kwargs['http_url']], - stdout=log, stderr=log) + # Start the second build + log_write(log, "INFO: Reproducible Build Test, starting build 2") + ret = self.do_build(**kwargs) if ret != 0: - log_write(log, "INFO: results could not be submitted, %d" % ret) - else: - log_write(log, "INFO: results were submitted successfully") - else: - # No http login/password, keep tarballs locally - with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: - sha1 = hashlib.sha1(f.read()).hexdigest() - resultfilename = "instance-%d-%s.tar.bz2" % (kwargs['instance'], sha1) - os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) - log_write(log, "INFO: results saved as %s" % resultfilename) - -def run_instance(**kwargs): - """Main per-instance loop - - Prepare the build, generate a configuration, run the build, and submit the - results. - """ + log_write(log, "INFO: build 2 failed") + return ret - idir = "instance-%d" % kwargs['instance'] + # Assuming both have built successfully + ret = self.check_reproducibility(**kwargs) + return ret - # If it doesn't exist, create the instance directory - if not os.path.exists(idir): - os.mkdir(idir) + def send_results(self, result, **kwargs): + """Prepare and store/send tarball with results - if kwargs['debug']: - kwargs['log'] = sys.stdout - else: - kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") - log_write(kwargs['log'], "INFO: instance started") + This function prepares the tarball with the results, and either + submits them to the official server (if the appropriate credentials + are available) or stores them locally as tarballs. + """ - while True: - check_version() + idir = "instance-%d" % kwargs['instance'] + log = kwargs['log'] - ret = prepare_build(**kwargs) - if ret != 0: - continue + outputdir = os.path.abspath(os.path.join(idir, "output")) + srcdir = os.path.join(idir, "buildroot") + resultdir = os.path.join(outputdir, "results") + + shutil.copyfile(os.path.join(outputdir, ".config"), + os.path.join(resultdir, "config")) + shutil.copyfile(os.path.join(outputdir, "defconfig"), + os.path.join(resultdir, "defconfig")) + shutil.copyfile(os.path.join(outputdir, "branch"), + os.path.join(resultdir, "branch")) + + def copy_if_exists(directory, src, dst=None): + if os.path.exists(os.path.join(outputdir, directory, src)): + shutil.copyfile(os.path.join(outputdir, directory, src), + os.path.join(resultdir, src if dst is None else dst)) + + copy_if_exists("build", "build-time.log") + copy_if_exists("build", "packages-file-list.txt") + copy_if_exists("build", "packages-file-list-host.txt") + copy_if_exists("build", "packages-file-list-staging.txt") + copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv") + + subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ + os.path.join(resultdir, "gitid")], + shell=True, cwd=srcdir) + + # Return True if the result should be rejected, False otherwise + def reject_results(): + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() + + # Reject results where qemu-user refused to build + regexp = re.compile(r'^package/qemu/qemu.mk:.*Refusing to build qemu-user') + for line in lastlines: + if regexp.match(line): + return True + + return False + + if reject_results(): + return + + def get_failure_reason(): + # Output is a tuple (package, version), or None. + lastlines = decode_bytes(subprocess.Popen( + ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + stdout=subprocess.PIPE).communicate()[0]).splitlines() + + regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') + for line in lastlines: + m = regexp.search(line) + if m: + return m.group(1).rsplit('-', 1) + + # not found + return None + + def extract_end_log(resultfile): + """Save the last part of the build log, starting from the failed package""" - resultdir = os.path.join(idir, "output", "results") - os.mkdir(resultdir) + def extract_last_500_lines(): + subprocess.call(["tail -500 %s > %s" % \ + (os.path.join(outputdir, "logfile"), resultfile)], + shell=True) - ret = gen_config(**kwargs) + reason = get_failure_reason() + if not reason: + extract_last_500_lines() + else: + f = open(os.path.join(outputdir, "logfile"), 'r') + mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + mf.seek(0) + # Search for first action on the failed package + offset = mf.find(encode_str('>>> %s' % ' '.join(reason))) + if offset != -1: + with open(resultfile, "w") as endlog: + endlog.write(decode_bytes(mf[offset:])) + else: + # not found, use last 500 lines as fallback + extract_last_500_lines() + + mf.close() + f.close() + + extract_end_log(os.path.join(resultdir, "build-end.log")) + + def copy_config_log_files(): + """Recursively copy any config.log files from the failing package""" + + reason = get_failure_reason() + if not reason: + return + + srcroot = os.path.join(outputdir, "build", '-'.join(reason)) + destroot = os.path.join(resultdir, '-'.join(reason)) + config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', + 'CMakeOutput.log') + + for root, dirs, files in os.walk(srcroot): + dest = os.path.join(destroot, os.path.relpath(root, srcroot)) + + for fname in files: + if fname in config_files: + if not os.path.exists(dest): + os.makedirs(dest) + shutil.copy(os.path.join(root, fname), os.path.join(dest, fname)) + + copy_config_log_files() + + resultf = open(os.path.join(resultdir, "status"), "w+") + if result == 0: + resultf.write("OK") + elif result == -1: + resultf.write("NOK") + elif result == -2: + resultf.write("TIMEOUT") + resultf.close() + + with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: + submitterf.write(kwargs['submitter']) + + # Yes, shutil.make_archive() would be nice, but it doesn't exist + # in Python 2.6. + ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], + cwd=outputdir, stdout=log, stderr=log) if ret != 0: - log_write(kwargs['log'], "WARN: failed to generate configuration") - continue + log_write(log, "ERROR: could not make results tarball") + sys.exit(1) + + if kwargs['upload']: + # Submit results. Yes, Python has some HTTP libraries, but + # none of the ones that are part of the standard library can + # upload a file without writing dozens of lines of code. + ret = subprocess.call(["curl", "-u", + "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "-H", "Expect:", + "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadsubmit=1", + kwargs['http_url']], + stdout=log, stderr=log) + if ret != 0: + log_write(log, "INFO: results could not be submitted, %d" % ret) + else: + log_write(log, "INFO: results were submitted successfully") + else: + # No http login/password, keep tarballs locally + with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: + sha1 = hashlib.sha1(f.read()).hexdigest() + resultfilename = "instance-%d-%s.tar.bz2" % (kwargs['instance'], sha1) + os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) + log_write(log, "INFO: results saved as %s" % resultfilename) + + def run_instance(self, **kwargs): + """Main per-instance loop + + Prepare the build, generate a configuration, run the build, and submit the + results. + """ - # Check if the build test is supposed to be a reproducible test - outputdir = os.path.abspath(os.path.join(idir, "output")) - with open(os.path.join(outputdir, ".config"), "r") as fconf: - reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() - if reproducible: - ret = do_reproducible_build(**kwargs) + idir = "instance-%d" % kwargs['instance'] + + # If it doesn't exist, create the instance directory + if not os.path.exists(idir): + os.mkdir(idir) + + if kwargs['debug']: + kwargs['log'] = sys.stdout else: - ret = do_build(**kwargs) + kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") + log_write(kwargs['log'], "INFO: instance started") + + while True: + check_version() + + ret = self.prepare_build(**kwargs) + if ret != 0: + continue + + resultdir = os.path.join(idir, "output", "results") + os.mkdir(resultdir) + + ret = self.gen_config(**kwargs) + if ret != 0: + log_write(kwargs['log'], "WARN: failed to generate configuration") + continue + + # Check if the build test is supposed to be a reproducible test + outputdir = os.path.abspath(os.path.join(idir, "output")) + with open(os.path.join(outputdir, ".config"), "r") as fconf: + reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() + if reproducible: + ret = self.do_reproducible_build(**kwargs) + else: + ret = self.do_build(**kwargs) - send_results(ret, **kwargs) + self.send_results(ret, **kwargs) # args / config file merging inspired by: # https://github.com/docopt/docopt/blob/master/examples/config_file_example.py @@ -839,7 +840,8 @@ def main(): buildpid = multiprocessing.Array('i', int(args['--ninstances'])) processes = [] for i in range(0, int(args['--ninstances'])): - p = multiprocessing.Process(target=run_instance, kwargs=dict( + builder = Builder() + p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( instance = i, njobs = args['--njobs'], sysinfo = sysinfo, From patchwork Thu Aug 1 02:46:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140197 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="G4fAdGGS"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZSj4vdKz9sNF for ; Thu, 1 Aug 2019 12:47:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 462E785FCB; Thu, 1 Aug 2019 02:47:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MnPkpS_d68Z1; Thu, 1 Aug 2019 02:46:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 78D9585FB6; Thu, 1 Aug 2019 02:46:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id B48361BF59D for ; Thu, 1 Aug 2019 02:46:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id AF43C85FB0 for ; Thu, 1 Aug 2019 02:46:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O58hJm4KbObQ for ; Thu, 1 Aug 2019 02:46:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 7607585FAE for ; Thu, 1 Aug 2019 02:46:57 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id i18so33181169pgl.11 for ; Wed, 31 Jul 2019 19:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=UYtWAsuXso4Wp1+oJXG44fKqkDyv+IXv4cj4CkeG24I=; b=G4fAdGGSMMQZCdWwKcKlDMwjtG5GewxoQslTk3kijFU82zKSBQDhYPP73KGXCtW4Kp g3QMnMTFnpSx5eKJARggCMPXl5berl8c7Z5eqic4io8OzJHq5yvaOghKd5mkz22C4Vgk LSvs+i7FnEQCEWD49INcpUZdJuxNF19cLoHGtNMCV3D+IC1WbK1auZ+NdckByPfuXSNw 872XNOTPQQV+ho8eeLfcS6KGvseNU8cPgwTN0EERlJHwLoNs357MsAZSYEt8DNatVG8G Kj0lMlEBbWhcN7IRB1S5YeWt/cWVqDzhazSrAhAMu++s55WsdFPi3tHOJfpX0+Y/o9oO o3yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=UYtWAsuXso4Wp1+oJXG44fKqkDyv+IXv4cj4CkeG24I=; b=kozZvWDHZ1vJ9Ge3PWUMix1x/NjEikUoQocrvrT8DoF4KoZNFp/0mfSBbjZRSXFwsv u+Ub4AsPxJe531xvDw2uWwVl34xHfGAL257/DpfST/lLixGyt55CWsH5uKcTK2Mpe+Fn V4NYvlGvNsLlFitSLIQTRgkLXsBQ7Uolx97PM+CJb3+X/sued/Ngm/7epWlLUAEvo+HR yj9N11/MEyWyCrYgTinV6UAUZm0VgYN+8AfOGG72qiFxE4C6tdmY9jOgmM5RbDGVINH1 dVgUY2VmGV+3GTzOfyuL4OUiRjOnZIxiD2O8pqQ5y4H6GGb2MOc2X+tO6VHgvxRb5INQ 5OvA== X-Gm-Message-State: APjAAAUnsWJD/wEG1Z2pTzKcxK8UbUJzuW5xMLdWrMRqMQ0/m/oW4Jyu fQV0fDG9XyrAHzKJx84RL63qmMPiePI= X-Google-Smtp-Source: APXvYqymXfUJ2BFcQflKQ6e3nb9QaU5+qh98dUEn8vLcjzYFm5U+qw02mESPSsggGB4b/oOt+EmYWQ== X-Received: by 2002:a63:c013:: with SMTP id h19mr86679883pgg.108.1564627616574; Wed, 31 Jul 2019 19:46:56 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.46.54 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:46:56 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:15 +0530 Message-Id: <20190801024643.11024-3-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 02/30] autobuild-run: move instance variable from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" As discussed in the previous patch, these common variables are needed in many functions, and it'll be better to have them as part of the class. This will make the code cleaner and make it easier to introduce variability for reproducibility testing. Succeeding patches will move all variables from kwargs to the Builder constructor. Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 6bd6856..9946ddc 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -271,6 +271,9 @@ class SystemInfo: return not missing_requirements class Builder: + def __init__(self, instance): + self.instance = instance + def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -279,7 +282,7 @@ class Builder: code, and cleaning up remaining stuff from previous builds. """ - idir = "instance-%d" % kwargs['instance'] + idir = "instance-%d" % self.instance log = kwargs['log'] log_write(log, "INFO: preparing a new build") @@ -353,7 +356,7 @@ class Builder: def gen_config(self, **kwargs): """Generate a new random configuration.""" - idir = "instance-%d" % kwargs['instance'] + idir = "instance-%d" % self.instance log = kwargs['log'] outputdir = os.path.abspath(os.path.join(idir, "output")) srcdir = os.path.join(idir, "buildroot") @@ -403,7 +406,7 @@ class Builder: """ log = kwargs['log'] - idir = "instance-%d" % kwargs['instance'] + idir = "instance-%d" % self.instance outputdir = os.path.join(idir, "output") srcdir = os.path.join(idir, "buildroot") reproducible_results = os.path.join(outputdir, "results", "reproducible_results") @@ -435,7 +438,7 @@ class Builder: def do_build(self, **kwargs): """Run the build itself""" - idir = "instance-%d" % kwargs['instance'] + idir = "instance-%d" % self.instance log = kwargs['log'] nice = kwargs['nice'] @@ -466,9 +469,9 @@ class Builder: build_monitor.daemon = True build_monitor.start() - kwargs['buildpid'][kwargs['instance']] = sub.pid + kwargs['buildpid'][self.instance] = sub.pid ret = sub.wait() - kwargs['buildpid'][kwargs['instance']] = 0 + kwargs['buildpid'][self.instance] = 0 # If build failed, monitor thread would have exited at this point if monitor_thread_hung_build_flag.is_set(): @@ -500,7 +503,7 @@ class Builder: perform the actual build. """ - idir = "instance-%d" % kwargs['instance'] + idir = "instance-%d" % self.instance outputdir = os.path.abspath(os.path.join(idir, "output")) srcdir = os.path.join(idir, "buildroot") log = kwargs['log'] @@ -538,7 +541,7 @@ class Builder: are available) or stores them locally as tarballs. """ - idir = "instance-%d" % kwargs['instance'] + idir = "instance-%d" % self.instance log = kwargs['log'] outputdir = os.path.abspath(os.path.join(idir, "output")) @@ -690,7 +693,7 @@ class Builder: # No http login/password, keep tarballs locally with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: sha1 = hashlib.sha1(f.read()).hexdigest() - resultfilename = "instance-%d-%s.tar.bz2" % (kwargs['instance'], sha1) + resultfilename = "instance-%d-%s.tar.bz2" % (self.instance, sha1) os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) log_write(log, "INFO: results saved as %s" % resultfilename) @@ -701,7 +704,7 @@ class Builder: results. """ - idir = "instance-%d" % kwargs['instance'] + idir = "instance-%d" % self.instance # If it doesn't exist, create the instance directory if not os.path.exists(idir): @@ -840,9 +843,8 @@ def main(): buildpid = multiprocessing.Array('i', int(args['--ninstances'])) processes = [] for i in range(0, int(args['--ninstances'])): - builder = Builder() + builder = Builder(instance = i) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - instance = i, njobs = args['--njobs'], sysinfo = sysinfo, http_url = args['--http-url'], From patchwork Thu Aug 1 02:46:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140200 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Kvcfp1y7"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZSn20t9z9sMr for ; Thu, 1 Aug 2019 12:47:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 41F1185FD6; Thu, 1 Aug 2019 02:47:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tV91oGk3Uetq; Thu, 1 Aug 2019 02:47:01 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7C1CD85FE4; Thu, 1 Aug 2019 02:47:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id AEA841BF59D for ; Thu, 1 Aug 2019 02:47:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id AC6222266C for ; Thu, 1 Aug 2019 02:47:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id biRs2mFA7wuh for ; Thu, 1 Aug 2019 02:46:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by silver.osuosl.org (Postfix) with ESMTPS id DCBA122274 for ; Thu, 1 Aug 2019 02:46:59 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id m30so33053000pff.8 for ; Wed, 31 Jul 2019 19:46:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=prAlPHMoWWUegR1oxXvixZESE5Qm3UdiM9wp5LmpHZE=; b=Kvcfp1y7RNm5rBWRAuSTC/GTwHR+/DZmLsscUVhGrPF4dNBOTf8fs04WqZdq/BghaU JJg+xqZKxQ3qdKzUg8TIqbnJkIjSrk8mu1FrwAvVGdg0aJdhEngeMvJJ4w/5H/cp9l9f ys6PxbIj9yNbNT9YNtIQFMAFfzM6WMGaRy7khXKJ0RYKv+mVr2QgsYatgaD5ZqAHATB3 6V9UnozTNNXSAY4N5cTuoErpSr5KG0cb2HKg8VkhuOiSnOkWhEsarVCPQnkJNMgAhX4z 8WjtYEeAaCiReKk1djSxbz7UjWWPRt10QfCWrt7chzf+wMlMc4KitMW1EWKRfhvS20Rs cdoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=prAlPHMoWWUegR1oxXvixZESE5Qm3UdiM9wp5LmpHZE=; b=buVvnbUod39sNwouK/HfRMQAhQxlYTdJGLto/BjRZqci6ivuN38VqZv+zbsvLPnZhm rrh3dHns2m0G+3DV5L2Hpy57dfaG2tNOG62LzAoXcHMU6EU5M8cPWgJsRYQ3Wz+nsALU bY0jOoDSklKCZm8k6WS3xldgGxpdkXJlzMzX4g2bTU3HrhAgCBuDiDD3Fak9+T/TLp0w nYbxezCP82PgW1nPTjpqA23np7EhxWJToKYUjXy0c6DzYWc+Txty4LJ6lohWedIFKdLY 7ketOhaBbqCbTIdhfbitUdhZPR5EjoV1sxtwmyLjOpqTbuxr2zUDWSZY4PyekJAPOQBB 40sA== X-Gm-Message-State: APjAAAUiceM6PdmUuG0/XxdS7N3atD4JdKA2D/A1GW56+RnBvMtAeWri 3HvKDh+UHQQ7Zkb9TeVC7qzsrHBDOR0= X-Google-Smtp-Source: APXvYqyMHbuu+pKeN68k0+2/B0tekM8sO/CANUjVRZeMeTHzJywIuWCbis9TVnRmxzlGrKOQ4xV4FA== X-Received: by 2002:a17:90a:b394:: with SMTP id e20mr6022536pjr.76.1564627618962; Wed, 31 Jul 2019 19:46:58 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.46.56 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:46:58 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:16 +0530 Message-Id: <20190801024643.11024-4-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 03/30] autobuild-run: move njobs from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 9946ddc..4d730ce 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -271,8 +271,9 @@ class SystemInfo: return not missing_requirements class Builder: - def __init__(self, instance): + def __init__(self, instance, njobs): self.instance = instance + self.njobs = njobs def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -455,7 +456,7 @@ class Builder: cmd = ["nice", "-n", str(nice), "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, - "BR2_JLEVEL=%s" % kwargs['njobs']] \ + "BR2_JLEVEL=%s" % self.njobs] \ + kwargs['make_opts'].split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) @@ -843,9 +844,10 @@ def main(): buildpid = multiprocessing.Array('i', int(args['--ninstances'])) processes = [] for i in range(0, int(args['--ninstances'])): - builder = Builder(instance = i) + builder = Builder( + instance = i, + njobs = args['--njobs']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - njobs = args['--njobs'], sysinfo = sysinfo, http_url = args['--http-url'], http_login = args['--http-login'], From patchwork Thu Aug 1 02:46:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140201 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HT8/oU7e"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZSr2PkCz9sMr for ; Thu, 1 Aug 2019 12:47:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id D152187E6C; Thu, 1 Aug 2019 02:47:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JUVSgJIXaSA6; Thu, 1 Aug 2019 02:47:03 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id A17F987E55; Thu, 1 Aug 2019 02:47:03 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 8DA7B1BF59D for ; Thu, 1 Aug 2019 02:47:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 87B5B87E23 for ; Thu, 1 Aug 2019 02:47:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9s0sPkson16D for ; Thu, 1 Aug 2019 02:47:02 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by hemlock.osuosl.org (Postfix) with ESMTPS id 1E0D487E55 for ; Thu, 1 Aug 2019 02:47:02 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id w10so33186922pgj.7 for ; Wed, 31 Jul 2019 19:47:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Sje947sqdPy118AFUppRM1f9qpvrxudkNpt1r5kPj3E=; b=HT8/oU7ebHSEbcXqat4nVZUrIgv2QKFEY9L93HcfwId6BeH0+CTILFh7lxyoC8EAsV +b96B2xPipT5TyACUZ4kTnLLMlO4ydq4OQ+TsjLyEmMGf0vvqOJeoQnnhiP63gElTqbC 2dAcYZU2MKDZ0zZyDog0NVYSN//gXaNP07oKB9OzMAWmfwz00ZKiDrmGNPWiEl5hmUoK HuHSuBm85WT+DJI5LQXnGPJSekkZAvk76WiRkvCCWkRi2XKUSrAO4snRVDAVF5KbIQ+s 09QOZpzVbJYIMxn6+ERtkNjAdpQ3uPMzKX66I8wubLSBTceh9vDDEk59vkMVD8wz1HQR J+dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Sje947sqdPy118AFUppRM1f9qpvrxudkNpt1r5kPj3E=; b=M8qdDqD60RZ4xfCGbxQOarp3TJEladhPNAY/FJCqFVCUfKEzNSPisPZi2IXidkiDSs JmzJycm7q/RIgNeJo7T3B959Fo5RdeViLTrpMW16HIIKNi4TgmpkwLthAgzeeInF7L7E m98vy0yC8VrGu0jMg/yzbWzQTE5auBVK8RgyphrlJ0jquVP+2UHNdWhJWp9k2yDcb3vB mZLfuQ2WHQojjHH2VbV2C28cOYTc9Z4bY0cpMtbIuV3ZmuAR/hbHzNCOvKTaYR/dlVpL tIGMRdbm+MXqIWnqU8YnuTDxwdjn+jyXrrKtskweyTTQrY4x/0XhhTTRaYem/0//CDcv eQ/w== X-Gm-Message-State: APjAAAUpw2ECQqK8qq+vnacy/Cfe0uWvlptpA1yvXv6a68PsjzBD3FL3 S8E9w75l6hV2EJyKE5zU8IrBHj1sf+E= X-Google-Smtp-Source: APXvYqwe+3Y/ANNn0uw/G2yhiFML5KAFUWu3eXwKJoKLvKGREa4+tkqp5jUYkkOKHO23eMTKub9zwQ== X-Received: by 2002:a63:8a49:: with SMTP id y70mr22051465pgd.271.1564627621269; Wed, 31 Jul 2019 19:47:01 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.46.59 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:00 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:17 +0530 Message-Id: <20190801024643.11024-5-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 04/30] autobuild-run: move sysinfo from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 4d730ce..ca2dd83 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -271,9 +271,10 @@ class SystemInfo: return not missing_requirements class Builder: - def __init__(self, instance, njobs): + def __init__(self, instance, njobs, sysinfo): self.instance = instance self.njobs = njobs + self.sysinfo = sysinfo def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -416,7 +417,7 @@ class Builder: build_2_image = os.path.join(outputdir, "images", "rootfs.tar") with open(reproducible_results, 'w') as diff: - if kwargs['sysinfo'].has("diffoscope"): + if self.sysinfo.has("diffoscope"): # Prefix to point diffoscope towards cross-tools prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string @@ -846,9 +847,9 @@ def main(): for i in range(0, int(args['--ninstances'])): builder = Builder( instance = i, - njobs = args['--njobs']) + njobs = args['--njobs'], + sysinfo = sysinfo) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - sysinfo = sysinfo, http_url = args['--http-url'], http_login = args['--http-login'], http_password = args['--http-password'], From patchwork Thu Aug 1 02:46:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140203 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="sWX4KICO"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZSx47TKz9sMr for ; Thu, 1 Aug 2019 12:47:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 42A7986BAD; Thu, 1 Aug 2019 02:47:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XOCfixi-St6s; Thu, 1 Aug 2019 02:47:06 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id A7BAB86B92; Thu, 1 Aug 2019 02:47:06 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id EF0871BF59D for ; Thu, 1 Aug 2019 02:47:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id EC7462266C for ; Thu, 1 Aug 2019 02:47:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZtPFo5DBHWX1 for ; Thu, 1 Aug 2019 02:47:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by silver.osuosl.org (Postfix) with ESMTPS id 6EADF22274 for ; Thu, 1 Aug 2019 02:47:04 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id y15so33053216pfn.5 for ; Wed, 31 Jul 2019 19:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P9sJ6e5bXM0uINj4uQIubtTSWTPFMDjlaGi829MtpzI=; b=sWX4KICOEtAWzf3ZOQOBOy3aeCGD04yjg2auNXuRM6AVxzEfvp0NRAJmr+K4CKZHmj TwHGg8gADkJ9CkbN7rxqaYg0ObnFZp05F4DwAOiQXQCFy8afYGEMq3wYN8xwv20a3BsD X60CO7K0EoMJDCc7vT7eCU+JuZG8NHiqQCLk6+94xBBi5cxF8LMesf3kMOmpsHrtxpLS frIUwnwZyx6qjPrY9a4CvG5EYbM/daigkZLGQV4ZAE8Fnoupg0JVAnxl7fWLZHLjXwEc pBCxsQrqCEtDAAz3xtz82yL8HGHoBg7xvF40sBNSldne/F08WOQqLRSmoNyqoryFqstR bumw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P9sJ6e5bXM0uINj4uQIubtTSWTPFMDjlaGi829MtpzI=; b=QX46m/ABOG72B9Tqikuo+beBP75ovhTHDujmqEb9TD6XMSFtuwSwCkzlS7Xhtvlrmt O8dFtYxxkkBGUBwqBFV0EZrDpZxSLyHIQlppFIrli6tAxBT7Xp9L6jxNWT05OespebBd nYiPXYR3hfIVtHmnxSi/xFMrhS1Irh7k3SXbRJyZM0FiQF8HQdSwJ0BhhCiBicjGLrHx MaCoGhh4qTIh3fI6LqJmG8MQwDFtGbdh+9MKR8QQy1Y4UNU72dxw79VdlrxcZIb15fr4 1gHhmTg1KTBd8FmmrpSnqAh1HBCT9LzqB1REFG/joq/USxSgm01p4YNYXnTtTNw+RYrz lxWw== X-Gm-Message-State: APjAAAXaILj9YlPAVzhPl/9P+L9femCnTWKxPRaDkHm8MtpcallDji5/ /eB9so9ZwZuGL+bre/hfqLvyLbKItpo= X-Google-Smtp-Source: APXvYqycxjfeNb+TCmnlp8dZO92z/InZYAcgPOOscptwuQFMVznQE/yIfY5A5MGDpXepNLV9xXrN5A== X-Received: by 2002:a63:61cf:: with SMTP id v198mr399889pgb.217.1564627623685; Wed, 31 Jul 2019 19:47:03 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.01 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:03 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:18 +0530 Message-Id: <20190801024643.11024-6-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 05/30] autobuild-run: move http variables from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor arguments - Fix whitespace issues --- scripts/autobuild-run | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index ca2dd83..9d16ce5 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -271,10 +271,14 @@ class SystemInfo: return not missing_requirements class Builder: - def __init__(self, instance, njobs, sysinfo): + def __init__(self, instance, njobs, sysinfo, + http_url, http_login, http_password): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo + self.http_url = http_url + self.http_login = http_login + self.http_password = http_password def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -681,11 +685,11 @@ class Builder: # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. ret = subprocess.call(["curl", "-u", - "%s:%s" % (kwargs['http_login'], kwargs['http_password']), + "%s:%s" % (self.http_login, self.http_password), "-H", "Expect:", "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", - kwargs['http_url']], + self.http_url], stdout=log, stderr=log) if ret != 0: log_write(log, "INFO: results could not be submitted, %d" % ret) @@ -848,11 +852,11 @@ def main(): builder = Builder( instance = i, njobs = args['--njobs'], - sysinfo = sysinfo) + sysinfo = sysinfo, + http_url = args['--http-url'], + http_login = args['--http-login'], + http_password = args['--http-password']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - http_url = args['--http-url'], - http_login = args['--http-login'], - http_password = args['--http-password'], submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), From patchwork Thu Aug 1 02:46:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140202 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="LwUX7589"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZSt5V5bz9sMr for ; Thu, 1 Aug 2019 12:47:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1DDA585FAA; Thu, 1 Aug 2019 02:47:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wKSKm6iEzDjb; Thu, 1 Aug 2019 02:47:08 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9129C85FE4; Thu, 1 Aug 2019 02:47:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 8FFC11BF59D for ; Thu, 1 Aug 2019 02:47:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8D2262266C for ; Thu, 1 Aug 2019 02:47:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1jEZfCRtyMk1 for ; Thu, 1 Aug 2019 02:47:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by silver.osuosl.org (Postfix) with ESMTPS id 0086022274 for ; Thu, 1 Aug 2019 02:47:06 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id l21so33177097pgm.3 for ; Wed, 31 Jul 2019 19:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l+UJWpbEhZ0RPpaZ6Mt7yq58a9kGFHhrKIbZi1l5Blc=; b=LwUX7589KsMndONdSM1yE1Zl4OAW++BCEERFOL8Q4+GXy/0Vfl4cEgZ0fYbtudAXKC EsDcKyujxBJ8TwzF6Mhg3wVAKBFwaooFB4BT47jRETh1VBTndW/8mblI72YKuGDB3cFe fegNmvE6opJYyeCtPFBpaUZ8eWNDtIJhFPBEIgJ8Kn0aIb8nqKiyd6SYWXabHYPgvnm1 k+oumTBpmH7vOsmXdqkVFy9rz1ugMmMPYA6KgSFXU8myfs/7FEOiG7a6+Ps6MTVtOh0u DTIoZpwEU5uj0UsHNtCMQQDjTmPQbkUHZagGWW+aqqh5p1VWLYIBiKY6YgvMPjaPvDWw +etg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l+UJWpbEhZ0RPpaZ6Mt7yq58a9kGFHhrKIbZi1l5Blc=; b=RzmazTKwxW9Nxzw7nrc73CQzkVHTnKEMM2kga7ZXXy4D8tfTpY+L20Vdv3QYx6IqZs pTwj+Qy4380SZTb83xYaxbj4mgvJN5MRKYpR4tSXLjsVZSMIVOCzmmlA5dWgmrCEofwh N9KMtSQLSr4V94q3tnZvKx5BT1epvA0Wvv/co82AhZ4eGvmAA+gOcAZnFNQ+scti0E+6 XjSJbOVkF/B29PM3LGCQ/V0fkGCFPDsuC0tngWfsZ6MJs7qSxApoY3ozF726jSUE1FoG NwXb0DCRDLyJtb6ua/217pFDIZWIB+HNIi44QihXLzJQcELcrrXq78hY4L4ADkPG9+aI K6BA== X-Gm-Message-State: APjAAAV2xlZiBgv0+mPiZV7DRhAlT83jPQ9SqfjT1b1FnG0804JFwC2o SYeKlKCal7Zn/Hts1cqMMhyfjAZjqZQ= X-Google-Smtp-Source: APXvYqx7f8mJYw7tuwv5lFbfRpG1ZGydc3Tr+R4yTUl1dPp00m0a4gLZO0NMxUvNE8kmQBazoOnxHg== X-Received: by 2002:a65:5cca:: with SMTP id b10mr119543749pgt.365.1564627626080; Wed, 31 Jul 2019 19:47:06 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.03 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:05 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:19 +0530 Message-Id: <20190801024643.11024-7-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 06/30] autobuild-run: move submitter from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 9d16ce5..426078c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -272,13 +272,15 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, - http_url, http_login, http_password): + http_url, http_login, http_password, + submitter): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo self.http_url = http_url self.http_login = http_login self.http_password = http_password + self.submitter = submitter def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -670,7 +672,7 @@ class Builder: resultf.close() with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: - submitterf.write(kwargs['submitter']) + submitterf.write(self.submitter) # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. @@ -855,9 +857,9 @@ def main(): sysinfo = sysinfo, http_url = args['--http-url'], http_login = args['--http-login'], - http_password = args['--http-password']) + http_password = args['--http-password'], + submitter = args['--submitter']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], From patchwork Thu Aug 1 02:46:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140205 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WFTeTaSw"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZT243Fnz9sNf for ; Thu, 1 Aug 2019 12:47:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6ABCF86B78; Thu, 1 Aug 2019 02:47:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ixN2mPF9e90t; Thu, 1 Aug 2019 02:47:11 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 859AC86BA4; Thu, 1 Aug 2019 02:47:11 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 7ED901BF59D for ; Thu, 1 Aug 2019 02:47:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 7C34B85FC1 for ; Thu, 1 Aug 2019 02:47:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Nf3CffoAlA6M for ; Thu, 1 Aug 2019 02:47:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 60CF986004 for ; Thu, 1 Aug 2019 02:47:09 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id n190so1741489pgn.0 for ; Wed, 31 Jul 2019 19:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9hHjyKhUYI9YMqEyOMdWZ8nDZvWj3hjtKrBhVCUUuGE=; b=WFTeTaSwjk+n6gKhEFhfXF/w1yzs++f3kbkBhjtM5+WmFpe4Y0IqhpscIFVth+Qh5n uROan97AnbGldCBlnb54S+OGNj8JEOPGHhRt2lCnyKAuymBU6u+aE171PdK59+CyYUiW ehcnGE8L/Nc1l2tUCY2b5QoFyzmC4xbcTnrLzfbQ2lmuflKTTEwdgkraGddWvfjfQ595 lMo+s12GMuhMQRVBoBkzsKpYQXuKXOUrsFWTLbeZX1bSSh7weN1268nyL0m3or309UqA AkugrpBYUBRCXCDoQDWzshlbqBBXHMTPettLuzryVzljCGxI/+TQXj+A/oBzyO/lICoe WGxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9hHjyKhUYI9YMqEyOMdWZ8nDZvWj3hjtKrBhVCUUuGE=; b=tPt1iIRQRtCgW0ce/Ns0FnnwNGWvRVmLSaMh+J2yZSh61tRWVXoeyH4pMGIzQf4+kK K7LN8wtqk5KJWoX16wa5rcYK5UZJpKco0K0nXXaVZh/7p7nGLSp6OR9lLQ6MBVWBNbaq T7D12nUDMu0uOlWeTZ8SZJBkYsvVd/dIHqhuB6CMA7yrG5+IXSGRuu5okHEmnJ3H45Go ECohjjMcMd8dcP/cJoLUQuaOqAmpjRsgI9qO6n1Pf+Xoz+yFhn+rhxqcHH+/j1Hj6Z0n b3oNhXMuKZU6FO/SL/3G4Q8qh+qUKQgchRJnEK2+iHHLMcgmgDW6X7sBUWL9suZVP7S9 c7Zw== X-Gm-Message-State: APjAAAWmIQ+U4uI6vULcztnqw0F/HDkjP8R+1s/AuOHNZTHIJoOzzv0w PlmtwVRMCgM78AOn2eNyPb89IRqVGZI= X-Google-Smtp-Source: APXvYqxV/48r3BQZmyvWZdi+g1QaQWO48sdyLxXl5aocKWD/CuweoHmKPjSivgarentEudT4Eg/H1Q== X-Received: by 2002:a63:ea50:: with SMTP id l16mr118694844pgk.160.1564627628519; Wed, 31 Jul 2019 19:47:08 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.06 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:08 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:20 +0530 Message-Id: <20190801024643.11024-8-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 07/30] autobuild-run: move make_opts from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - New patch, split accidental squash of niceness move and make_opts move --- scripts/autobuild-run | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 426078c..c2d137c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,7 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter): + submitter, make_opts): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -281,6 +281,7 @@ class Builder: self.http_login = http_login self.http_password = http_password self.submitter = submitter + self.make_opts = make_opts def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -464,7 +465,7 @@ class Builder: "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % self.njobs] \ - + kwargs['make_opts'].split() + + self.make_opts.split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) # Setup hung build monitoring thread @@ -496,7 +497,7 @@ class Builder: cmd = ["make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "legal-info"] \ - + kwargs['make_opts'].split() + + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) if ret != 0: log_write(log, "INFO: build failed during legal-info") @@ -858,9 +859,9 @@ def main(): http_url = args['--http-url'], http_login = args['--http-login'], http_password = args['--http-password'], - submitter = args['--submitter']) + submitter = args['--submitter'], + make_opts = (args['--make-opts'] or '')) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], From patchwork Thu Aug 1 02:46:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140206 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Y8L1o/Wb"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZT32Lf8z9sMr for ; Thu, 1 Aug 2019 12:47:18 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id E2CDA22699; Thu, 1 Aug 2019 02:47:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ib-UufGAWrtN; Thu, 1 Aug 2019 02:47:14 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id E6E7D2266C; Thu, 1 Aug 2019 02:47:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 4B54B1BF59D for ; Thu, 1 Aug 2019 02:47:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 463A587E4D for ; Thu, 1 Aug 2019 02:47:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e1-NDC2TO4jj for ; Thu, 1 Aug 2019 02:47:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by hemlock.osuosl.org (Postfix) with ESMTPS id D2EDB87E58 for ; Thu, 1 Aug 2019 02:47:11 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id m30so33053256pff.8 for ; Wed, 31 Jul 2019 19:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TDaMfyGks+XpP3BmK5qQOdRgXsusVv8Th6xg2XSN6jk=; b=Y8L1o/Wb/AjzKkfEyAyJAk/EPMEXnQRvN0Go7rom9TGWxMi9uIXGG/hCQRzELeb//N 1wETOJDBx+ET6O677ZZpkfYM41Z+ciq3FfN6OB2T38OT+6OukLT5QWgY68BP2F5EkyHB ZGkBH/U1yWCYvwznmZF47Jij+3Ka3hSJc0eLlIz0f8yStxsDrVQj582Hb2uecMlVEq9i jJ+fXp8mKkv7qR40TZJP0VdrrJP0ptwpoYjYSV08WSWphYQA8cl5CFWeQL4MPvJm0XzR 3olwbgDe8ptJmy1PWQfRk3KJMD5nwGs1pxXcflNooZUk5jkqxCYNhUKNYPjCJpqcw0Iv y2NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TDaMfyGks+XpP3BmK5qQOdRgXsusVv8Th6xg2XSN6jk=; b=C6jo8lN5toYs58UsINpbodiWU7bUh+4n8PPIssj1bj7r2mDa7kXRzaNC5q4tWg1/R4 CpA/E2lJ/90tSYOMskDMncXv7KKN5A2CarAg+1CFYxoMsPo/Y2mIwFwFsQ5injhhcItf wTTwMbH7Ry6wwNZMa/RLCdZOa/OfM4a7fbsu9cSAVZgoSGH5WWAMxLDGf3QYrtbZIQ/z EJFPhJWg7fVpeerla9+ZDZlR/Uy2jRPHAXvYmbdQFE2keqpWxTssJx3X9UCY4XoqHIh7 XK9t9himQggHscDASa/002v60BuLFklftJwUsN2Vpu/1EbJshgGj+dKMqCehnRdPKMpG nS6Q== X-Gm-Message-State: APjAAAVLo9wqwGg6m2jwLpUxx+mZnWTmhlhLfU4JmMhmCCinKMj7Qt+h p4vuTnQlH4tydpzrf+zyLc3jAyZ2JL8= X-Google-Smtp-Source: APXvYqwVCds23X9JQ7ykTF8o45823y5EZJSIXbKW3dZLx8OlfB4/ylOTY7xO6+lJ4O/TIoVfGQ4F5w== X-Received: by 2002:a63:7d05:: with SMTP id y5mr118081952pgc.425.1564627630985; Wed, 31 Jul 2019 19:47:10 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.08 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:10 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:21 +0530 Message-Id: <20190801024643.11024-9-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 08/30] autobuild-run: move niceness from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument - Fix accidentally moving make_opts in the same patch --- scripts/autobuild-run | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index c2d137c..87d0bd8 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,7 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter, make_opts): + submitter, make_opts, nice): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -282,6 +282,7 @@ class Builder: self.http_password = http_password self.submitter = submitter self.make_opts = make_opts + self.nice = nice def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -449,7 +450,6 @@ class Builder: idir = "instance-%d" % self.instance log = kwargs['log'] - nice = kwargs['nice'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the @@ -461,7 +461,7 @@ class Builder: f = open(os.path.join(outputdir, "logfile"), "w+") log_write(log, "INFO: build started") - cmd = ["nice", "-n", str(nice), + cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % self.njobs] \ @@ -860,9 +860,9 @@ def main(): http_login = args['--http-login'], http_password = args['--http-password'], submitter = args['--submitter'], - make_opts = (args['--make-opts'] or '')) + make_opts = (args['--make-opts'] or ''), + nice = (args['--nice'] or 0)) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], upload = upload, From patchwork Thu Aug 1 02:46:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140204 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jl3DjQs6"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZT16tX2z9sNF for ; Thu, 1 Aug 2019 12:47:17 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 814AC86031; Thu, 1 Aug 2019 02:47:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p7OAM4ZcvvzF; Thu, 1 Aug 2019 02:47:16 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id EC2C185FB0; Thu, 1 Aug 2019 02:47:15 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 9B8751BF59D for ; Thu, 1 Aug 2019 02:47:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 98A4C85FE4 for ; Thu, 1 Aug 2019 02:47:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id t3VmTwmqS6Sc for ; Thu, 1 Aug 2019 02:47:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 2DC4C85FB0 for ; Thu, 1 Aug 2019 02:47:14 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id w10so33187209pgj.7 for ; Wed, 31 Jul 2019 19:47:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5EftJHixxuhoHpi4E0UF9jhCseR78k9fvs4xIvsHehc=; b=jl3DjQs64WI3h/sPeGx34yqlyBGAGlj8lMhuaApGiqNAENezsKbHD//b46E8H6Pg8o O84EcWzQaerdnL0NNFgqG3MxwlRBjhMPYoxEZdlxVIz+GYxcaCQvi/He66+SNSo+g+3U YrBw9Wr0UdrJbEv6H9SyI+9OBWfrUwQoHb489PkEN/rkHbsoQvQegOYJi/ydN3YTqyiy JqMvkNzhuTeD9fPwGw0fCizKzCG2v+PlnR8jUweqfiL7GciLrRBUV9vIgHtvgHMTFBe+ 4YUmrXewP7j/rCvgTGEPT5mrUoI+se21F4VbyjGTlkyhLRvzdzuOSrACXab5LkwWFw6g +Ddw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5EftJHixxuhoHpi4E0UF9jhCseR78k9fvs4xIvsHehc=; b=ENTCDMYFrp91KhHwVCya0eBIPX0SI2/8KHy3PxIcwc6cORevH2e8cjayzfHVx4MBX8 tTdR4Cl0hj5qsA8fW459YrGp9OZBrPokUsh0uAELtD64DtsYaSNG5vyxnVbvVGkDjJtG 6FPdABPAHZyOt/fzDt/hwK7GBWI9dLO/xRK/gMyBteNykzrLHzA9ZHQA3o49QQA2gXHG 6e6gSpTZ1PLYubrZlkUleLrkI2nZ1lcMfCLBUKd+Yn/y5sKgpv1zylI5lvlshPkjYSSn yUpGh+woYD2tIvCsLrIx84GZT60zS5cZbuLDRAw6hGp1TW8x07G1tCncziATquUQJN/t RELQ== X-Gm-Message-State: APjAAAUzzZtKecoWc5conjIjWcd/5tXmqqNOfFPhRzCaaxV3yzIONbkE 8UrlK97gU/YMO63QhKrGDODddYkuUpY= X-Google-Smtp-Source: APXvYqxgbkL41RZS+pnnL2tCC3m5ObF5JeFkruRW3XW/KoRnF48SBIv5O38ddjPseCCK8+uGuSYahQ== X-Received: by 2002:a62:303:: with SMTP id 3mr50397313pfd.118.1564627633438; Wed, 31 Jul 2019 19:47:13 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.11 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:13 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:22 +0530 Message-Id: <20190801024643.11024-10-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 09/30] autobuild-run: move toolchains_csv from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 87d0bd8..3bd10f7 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,7 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter, make_opts, nice): + submitter, make_opts, nice, toolchains_csv): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -283,6 +283,7 @@ class Builder: self.submitter = submitter self.make_opts = make_opts self.nice = nice + self.toolchains_csv = toolchains_csv def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -381,7 +382,7 @@ class Builder: args = [os.path.join(srcdir, "utils/genrandconfig"), "-o", outputdir, "-b", srcdir] - toolchains_csv = kwargs['toolchains_csv'] + toolchains_csv = self.toolchains_csv if toolchains_csv: if not os.path.isabs(toolchains_csv): toolchains_csv = os.path.join(srcdir, toolchains_csv) @@ -861,9 +862,9 @@ def main(): http_password = args['--http-password'], submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), - nice = (args['--nice'] or 0)) + nice = (args['--nice'] or 0), + toolchains_csv = args['--toolchains-csv']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], upload = upload, buildpid = buildpid, From patchwork Thu Aug 1 02:46:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140207 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WuAngY14"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZT60MrPz9sMr for ; Thu, 1 Aug 2019 12:47:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5923A85FE4; Thu, 1 Aug 2019 02:47:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YkC-0-wJO2jv; Thu, 1 Aug 2019 02:47:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id CBE8285FF3; Thu, 1 Aug 2019 02:47:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 2F4C01BF59D for ; Thu, 1 Aug 2019 02:47:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2CB2E86049 for ; Thu, 1 Aug 2019 02:47:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v1UZN5KKOuwF for ; Thu, 1 Aug 2019 02:47:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 7B6E885FAE for ; Thu, 1 Aug 2019 02:47:16 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id l21so33177307pgm.3 for ; Wed, 31 Jul 2019 19:47:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5o6PCskCymzBNT0CypYL0tOvHLU++nQ7F15CNgcXojs=; b=WuAngY14ujvmI+xkP/n+h0EL2Cf4Xi9L2R9TOx3cpbqAkAlotZAqrzKPhUyvgdBTPF ZsGEAwMyDUklWoeh2Ug4YgwIgR5GNH/y6rd6RodcGxi9q/GrgPTWxzQtu1fb03AvcyyN HDXImOsNTnJCbWq9tO9W1feme+HoW67UnqB/8dln+8aEqzF2azzeUuU5Ym2u/SfwAtEV UM/4pos2sLxytqj9E3Q2lygurce+d+gPtHcVkVh2BR7wIl7sLASVFaYzIzSTnVeJn0dQ eBtcRAYgMlCf0cjTyMMNj2jOvpGk3dUguBTPYhak56R1uIgIBLbU0vMTGFfWn7rjEVKo 2OTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5o6PCskCymzBNT0CypYL0tOvHLU++nQ7F15CNgcXojs=; b=gnJaqmpzID1d2hyDF5UGkKh9rckIVsqGNYJFTGlmQY0YNN9duqM43nq+cyX2JH+9qF ZMS0KPLMqcv0tDfl8lpEgl0vzYvE9GORmKtxOTsYy8438Gj7om7p6tO7EQGC4eH99OGG rpkNAvCEL1Fjmko1xZGxiFzjwZQ+K/BbjF8qhlWKp5DgtlH+wY5xbEo5P2WimLNDcEMT nFx/epawXrSDt/lFyB5vPaykpGLIB28Yn0qAiRe3QGzk0BhpGggVi+oG+G6GDLK20oqW lMhkL5Es2zovK3FO2TzQ5lGyEoZ3LGL33ehPtZSgGUyqcX9Alb8nhPciQmDlp6DVZFZk tdgQ== X-Gm-Message-State: APjAAAUY7icoEDuyFdbcavQrDS6+0pOhx28tkT2Y6a3+x7H9JVOZ5MQh bMAGkdNLViRoT0D+EjH8MtdAXIESFDI= X-Google-Smtp-Source: APXvYqzfE3CCsuMYjhpkSWKH0LggG1JSw9OxOk2w+HZXSYnPLBE9y6YFZ7FYrtHt1c5TzQclT8jNjQ== X-Received: by 2002:a17:90a:ba93:: with SMTP id t19mr5882098pjr.139.1564627635745; Wed, 31 Jul 2019 19:47:15 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.13 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:15 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:23 +0530 Message-Id: <20190801024643.11024-11-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 10/30] autobuild-run: move repo from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 3bd10f7..e64f6db 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -273,7 +273,8 @@ class SystemInfo: class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, - submitter, make_opts, nice, toolchains_csv): + submitter, make_opts, nice, toolchains_csv, + repo): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -284,6 +285,7 @@ class Builder: self.make_opts = make_opts self.nice = nice self.toolchains_csv = toolchains_csv + self.repo = repo def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -336,7 +338,7 @@ class Builder: # didn't exist already. srcdir = os.path.join(idir, "buildroot") if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", kwargs['repo'], srcdir], + ret = subprocess.call(["git", "clone", self.repo, srcdir], stdout=log, stderr=log) if ret != 0: log_write(log, "ERROR: could not clone Buildroot sources") @@ -863,9 +865,9 @@ def main(): submitter = args['--submitter'], make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), - toolchains_csv = args['--toolchains-csv']) + toolchains_csv = args['--toolchains-csv'], + repo = args['--repo']) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - repo = args['--repo'], upload = upload, buildpid = buildpid, debug = args['--debug'] From patchwork Thu Aug 1 02:46:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140208 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="O1KBkNJk"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZT901gDz9sMr for ; Thu, 1 Aug 2019 12:47:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 7AA65226D7; Thu, 1 Aug 2019 02:47:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K+UYVVXuFy-q; Thu, 1 Aug 2019 02:47:21 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id B7CB2226A9; Thu, 1 Aug 2019 02:47:20 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 56FCD1BF59D for ; Thu, 1 Aug 2019 02:47:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 50CAA85FE4 for ; Thu, 1 Aug 2019 02:47:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JEf2M1-GQTVu for ; Thu, 1 Aug 2019 02:47:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id CFBED85FF3 for ; Thu, 1 Aug 2019 02:47:18 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id c2so31387213plz.13 for ; Wed, 31 Jul 2019 19:47:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GZxQocoqcj49NL44N8bS+rVXaJUZSla4XKb59boib3w=; b=O1KBkNJk7AEotDnCSIwwvafJVBYQlu9IsTQhESDaeYVnQpWOjJaqqpSi9fTnIVioyV x6Om9WtemAlh96OjXfrA4iQdGCCfRPxiUFLtPrLBeF2/QuDR5EGEc1q5pm1juiPuL0e9 8h4/Vq9+Z326jjZxwLZeF6agGgsQ6x3FZ3lAZZBfWdNeTK6PTkHBxBfIejUac0KOEzDT p0E/Iy1FHNXKOOT9vewepQAoAnkSxJ9UVsWwXMEHbnAE4UA72Nay7qHPKaB8KX6DHkba sv8rH3LZdrMwiNnxxIx24yZtkHnFHrF3ymywBSd6M+E2WLIPFuomH2dPClYlhhdAHtLV 7eyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GZxQocoqcj49NL44N8bS+rVXaJUZSla4XKb59boib3w=; b=T2qTEBJEimONUSWQriw15XyQRzXfm57k0r7h5AQDLGi6oLBP/9FcCe4L6EL+kOB6Wz /CCmiVvoDc3MtPH8wd8hrxx6bKroMyGSZGGmYnzyrDEG2gOLS7+KPgD086K9dcJnwdjl 9IGxMJ4XZ024dx6+jjd/zB/H7xjJt1JYgnaRatwJnwSFaQxMuvjcRVTTT5CtUdyUwDFW EMQkNr+vKg44of6p6iJB/N/NmDkhHs080GJk20aILdY6z4+WvL+AfkVsKS0K2penwVmq QO1UpwzPnOhebxhI1dkgfNfWTG9GeqevNkPOx9zqYV6d95fGRzNWZooQjlz6sZk6xSc/ ++wg== X-Gm-Message-State: APjAAAWdEcS4JyUN0iqDkZMWU5fh3JHLcerKZ1sjjnAlPjRNXj9ac7Eh w4gU7CZNDIo2mEnWvuwkDinX+R4jlkA= X-Google-Smtp-Source: APXvYqzb34L0xearDCgulDXh7O6ms/T1mQ57vHaG6lcrFTbIlStwsnquwqJVVzAtvBY6OppM2SAwJw== X-Received: by 2002:a17:902:2ea2:: with SMTP id r31mr19648692plb.200.1564627638030; Wed, 31 Jul 2019 19:47:18 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.15 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:17 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:24 +0530 Message-Id: <20190801024643.11024-12-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 11/30] autobuild-run: move upload variable from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index e64f6db..e217117 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -274,7 +274,7 @@ class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, submitter, make_opts, nice, toolchains_csv, - repo): + repo, upload): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -286,6 +286,7 @@ class Builder: self.nice = nice self.toolchains_csv = toolchains_csv self.repo = repo + self.upload = upload def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -686,7 +687,7 @@ class Builder: log_write(log, "ERROR: could not make results tarball") sys.exit(1) - if kwargs['upload']: + if self.upload: # Submit results. Yes, Python has some HTTP libraries, but # none of the ones that are part of the standard library can # upload a file without writing dozens of lines of code. @@ -866,9 +867,9 @@ def main(): make_opts = (args['--make-opts'] or ''), nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], - repo = args['--repo']) + repo = args['--repo'], + upload = upload) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - upload = upload, buildpid = buildpid, debug = args['--debug'] )) From patchwork Thu Aug 1 02:46:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140209 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="eyU8uMGb"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZT95Y1kz9sNF for ; Thu, 1 Aug 2019 12:47:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id E20E68600D; Thu, 1 Aug 2019 02:47:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DUA04Dwyg3Il; Thu, 1 Aug 2019 02:47:23 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4F0578602D; Thu, 1 Aug 2019 02:47:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id D8A111BF59D for ; Thu, 1 Aug 2019 02:47:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D59ED86BB7 for ; Thu, 1 Aug 2019 02:47:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EgadzRf20nsp for ; Thu, 1 Aug 2019 02:47:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id 1929586B94 for ; Thu, 1 Aug 2019 02:47:21 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id t132so33171833pgb.9 for ; Wed, 31 Jul 2019 19:47:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MHod2FHMLaBwSOLqciL+ZW/QXgkguLYbMOvAlz5xHWE=; b=eyU8uMGbFxD8ljdQ9X/p6zw7VIAzKpVS+I3j2inlNsdolfwU3NL83QK3P5r5CTo3eo oc10NWDR5955U8LzU0zexRg5EzckqHD4/gXFMmUw24PONee7XWCOfMm2O5YY2JI3gcwl v3AmoHbQONxJndOuMfM4Q8DcJe7YlzI0yKfuFbFbOK6nRA2KXLGyfqFK4jiQJTTIF3JC 8cctN6EzfyT4pMFnmJ7Qb/tQt9KSCTC8MmwBxntb09BaBwxwRxcRMPx07Id97yJKy7+O AvWL1BLP50lyHbbgNGwoYNhcaSFFEEfgI7br9I9vnxmysZ2fzF6AIeDtSFgnJMJZ5/Xj O26Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MHod2FHMLaBwSOLqciL+ZW/QXgkguLYbMOvAlz5xHWE=; b=XMtgPFMWlJehbI/nQm+5eQ0KzyLXF8D+Ze6uys6152Pu+1CtvXrS4jNNuvuD3yhQLl GzzPNqwXE4X5uBaRAhkKrmwFS+FkvaGlUNO6d7+0vZ6icLFhiDZRm16yep5X/tb70c5M x7ogHVNDCxMEAlU/5L1HM+N2RIYSSQw+ZM6s9lvEeM8XIxm+PUjbPo5856F0nlUNo/5y k6Y+yWBgXZrLvR2mKuDnSiFvPe2jw9hmy9phZKBJQ+O1bHHSkLARUU1sXvR09s+/wwLs mtlBNszSMkDuMWzwqonax2kO73Yvx3k02DMnwiLyJgXr/+AFcyJ6Rf5oX5EAnxCWCjyt Im5g== X-Gm-Message-State: APjAAAXNblf+fX8afxymKUhn8cGr3SPEkrv4ku1uYB/HZnWiPZE2Jt6J ilM8lkZvwbwuaXUbikutuJoGWQCD5I0= X-Google-Smtp-Source: APXvYqyr9R1EukOcd1wEWhrl3R+7tFbCxbje3uQG/dMWHwm0WU8tHHmRjA7ToL8gF1uvtawXHWqTZg== X-Received: by 2002:aa7:9ad2:: with SMTP id x18mr52845269pfp.192.1564627640309; Wed, 31 Jul 2019 19:47:20 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.18 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:20 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:25 +0530 Message-Id: <20190801024643.11024-13-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 12/30] autobuild-run: move buildpid from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index e217117..05de063 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -274,7 +274,7 @@ class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, submitter, make_opts, nice, toolchains_csv, - repo, upload): + repo, upload, buildpid): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -287,6 +287,7 @@ class Builder: self.toolchains_csv = toolchains_csv self.repo = repo self.upload = upload + self.buildpid = buildpid def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -482,9 +483,9 @@ class Builder: build_monitor.daemon = True build_monitor.start() - kwargs['buildpid'][self.instance] = sub.pid + self.buildpid[self.instance] = sub.pid ret = sub.wait() - kwargs['buildpid'][self.instance] = 0 + self.buildpid[self.instance] = 0 # If build failed, monitor thread would have exited at this point if monitor_thread_hung_build_flag.is_set(): @@ -868,9 +869,9 @@ def main(): nice = (args['--nice'] or 0), toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], - upload = upload) + upload = upload, + buildpid = buildpid) p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - buildpid = buildpid, debug = args['--debug'] )) p.start() From patchwork Thu Aug 1 02:46:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140210 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="eQpKb6se"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTD38CQz9sMr for ; Thu, 1 Aug 2019 12:47:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E4FDD87E5B; Thu, 1 Aug 2019 02:47:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id geKCCggfBlFx; Thu, 1 Aug 2019 02:47:26 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 4AAEF87E4D; Thu, 1 Aug 2019 02:47:26 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 9B5B31BF59D for ; Thu, 1 Aug 2019 02:47:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 934BF2266F for ; Thu, 1 Aug 2019 02:47:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8uga7tken+vN for ; Thu, 1 Aug 2019 02:47:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by silver.osuosl.org (Postfix) with ESMTPS id 59999226B3 for ; Thu, 1 Aug 2019 02:47:23 +0000 (UTC) Received: by mail-pf1-f196.google.com with SMTP id y15so33053651pfn.5 for ; Wed, 31 Jul 2019 19:47:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vFEF8xNwPY/nI4pGo1F9VQO3SGe4ZjXobS9tnyaybLM=; b=eQpKb6se9meVe7nyjqPIZoInEndQCDKDIQ7D2NKVBYhfPKGs26SDd5zTrVNAvK2Ywo kGBtpN4lkeYGUGfGwOF+/hQzpJZ0Pnq5veJyWIfWZjJPkU7WaXmeWBdecHipgKKkbNa5 uZ1mnKTsp9OKet7GPrctAC6TNjcpakB8iSykI91NOBgn/Rnqqthp+5OGFJHU90PdOIqv rzGeBHNj7SyF3v4s3YzcHfgDTzey9U5FCbv9f1L3hvgGzw5sG8FZKHAFLNfkhs+fqvRS FKN9tiZGaaI5x53OgFY4bMwiCzG9A0dDmCrp//1+hN2ybqKnkxc3gokFjpw+1+p/4sgt WpLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vFEF8xNwPY/nI4pGo1F9VQO3SGe4ZjXobS9tnyaybLM=; b=WqmlW8dyNeAECooA0qZp6SCvM9r3tCakHYc7A4uH6h5/XPayBSmCboyE6Mqdc1l4AK wol9dpYX9PBhN4OBSOlhs3iRu9fN+5IrWff9/nYgw2q50l4NE60h2xGFEQCpO3/GeY+H dWyiW4nNKlzH3BjLh1RBw0AxNGDvln2xYE3/4R7xfSqrRAPDfBpA/y81FF8BdBM7Pjdc X7NY+l0pIB9dq95RvhNb+cGwMcxpjZyHAaDXOcNJEOQMHHiuP+/s8noTtqXiBuIombmJ Y29+AqsXVGc1pco5gVmlO3GlUag+zC1YVTlo7/mETiX4WMefEw7p4fv6jhZB/RnHXteL zmyw== X-Gm-Message-State: APjAAAW1QpSwV4TDcx8CULYvAnK1Fu9puHq9y5NmU+oE2G8bazF1srx/ 1kCOC3fu+RygQpk0obA42mj2xdAoATg= X-Google-Smtp-Source: APXvYqydhT47mvyIpQywKs2OrdLWrXAR0yyCZEtMXDX4P/7uhwfGqjpNhUDtUrmyj8vwPvFUfov5+g== X-Received: by 2002:a62:38c6:: with SMTP id f189mr50780179pfa.157.1564627642581; Wed, 31 Jul 2019 19:47:22 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.20 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:22 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:26 +0530 Message-Id: <20190801024643.11024-14-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 13/30] autobuild-run: move debug from kwargs to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Explicitly state class constructor argument --- scripts/autobuild-run | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 05de063..923652c 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -274,7 +274,7 @@ class Builder: def __init__(self, instance, njobs, sysinfo, http_url, http_login, http_password, submitter, make_opts, nice, toolchains_csv, - repo, upload, buildpid): + repo, upload, buildpid, debug): self.instance = instance self.njobs = njobs self.sysinfo = sysinfo @@ -288,6 +288,7 @@ class Builder: self.repo = repo self.upload = upload self.buildpid = buildpid + self.debug = debug def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -378,7 +379,7 @@ class Builder: log_write(log, "INFO: generate the configuration") - if kwargs['debug']: + if self.debug: devnull = log else: devnull = open(os.devnull, "w") @@ -724,7 +725,7 @@ class Builder: if not os.path.exists(idir): os.mkdir(idir) - if kwargs['debug']: + if self.debug: kwargs['log'] = sys.stdout else: kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") @@ -870,10 +871,9 @@ def main(): toolchains_csv = args['--toolchains-csv'], repo = args['--repo'], upload = upload, - buildpid = buildpid) - p = multiprocessing.Process(target=builder.run_instance, kwargs=dict( - debug = args['--debug'] - )) + buildpid = buildpid, + debug = args['--debug']) + p = multiprocessing.Process(target=builder.run_instance) p.start() processes.append(p) From patchwork Thu Aug 1 02:46:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140211 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OonEmGni"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTG6jZQz9sMr for ; Thu, 1 Aug 2019 12:47:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 2D6D186B90; Thu, 1 Aug 2019 02:47:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hs3rMId60vL0; Thu, 1 Aug 2019 02:47:27 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 72F5986B94; Thu, 1 Aug 2019 02:47:27 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 634C21BF59D for ; Thu, 1 Aug 2019 02:47:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5803186081 for ; Thu, 1 Aug 2019 02:47:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8ySEoFRG29Ax for ; Thu, 1 Aug 2019 02:47:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by fraxinus.osuosl.org (Postfix) with ESMTPS id A184D8602D for ; Thu, 1 Aug 2019 02:47:25 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id k189so14134682pgk.13 for ; Wed, 31 Jul 2019 19:47:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JULdsXfxnFfDm4CpNdKO1WAk6JndMs66hs4Oh5v+SLM=; b=OonEmGnitlPqsjOljkth55Esl97iu8k8T+lzrnNyRhJMKK4+RZiW0g15BMOA00Ajkx bXUc5rs4LMJWscwO/1kweCwr/aHvQR6VDfDSfmwzhnKc84Rx5morhfEIFabfoR+Pp49P UNLMr3FE5b4zIYVJL46KzvzGjrLDJfwblH8tfHQGZAfHDFUi4eixCIk9yepddrhGsLMy Kyc2XlxNoCH76kHUVgG6y20cFxOyt0RC6vf0kfeGz3FMCdN5q8MnhB9g473bgJctQU+q 27QjNFqg2+w7IitlD72SQvVHhaTBVh6HlNzZPxRVwa7+Kvq/dsYeqY96D2BBqJxzujke w0qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JULdsXfxnFfDm4CpNdKO1WAk6JndMs66hs4Oh5v+SLM=; b=IZOQGpgePR9xqmu8KZoXc3lHEgT6W5FqMGF9yS3Bi83jAtLDG6gdeoeyEItoHt8RE5 bKnN06BYs/T+vrDxoDgqiMSFiR7t71nG6ebfG5Gay6ly5GF50rny/Lju+mv4ttirgBYd vieHJNGw6RynaamcOES8Nzl19SM90tM3rL2eOgG65oCUwcvWQ4LUqxkOgoEyejHQeqcR RqQlVLdkdyCqwDEdWg9bvghkdQ0xRXhY032Of/h3j4K9FhKQv30yiTjOHoCCthyg3Non LBmSod9j6VtehHTC8tfkhqJ52z6pkadd305zyfrPujZPP7KyRwA5ZCEtOJkZoGMwXTv5 duLg== X-Gm-Message-State: APjAAAXsWkl036N/pF+ZPxXNvM+VmJw6bKfV4W7pAbPUtyk1xT/9t4Ag sZ7495NxJA1PkB543gXs0l7SChLiojw= X-Google-Smtp-Source: APXvYqz4MZaauCXeMjj6wYDh2uJx2GuVDlQRsNDRSrj+A/SErTx1DR4pAdMH2bvB0XXDUNhqVmXkIQ== X-Received: by 2002:a62:2aca:: with SMTP id q193mr51770242pfq.209.1564627644876; Wed, 31 Jul 2019 19:47:24 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.22 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:24 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:27 +0530 Message-Id: <20190801024643.11024-15-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 14/30] autobuild-run: define instance directory as a part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" We need idir as a class member before moving log out of kwargs. Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 49 +++++++++++++++++++------------------------ 1 file changed, 22 insertions(+), 27 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 923652c..d1ecbb3 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -290,6 +290,9 @@ class Builder: self.buildpid = buildpid self.debug = debug + # frequently needed directories + self.idir = "instance-%d" % self.instance + def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -298,13 +301,12 @@ class Builder: code, and cleaning up remaining stuff from previous builds. """ - idir = "instance-%d" % self.instance log = kwargs['log'] log_write(log, "INFO: preparing a new build") # Create the download directory if it doesn't exist - dldir = os.path.join(idir, "dl") + dldir = os.path.join(self.idir, "dl") if not os.path.exists(dldir): os.mkdir(dldir) @@ -339,7 +341,7 @@ class Builder: # Clone Buildroot. This only happens if the source directory # didn't exist already. - srcdir = os.path.join(idir, "buildroot") + srcdir = os.path.join(self.idir, "buildroot") if not os.path.exists(srcdir): ret = subprocess.call(["git", "clone", self.repo, srcdir], stdout=log, stderr=log) @@ -360,7 +362,7 @@ class Builder: return -1 # Create an empty output directory. We remove it first, in case a previous build was aborted. - outputdir = os.path.join(idir, "output") + outputdir = os.path.join(self.idir, "output") if os.path.exists(outputdir): # shutil.rmtree doesn't remove write-protected files subprocess.call(["rm", "-rf", outputdir]) @@ -372,10 +374,9 @@ class Builder: def gen_config(self, **kwargs): """Generate a new random configuration.""" - idir = "instance-%d" % self.instance log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") log_write(log, "INFO: generate the configuration") @@ -422,9 +423,8 @@ class Builder: """ log = kwargs['log'] - idir = "instance-%d" % self.instance - outputdir = os.path.join(idir, "output") - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.join(self.idir, "output") + srcdir = os.path.join(self.idir, "buildroot") reproducible_results = os.path.join(outputdir, "results", "reproducible_results") # Using only tar images for now build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") @@ -454,16 +454,15 @@ class Builder: def do_build(self, **kwargs): """Run the build itself""" - idir = "instance-%d" % self.instance log = kwargs['log'] # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the # Buildroot sources, but to the location of the autobuilder # script. - dldir = os.path.abspath(os.path.join(idir, "dl")) - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + dldir = os.path.abspath(os.path.join(self.idir, "dl")) + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") f = open(os.path.join(outputdir, "logfile"), "w+") log_write(log, "INFO: build started") @@ -518,9 +517,8 @@ class Builder: perform the actual build. """ - idir = "instance-%d" % self.instance - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") log = kwargs['log'] # Start the first build @@ -556,11 +554,10 @@ class Builder: are available) or stores them locally as tarballs. """ - idir = "instance-%d" % self.instance log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(idir, "output")) - srcdir = os.path.join(idir, "buildroot") + outputdir = os.path.abspath(os.path.join(self.idir, "output")) + srcdir = os.path.join(self.idir, "buildroot") resultdir = os.path.join(outputdir, "results") shutil.copyfile(os.path.join(outputdir, ".config"), @@ -719,16 +716,14 @@ class Builder: results. """ - idir = "instance-%d" % self.instance - # If it doesn't exist, create the instance directory - if not os.path.exists(idir): - os.mkdir(idir) + if not os.path.exists(self.idir): + os.mkdir(self.idir) if self.debug: kwargs['log'] = sys.stdout else: - kwargs['log'] = open(os.path.join(idir, "instance.log"), "a+") + kwargs['log'] = open(os.path.join(self.idir, "instance.log"), "a+") log_write(kwargs['log'], "INFO: instance started") while True: @@ -738,7 +733,7 @@ class Builder: if ret != 0: continue - resultdir = os.path.join(idir, "output", "results") + resultdir = os.path.join(self.idir, "output", "results") os.mkdir(resultdir) ret = self.gen_config(**kwargs) @@ -747,7 +742,7 @@ class Builder: continue # Check if the build test is supposed to be a reproducible test - outputdir = os.path.abspath(os.path.join(idir, "output")) + outputdir = os.path.abspath(os.path.join(self.idir, "output")) with open(os.path.join(outputdir, ".config"), "r") as fconf: reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() if reproducible: From patchwork Thu Aug 1 02:46:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140213 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="j7P/jbNh"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTP1LGtz9sMr for ; Thu, 1 Aug 2019 12:47:37 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8F1AF226B3; Thu, 1 Aug 2019 02:47:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rA-CSSeJFuK3; Thu, 1 Aug 2019 02:47:30 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 499322267B; Thu, 1 Aug 2019 02:47:30 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 371F21BF59D for ; Thu, 1 Aug 2019 02:47:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 347362267B for ; Thu, 1 Aug 2019 02:47:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VBwojlln7ga6 for ; Thu, 1 Aug 2019 02:47:28 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by silver.osuosl.org (Postfix) with ESMTPS id 2AED92266F for ; Thu, 1 Aug 2019 02:47:28 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id i18so33181821pgl.11 for ; Wed, 31 Jul 2019 19:47:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=n8IeSIJyIy9uECnxTJSeftrSeRw3fFR3eyDIRrQ8ZNc=; b=j7P/jbNhoahWWuBOELkNd1B7Y1Rf7yruet8fwJ1avxpaiWdC97dAlKncNxY0i0lMjK e5FNggbaQNierWNVGbkRqxlLocRZx4Zgas1SoTC9jn9a64xiAjE1khpE3Y460P4+kXxk ChfJA3ygdf+nnn+eW67EbUAfzJjba+I/swJnA0I2vU+pIKK9N2nTwg1zq6lUoXTyPQ4I 9NUt+GvQ0Z7lqyY8ytKEuQuUdJHM537lbHXYtqEKlBZyzNAYhx6LWa54Q9+LOijY9/Rz N/4Zijd5eibqemg1OCA4oFfskkli0DZMqrWxvsqKug+ZPGPGSyWNfAn3DGI/lwAMflsO WwJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=n8IeSIJyIy9uECnxTJSeftrSeRw3fFR3eyDIRrQ8ZNc=; b=G2+Y7I987tJwZD7zEHfwp/cxvmMDyXnsqZjt7YiFQFCfDZzV1nZWZUoEOj5IsM0o7b MJHoGv+QF1++3M/Llp4CAnrHZZNqn08mamqcxZvE2bE3oXVw08mNkYj0SrnsqH1kpjXO oRDyFGbRaHa6xR1ZcpEKvLE1pFKnZqL40ffLuMjfYjdzjX7g6y+ZkXm5gKNxwwxOaFQK dBEwItrN6TzxauyTcQFliq0ju6jNUEWMLQsjTMycx/AJA8DyMQM/4HjoNtZ9YykbhNSD fhJyzUqU44eGUtiiHJs/AqeQwPVYJzvkEJSizvCUSj/nuyysO3u7uUqA0Jj3YYrbKb+H nzWg== X-Gm-Message-State: APjAAAWZG8u7NyklpGlJ5o1or6IFa9BButYYqF57lCyt2YpLIBM2mOAv MYWty3r3M2p/NQ+8hLO01yec4eQXOmA= X-Google-Smtp-Source: APXvYqyfRGxg7pJdGOAKBlZjrRn4RkqXrdo2IbXd30SkN0WYtmjPvhKRlWwkvL0V9IpolC55SPLXxw== X-Received: by 2002:aa7:84d4:: with SMTP id x20mr50101940pfn.60.1564627647321; Wed, 31 Jul 2019 19:47:27 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.25 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:26 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:28 +0530 Message-Id: <20190801024643.11024-16-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 15/30] autobuild-run: move log variable to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Fix whitespace errors --- scripts/autobuild-run | 94 ++++++++++++++++++++----------------------- 1 file changed, 43 insertions(+), 51 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index d1ecbb3..cb00745 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -293,6 +293,11 @@ class Builder: # frequently needed directories self.idir = "instance-%d" % self.instance + if self.debug: + self.log = sys.stdout + else: + self.log = open(os.path.join(self.idir, "instance.log"), "a+") + def prepare_build(self, **kwargs): """Prepare for the next build of the specified instance @@ -301,9 +306,7 @@ class Builder: code, and cleaning up remaining stuff from previous builds. """ - log = kwargs['log'] - - log_write(log, "INFO: preparing a new build") + log_write(self.log, "INFO: preparing a new build") # Create the download directory if it doesn't exist dldir = os.path.join(self.idir, "dl") @@ -332,33 +335,33 @@ class Builder: if not flist: break f = flist[randint(0, len(flist) - 1)] - log_write(log, "INFO: removing %s from downloads" % + log_write(self.log, "INFO: removing %s from downloads" % os.path.relpath(f, dldir)) os.remove(f) branch = get_branch() - log_write(log, "INFO: testing branch '%s'" % branch) + log_write(self.log, "INFO: testing branch '%s'" % branch) # Clone Buildroot. This only happens if the source directory # didn't exist already. srcdir = os.path.join(self.idir, "buildroot") if not os.path.exists(srcdir): ret = subprocess.call(["git", "clone", self.repo, srcdir], - stdout=log, stderr=log) + stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not clone Buildroot sources") + log_write(self.log, "ERROR: could not clone Buildroot sources") return -1 # Update the Buildroot sources. abssrcdir = os.path.abspath(srcdir) - ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=log, stderr=log) + ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not fetch Buildroot sources") + log_write(self.log, "ERROR: could not fetch Buildroot sources") return -1 - ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=log, stderr=log) + ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not check out Buildroot sources") + log_write(self.log, "ERROR: could not check out Buildroot sources") return -1 # Create an empty output directory. We remove it first, in case a previous build was aborted. @@ -374,14 +377,13 @@ class Builder: def gen_config(self, **kwargs): """Generate a new random configuration.""" - log = kwargs['log'] outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") - log_write(log, "INFO: generate the configuration") + log_write(self.log, "INFO: generate the configuration") if self.debug: - devnull = log + devnull = self.log else: devnull = open(os.devnull, "w") @@ -394,12 +396,12 @@ class Builder: toolchains_csv = os.path.join(srcdir, toolchains_csv) args.extend(["--toolchains-csv", toolchains_csv]) - ret = subprocess.call(args, stdout=devnull, stderr=log) + ret = subprocess.call(args, stdout=devnull, stderr=self.log) return ret def stop_on_build_hang(self, monitor_thread_hung_build_flag, monitor_thread_stop_flag, sub_proc, - outputdir, log): + outputdir): build_time_logfile = os.path.join(outputdir, "build/build-time.log") while True: if monitor_thread_stop_flag.is_set(): @@ -410,7 +412,7 @@ class Builder: if mtime < datetime.datetime.now() - datetime.timedelta(minutes=HUNG_BUILD_TIMEOUT): if sub_proc.poll() is None: monitor_thread_hung_build_flag.set() # Used by do_build() to determine build hang - log_write(log, "INFO: build hung") + log_write(self.log, "INFO: build hung") sub_proc.kill() break monitor_thread_stop_flag.wait(30) @@ -422,7 +424,6 @@ class Builder: installed, fallback to cmp """ - log = kwargs['log'] outputdir = os.path.join(self.idir, "output") srcdir = os.path.join(self.idir, "buildroot") reproducible_results = os.path.join(outputdir, "results", "reproducible_results") @@ -436,26 +437,24 @@ class Builder: prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string prefix = prefix[13:-1] - log_write(log, "INFO: running diffoscope on images") + log_write(self.log, "INFO: running diffoscope on images") subprocess.call(["diffoscope", build_1_image, build_2_image, - "--tool-prefix-binutils", prefix], stdout=diff, stderr=log) + "--tool-prefix-binutils", prefix], stdout=diff, stderr=self.log) else: - log_write(log, "INFO: diffoscope not installed, falling back to cmp") - subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=log) + log_write(self.log, "INFO: diffoscope not installed, falling back to cmp") + subprocess.call(["cmp", "-b", build_1_image, build_2_image], stdout=diff, stderr=self.log) if os.stat(reproducible_results).st_size > 0: - log_write(log, "INFO: Build is non-reproducible.") + log_write(self.log, "INFO: Build is non-reproducible.") return -1 # rootfs images match byte-for-byte -> reproducible image - log_write(log, "INFO: Build is reproducible!") + log_write(self.log, "INFO: Build is reproducible!") return 0 def do_build(self, **kwargs): """Run the build itself""" - log = kwargs['log'] - # We need the absolute path to use with O=, because the relative # path to the output directory here is not relative to the # Buildroot sources, but to the location of the autobuilder @@ -464,7 +463,7 @@ class Builder: outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") f = open(os.path.join(outputdir, "logfile"), "w+") - log_write(log, "INFO: build started") + log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, @@ -479,7 +478,7 @@ class Builder: build_monitor = Thread(target=self.stop_on_build_hang, args=(monitor_thread_hung_build_flag, monitor_thread_stop_flag, - sub, outputdir, log)) + sub, outputdir)) build_monitor.daemon = True build_monitor.start() @@ -489,7 +488,7 @@ class Builder: # If build failed, monitor thread would have exited at this point if monitor_thread_hung_build_flag.is_set(): - log_write(log, "INFO: build timed out [%d]" % ret) + log_write(self.log, "INFO: build timed out [%d]" % ret) return -2 else: # Stop monitor thread as this build didn't timeout @@ -497,7 +496,7 @@ class Builder: # Monitor thread should be exiting around this point if ret != 0: - log_write(log, "INFO: build failed [%d]" % ret) + log_write(self.log, "INFO: build failed [%d]" % ret) return -1 cmd = ["make", "O=%s" % outputdir, "-C", srcdir, @@ -505,9 +504,9 @@ class Builder: + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) if ret != 0: - log_write(log, "INFO: build failed during legal-info") + log_write(self.log, "INFO: build failed during legal-info") return -1 - log_write(log, "INFO: build successful") + log_write(self.log, "INFO: build successful") return 0 def do_reproducible_build(self, **kwargs): @@ -519,13 +518,12 @@ class Builder: outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") - log = kwargs['log'] # Start the first build - log_write(log, "INFO: Reproducible Build Test, starting build 1") + log_write(self.log, "INFO: Reproducible Build Test, starting build 1") ret = self.do_build(**kwargs) if ret != 0: - log_write(log, "INFO: build 1 failed, skipping build 2") + log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret # First build has been built, move files and start build 2 @@ -536,10 +534,10 @@ class Builder: subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) # Start the second build - log_write(log, "INFO: Reproducible Build Test, starting build 2") + log_write(self.log, "INFO: Reproducible Build Test, starting build 2") ret = self.do_build(**kwargs) if ret != 0: - log_write(log, "INFO: build 2 failed") + log_write(self.log, "INFO: build 2 failed") return ret # Assuming both have built successfully @@ -554,8 +552,6 @@ class Builder: are available) or stores them locally as tarballs. """ - log = kwargs['log'] - outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") resultdir = os.path.join(outputdir, "results") @@ -681,9 +677,9 @@ class Builder: # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=log, stderr=log) + cwd=outputdir, stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "ERROR: could not make results tarball") + log_write(self.log, "ERROR: could not make results tarball") sys.exit(1) if self.upload: @@ -696,18 +692,18 @@ class Builder: "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", self.http_url], - stdout=log, stderr=log) + stdout=self.log, stderr=self.log) if ret != 0: - log_write(log, "INFO: results could not be submitted, %d" % ret) + log_write(self.log, "INFO: results could not be submitted, %d" % ret) else: - log_write(log, "INFO: results were submitted successfully") + log_write(self.log, "INFO: results were submitted successfully") else: # No http login/password, keep tarballs locally with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: sha1 = hashlib.sha1(f.read()).hexdigest() resultfilename = "instance-%d-%s.tar.bz2" % (self.instance, sha1) os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) - log_write(log, "INFO: results saved as %s" % resultfilename) + log_write(self.log, "INFO: results saved as %s" % resultfilename) def run_instance(self, **kwargs): """Main per-instance loop @@ -720,11 +716,7 @@ class Builder: if not os.path.exists(self.idir): os.mkdir(self.idir) - if self.debug: - kwargs['log'] = sys.stdout - else: - kwargs['log'] = open(os.path.join(self.idir, "instance.log"), "a+") - log_write(kwargs['log'], "INFO: instance started") + log_write(self.log, "INFO: instance started") while True: check_version() @@ -738,7 +730,7 @@ class Builder: ret = self.gen_config(**kwargs) if ret != 0: - log_write(kwargs['log'], "WARN: failed to generate configuration") + log_write(self.log, "WARN: failed to generate configuration") continue # Check if the build test is supposed to be a reproducible test From patchwork Thu Aug 1 02:46:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140212 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="djGR5+1X"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTL3P2Yz9sMr for ; Thu, 1 Aug 2019 12:47:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id B77AB87E6D; Thu, 1 Aug 2019 02:47:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J6o6okIwThEh; Thu, 1 Aug 2019 02:47:32 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 1F13187E59; Thu, 1 Aug 2019 02:47:32 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id EAC1B1BF59D for ; Thu, 1 Aug 2019 02:47:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E7DF287E59 for ; Thu, 1 Aug 2019 02:47:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jgShVcSBrr6W for ; Thu, 1 Aug 2019 02:47:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by hemlock.osuosl.org (Postfix) with ESMTPS id 62F2287E4D for ; Thu, 1 Aug 2019 02:47:30 +0000 (UTC) Received: by mail-pg1-f196.google.com with SMTP id o13so33165283pgp.12 for ; Wed, 31 Jul 2019 19:47:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Pm3yZoBcuycz57fAm4ovvVjemf6JsU9dWM0c92PGE/0=; b=djGR5+1XAnACaGBf6LuZ9SSycM+DnHUoKXahk6hKAhNwIfFu5XGpgff1cMuw5xonAT GewEmoEfeb79We39Hxu9RxG0m8zoeScmVP68seGMCXHqWbMVqPxtUt5WU7x3+4YJZ1os JDO3E/6zocB7i8UnKoB7mJCl5/xfpJ9Ap0ssy4AA0L8Itf1mMNadwQWXwHtUqAlNKB6D NQucIOordRnPZSYuLrGhzffK+f8wCjv/78v+tH3XhSJ++cmJjA6gPVhKWbYn44O2ArLu XNuf8e5DDChPhkQIIH6n1sAq7ySnOppWijnu/QhmBngfLIFr/GnQ+sLd6D1PVJGaZSVN g3Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Pm3yZoBcuycz57fAm4ovvVjemf6JsU9dWM0c92PGE/0=; b=Iy3UlJ+SNmAQ+RCMK3OobpUKD8z7XLl2mnI395XIgKPhZWpTX13HaOwAtmRlePQ8GZ PycXG7keE01sOlJDnTzp3H1YHyhwQZoxEibl82+ocTSSX3UiFcDHgnL4U0jZUsz8BTcF dlX9wWW7YP0h0Gnfc8Q00THSuZSyU/L+SqTTb0HoLpxo+S2xPiswLkNEV/8ZZ4TxLDuV Mt88XazcmwptI/j/93eqFyugFCFl//t4RdZWeKNNvcP7NudHwX/lDKhbAk6L/XgrsePt Lu6E1SwA/ap8/IESGGUDxvO/fdI1T4Y1IYl9QkASUBoJyzQFNpfqJACBpzYsIyojz3I6 yekw== X-Gm-Message-State: APjAAAUp9s45XJGyIWn5s1hboH0mz4+buXGWUoE0HD+l18PnQhrUwzEn dxwfw76Pychw52d+qTFPu7drbw6DIx8= X-Google-Smtp-Source: APXvYqwpmlgOKjN/878ZYF/EDcpnmT6my+ije4QRGXCl9yaOKJZbf8iwSDkLKFuNOtcY99YbASz6lA== X-Received: by 2002:a63:184b:: with SMTP id 11mr56354080pgy.112.1564627649615; Wed, 31 Jul 2019 19:47:29 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.27 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:29 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:29 +0530 Message-Id: <20190801024643.11024-17-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 16/30] autobuild-run: remove kwargs argument from function calls and definitions X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index cb00745..adc8795 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -298,7 +298,7 @@ class Builder: else: self.log = open(os.path.join(self.idir, "instance.log"), "a+") - def prepare_build(self, **kwargs): + def prepare_build(self): """Prepare for the next build of the specified instance This function prepares the build by making sure all the needed @@ -375,7 +375,7 @@ class Builder: return 0 - def gen_config(self, **kwargs): + def gen_config(self): """Generate a new random configuration.""" outputdir = os.path.abspath(os.path.join(self.idir, "output")) srcdir = os.path.join(self.idir, "buildroot") @@ -417,7 +417,7 @@ class Builder: break monitor_thread_stop_flag.wait(30) - def check_reproducibility(self, **kwargs): + def check_reproducibility(self): """Check reproducibility of builds Use diffoscope on the built images, if diffoscope is not @@ -452,7 +452,7 @@ class Builder: log_write(self.log, "INFO: Build is reproducible!") return 0 - def do_build(self, **kwargs): + def do_build(self): """Run the build itself""" # We need the absolute path to use with O=, because the relative @@ -509,7 +509,7 @@ class Builder: log_write(self.log, "INFO: build successful") return 0 - def do_reproducible_build(self, **kwargs): + def do_reproducible_build(self): """Run the builds for reproducibility testing Build twice with the same configuration. Calls do_build() to @@ -521,7 +521,7 @@ class Builder: # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") - ret = self.do_build(**kwargs) + ret = self.do_build() if ret != 0: log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret @@ -535,16 +535,16 @@ class Builder: # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") - ret = self.do_build(**kwargs) + ret = self.do_build() if ret != 0: log_write(self.log, "INFO: build 2 failed") return ret # Assuming both have built successfully - ret = self.check_reproducibility(**kwargs) + ret = self.check_reproducibility() return ret - def send_results(self, result, **kwargs): + def send_results(self, result): """Prepare and store/send tarball with results This function prepares the tarball with the results, and either @@ -705,7 +705,7 @@ class Builder: os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) log_write(self.log, "INFO: results saved as %s" % resultfilename) - def run_instance(self, **kwargs): + def run_instance(self): """Main per-instance loop Prepare the build, generate a configuration, run the build, and submit the @@ -721,14 +721,14 @@ class Builder: while True: check_version() - ret = self.prepare_build(**kwargs) + ret = self.prepare_build() if ret != 0: continue resultdir = os.path.join(self.idir, "output", "results") os.mkdir(resultdir) - ret = self.gen_config(**kwargs) + ret = self.gen_config() if ret != 0: log_write(self.log, "WARN: failed to generate configuration") continue @@ -738,11 +738,11 @@ class Builder: with open(os.path.join(outputdir, ".config"), "r") as fconf: reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() if reproducible: - ret = self.do_reproducible_build(**kwargs) + ret = self.do_reproducible_build() else: - ret = self.do_build(**kwargs) + ret = self.do_build() - self.send_results(ret, **kwargs) + self.send_results(ret) # args / config file merging inspired by: # https://github.com/docopt/docopt/blob/master/examples/config_file_example.py From patchwork Thu Aug 1 02:46:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140215 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="nK4j2Skg"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTW4C5Dz9sMr for ; Thu, 1 Aug 2019 12:47:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id C409C86B94; Thu, 1 Aug 2019 02:47:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TsQzWoLQIyT3; Thu, 1 Aug 2019 02:47:40 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 67D0A86BA4; Thu, 1 Aug 2019 02:47:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 2F4331BF59D for ; Thu, 1 Aug 2019 02:47:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 2CAF7226A0 for ; Thu, 1 Aug 2019 02:47:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hcP-HojyMh7R for ; Thu, 1 Aug 2019 02:47:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by silver.osuosl.org (Postfix) with ESMTPS id BECF6226AF for ; Thu, 1 Aug 2019 02:47:32 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id d1so363957pgp.4 for ; Wed, 31 Jul 2019 19:47:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EgOLzDsviE0OUUW1xbY5ID+MsgXqQYItfBVJHkjMr8Q=; b=nK4j2SkgpCmTne2KoNQIijWsVxGc2klVHnsJxMtBSK8OWX7AAG/7zn2n8S8JlB4mgz Cchq99/jORFFS5AXExGqAE0igZC5DC6g4BnZLwGuIoYzrT0RBYMVH1Ar4Bk7j7YdP8Is THBHJ7pP/DkbIefRRZiMwsmwMmJee11yUspZ3RPqZGNlcS5Ju7emgshww4Gy5b0T3Uro YVkpx4I3dmnT4vsnP3rd/OGQSX4kfFEF5CE0nvHBq0ebuGdxKGyWk3jJl/gcxKdTHHZz kCe64Q2FGamLxO1VoUa/ZtdpJoSu0l2p6jLR4ZsCLIwDYue89qEs2q1FvABGupoIsp6+ dlHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EgOLzDsviE0OUUW1xbY5ID+MsgXqQYItfBVJHkjMr8Q=; b=hQwniACrEQ/+qBo4/Vm5kW/43WrGAeG74mQqt8M1oPwkYVdUSva8q7V7cqdTQCzYOO XwMrnF0NJ5R5mvN9PjmlxJYb+06n4TCdqJq3ZjMS0bFPfdxqMpIGgFmtVWm9/RRIW/hP W6A3NHuSCwM8KkRpl1b8O7McMYxZ+SLg5jS1BDY8yr5qhZS6WZMr/uHcQrmpVnh3dcsZ P0Kwx7Wofb18RgX8gL8oX0CqBUSdhWGT9BhsMCW8Upoo73KQXUW5ERA5vL2krllkO0KF 14ghULTgGT0WO+7MijAgN1Iv1vwLJO3z584lvTAvE+id1n738M/2uArzFFS1tl+5pzFk /TTw== X-Gm-Message-State: APjAAAWyYum4v2XKTH5CaTfVrbOGjpKR33DpAIQejAqTumJJnyZSaanp 8U4gm4wmhoe9ntAazweegrmE23cpYEo= X-Google-Smtp-Source: APXvYqzIm3El2dUE2RitM112Uw5p/fsnwOhiojLOfpb4pk4oSfcP9IZwP//TT+k7Vp7aKJdH3x3jmQ== X-Received: by 2002:a17:90a:bf0e:: with SMTP id c14mr5716551pjs.55.1564627652035; Wed, 31 Jul 2019 19:47:32 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.29 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:31 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:30 +0530 Message-Id: <20190801024643.11024-18-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 17/30] autobuild-run: define source directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Fix whitespace issue --- scripts/autobuild-run | 29 ++++++++++++----------------- 1 file changed, 12 insertions(+), 17 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index adc8795..741d316 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -292,6 +292,7 @@ class Builder: # frequently needed directories self.idir = "instance-%d" % self.instance + self.srcdir = os.path.join(self.idir, "buildroot") if self.debug: self.log = sys.stdout @@ -344,16 +345,15 @@ class Builder: # Clone Buildroot. This only happens if the source directory # didn't exist already. - srcdir = os.path.join(self.idir, "buildroot") - if not os.path.exists(srcdir): - ret = subprocess.call(["git", "clone", self.repo, srcdir], + if not os.path.exists(self.srcdir): + ret = subprocess.call(["git", "clone", self.repo, self.srcdir], stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not clone Buildroot sources") return -1 # Update the Buildroot sources. - abssrcdir = os.path.abspath(srcdir) + abssrcdir = os.path.abspath(self.srcdir) ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not fetch Buildroot sources") @@ -378,7 +378,6 @@ class Builder: def gen_config(self): """Generate a new random configuration.""" outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") log_write(self.log, "INFO: generate the configuration") @@ -387,13 +386,13 @@ class Builder: else: devnull = open(os.devnull, "w") - args = [os.path.join(srcdir, "utils/genrandconfig"), - "-o", outputdir, "-b", srcdir] + args = [os.path.join(self.srcdir, "utils/genrandconfig"), + "-o", outputdir, "-b", self.srcdir] toolchains_csv = self.toolchains_csv if toolchains_csv: if not os.path.isabs(toolchains_csv): - toolchains_csv = os.path.join(srcdir, toolchains_csv) + toolchains_csv = os.path.join(self.srcdir, toolchains_csv) args.extend(["--toolchains-csv", toolchains_csv]) ret = subprocess.call(args, stdout=devnull, stderr=self.log) @@ -425,7 +424,6 @@ class Builder: """ outputdir = os.path.join(self.idir, "output") - srcdir = os.path.join(self.idir, "buildroot") reproducible_results = os.path.join(outputdir, "results", "reproducible_results") # Using only tar images for now build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") @@ -434,7 +432,7 @@ class Builder: with open(reproducible_results, 'w') as diff: if self.sysinfo.has("diffoscope"): # Prefix to point diffoscope towards cross-tools - prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"]) + prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string prefix = prefix[13:-1] log_write(self.log, "INFO: running diffoscope on images") @@ -461,13 +459,12 @@ class Builder: # script. dldir = os.path.abspath(os.path.join(self.idir, "dl")) outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") f = open(os.path.join(outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, - "-C", srcdir, "BR2_DL_DIR=%s" % dldir, + "-C", self.srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) @@ -499,7 +496,7 @@ class Builder: log_write(self.log, "INFO: build failed [%d]" % ret) return -1 - cmd = ["make", "O=%s" % outputdir, "-C", srcdir, + cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) @@ -517,7 +514,6 @@ class Builder: """ outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") @@ -531,7 +527,7 @@ class Builder: # Clean up build 1 f = open(os.path.join(outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f) + subprocess.call(["make", "O=%s" % outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") @@ -553,7 +549,6 @@ class Builder: """ outputdir = os.path.abspath(os.path.join(self.idir, "output")) - srcdir = os.path.join(self.idir, "buildroot") resultdir = os.path.join(outputdir, "results") shutil.copyfile(os.path.join(outputdir, ".config"), @@ -576,7 +571,7 @@ class Builder: subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], - shell=True, cwd=srcdir) + shell=True, cwd=self.srcdir) # Return True if the result should be rejected, False otherwise def reject_results(): From patchwork Thu Aug 1 02:46:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140214 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="vKpUt3j2"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTT3DHVz9sNF for ; Thu, 1 Aug 2019 12:47:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id C276A87E19; Thu, 1 Aug 2019 02:47:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vrkdk2LhZKCM; Thu, 1 Aug 2019 02:47:39 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 263B387E59; Thu, 1 Aug 2019 02:47:39 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 9AB751BF59D for ; Thu, 1 Aug 2019 02:47:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9253F226FC for ; Thu, 1 Aug 2019 02:47:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wwAzXE3J1TjV for ; Thu, 1 Aug 2019 02:47:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by silver.osuosl.org (Postfix) with ESMTPS id 155A4226B7 for ; Thu, 1 Aug 2019 02:47:35 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id m30so33053772pff.8 for ; Wed, 31 Jul 2019 19:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kG2d14hMokqPCcnbXXf0rNQtRalwe8OWSiiUmlDWwBE=; b=vKpUt3j2FS6oN/dwMPjVaqQMiKYv+Jul9zt1+BmHjGlWGVpd63nnMPrbMBrl8u08di sDMpTNgZ9nu7mZDt/bOsWFgxXawDzpKPFk2Np5WfbQLbJiQbPrr0tubjXsc3oF4J/yj4 YAg4nXmOVdc3QpW7ph4GGD137ZqFtR2vOy71dwXP1jw4T8KBglraXP66m3xY+HspLr7t nzlBQz2uOjtWiJkCrRc4EU6qorVw1N5DPEOIT9cm+/5Ei9nJA/kz2c/trOZh8vJaQ2yt afasicX2Te15KbD7p81Yj6DiDg/b8YhYyKWZtH6Q1SQSlShemEEtLsX+sNB+yIXmnOVu 1PWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kG2d14hMokqPCcnbXXf0rNQtRalwe8OWSiiUmlDWwBE=; b=j35JW7f7z7DnHwxoaR2ePNGf2r/SKW0YYGIv9XxL/HylgfzO8BmlzPmDgyU+oBT1o+ iPJXYdoLDjbX9LLZVzUphlBXuil32zAfw956lNZQtqWUF3cYQGywVf3DE2knnoUqJnOz QmD+DYBj4AyhTR7XUAITPKEYX6o8Yksli0LtWi5w8akBe66ypOL0Y2dlyzrKPBMPn96m gVx66d+TbwhyU47Gh2edbXjCwBLgkiMY5iRmrUCp8OR9wtAA33S6R3JZcB1Gd9UIb1gk FL94w8dJga7qckKkXN9F8gWhmB4/J5Oe2iNDCZclYqLd5rKiMFpW1hoegh1W5+GXaV4o BjxA== X-Gm-Message-State: APjAAAVMej+GQJb7U1ShDBVL8cWJ7lPR1AutpQu6VKI3xZm85fVXB3kn W/fyDX0SUUyxtlS+TQkOfhP2zQQj42E= X-Google-Smtp-Source: APXvYqycO0Xmk7hlFXHOmL0lUW2ij6ap/qiu6rG5rqC/FqEtCdND172K2z9rs9rS7RYx9Xn9XVDtLA== X-Received: by 2002:a62:be04:: with SMTP id l4mr48981687pff.77.1564627654364; Wed, 31 Jul 2019 19:47:34 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.32 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:34 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:31 +0530 Message-Id: <20190801024643.11024-19-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 18/30] autobuild-run: define download directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 15 +++++++-------- 1 file changed, 7 insertions(+), 8 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 741d316..84b4dee 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -293,6 +293,7 @@ class Builder: # frequently needed directories self.idir = "instance-%d" % self.instance self.srcdir = os.path.join(self.idir, "buildroot") + self.dldir = os.path.join(self.idir, "dl") if self.debug: self.log = sys.stdout @@ -310,9 +311,8 @@ class Builder: log_write(self.log, "INFO: preparing a new build") # Create the download directory if it doesn't exist - dldir = os.path.join(self.idir, "dl") - if not os.path.exists(dldir): - os.mkdir(dldir) + if not os.path.exists(self.dldir): + os.mkdir(self.dldir) # recursively find files under root def find_files(root): @@ -332,12 +332,12 @@ class Builder: # regularly re-download files to check that their upstream # location is still correct. for i in range(0, 5): - flist = list(find_files(dldir)) + flist = list(find_files(self.dldir)) if not flist: break f = flist[randint(0, len(flist) - 1)] log_write(self.log, "INFO: removing %s from downloads" % - os.path.relpath(f, dldir)) + os.path.relpath(f, self.dldir)) os.remove(f) branch = get_branch() @@ -457,14 +457,13 @@ class Builder: # path to the output directory here is not relative to the # Buildroot sources, but to the location of the autobuilder # script. - dldir = os.path.abspath(os.path.join(self.idir, "dl")) outputdir = os.path.abspath(os.path.join(self.idir, "output")) f = open(os.path.join(outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), "make", "O=%s" % outputdir, - "-C", self.srcdir, "BR2_DL_DIR=%s" % dldir, + "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) @@ -497,7 +496,7 @@ class Builder: return -1 cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, - "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + "BR2_DL_DIR=%s" % self.dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) if ret != 0: From patchwork Thu Aug 1 02:46:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140216 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="IC1uUHY5"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTX3Mt2z9sNF for ; Thu, 1 Aug 2019 12:47:44 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 0BB5387E94; Thu, 1 Aug 2019 02:47:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1AkB+A9kWHRQ; Thu, 1 Aug 2019 02:47:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 3BEB987E76; Thu, 1 Aug 2019 02:47:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 5393B1BF59D for ; Thu, 1 Aug 2019 02:47:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 508EA85FD5 for ; Thu, 1 Aug 2019 02:47:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QVKKZtYrAGZR for ; Thu, 1 Aug 2019 02:47:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f193.google.com (mail-pl1-f193.google.com [209.85.214.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 6C8F785FB9 for ; Thu, 1 Aug 2019 02:47:37 +0000 (UTC) Received: by mail-pl1-f193.google.com with SMTP id b3so31442399plr.4 for ; Wed, 31 Jul 2019 19:47:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=M/8yMvcSoGbZLtxyQ9WZwNGCADcA1gBYLfy5FsNVw7k=; b=IC1uUHY533nLq8A3+IHDXhLIc11wfEEeooJVgSikc3794Jdg637PosAHOBAw6n3gJK xojeJ/vEtkpbRpVVKbUK0FLinPaDyuWXR142iYs39yT8nSjPy5r5yqsiplkYOQsm7U6M LQFEXhsnZ8FoZ/QZTf0xabn3Y2cog6Ex2AiRhirUiiEkKdjUvrKZxCTqfxPPSDqW3D73 AY4+96uF1R/AZIBGTEYILzpQZqHb3SDWSMeRFSTnzCKfNMzrWYUqazKv0zBenI6qwEYc AdL5M/Qdgucc6dA9LBYXug4ALnUPAagVE6kHq2beTaXhszcbMRC7iIc09fp8hWkQvSt6 JnQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M/8yMvcSoGbZLtxyQ9WZwNGCADcA1gBYLfy5FsNVw7k=; b=NlFiQFsAcvbcuG/WlzPoWzJHjTxG3DoWwzsdBJ+AwyGDpQRTUoLxKQjvYdR5L4DZet iaL5sBS1AeTR+ErYRBgGGZoDxTE37uAuiiYsOfe2XTQlFDmQSPYx8ekzQSy/CBsRFFk3 ik9HbBWsDEA+IgtN/9aXfOxDGxDSSRzJ/FAByPA5meBOU5wVyg5bdGcx39waPh/rSDZQ 6myxEQx2+FjaCHkZ78KAkyjWC7m5RnYMhL7H5mgp+uarAmwD4+3Z5ONCoyKK5hpuWlDa fw5LLQmSMKfyfFDBV9sZEyvo2WK/ONqHGlsmbPzr8aCVIsVghCwRRPE0e2tXy9Zlbk2Y xokg== X-Gm-Message-State: APjAAAUcYQsnRgdTb07ClGgEUQHQpOnWaGW9M8Ex5Wl4Zfxr0rYQsoNb kS8r/wQM/PnK0luO4Hlxx+hx6KZPT2Q= X-Google-Smtp-Source: APXvYqzdsQTIY3eMpu63Y53eiMsoGgbNGR8o0zrm27mFi2B4hZsYWvK5NRiNEkKVx6NH3Ezw8+6xvw== X-Received: by 2002:a17:902:7c05:: with SMTP id x5mr124685005pll.321.1564627656664; Wed, 31 Jul 2019 19:47:36 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.34 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:36 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:32 +0530 Message-Id: <20190801024643.11024-20-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 19/30] autobuild-run: define output directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Absolute path for output dir is required when using outputdir with system commands like make or tail. It was necessary because the relative path would be relative to the autobuilder script rather than the buildroot source directory. It is OK to use an absolute path everywhere because an absolute path is always correct. Signed-off-by: Atharva Lele --- Changes v2 -> v3: - Use outputdir as argument in stop_on_build_hang(), required for later changes (different output dir for reproducibility testing) Changes v1 -> v2: - Fixed inaccuracies in commit message - Fixed whitespace issue --- scripts/autobuild-run | 81 ++++++++++++++++++++----------------------- 1 file changed, 37 insertions(+), 44 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 84b4dee..e225211 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -294,6 +294,11 @@ class Builder: self.idir = "instance-%d" % self.instance self.srcdir = os.path.join(self.idir, "buildroot") self.dldir = os.path.join(self.idir, "dl") + # We need the absolute path to use with O=, because the relative + # path to the output directory here is not relative to the + # Buildroot sources, but to the location of the autobuilder + # script. + self.outputdir = os.path.abspath(os.path.join(self.idir, "output")) if self.debug: self.log = sys.stdout @@ -365,19 +370,17 @@ class Builder: return -1 # Create an empty output directory. We remove it first, in case a previous build was aborted. - outputdir = os.path.join(self.idir, "output") - if os.path.exists(outputdir): + if os.path.exists(self.outputdir): # shutil.rmtree doesn't remove write-protected files - subprocess.call(["rm", "-rf", outputdir]) - os.mkdir(outputdir) - with open(os.path.join(outputdir, "branch"), "w") as branchf: + subprocess.call(["rm", "-rf", self.outputdir]) + os.mkdir(self.outputdir) + with open(os.path.join(self.outputdir, "branch"), "w") as branchf: branchf.write(branch) return 0 def gen_config(self): """Generate a new random configuration.""" - outputdir = os.path.abspath(os.path.join(self.idir, "output")) log_write(self.log, "INFO: generate the configuration") @@ -387,7 +390,7 @@ class Builder: devnull = open(os.devnull, "w") args = [os.path.join(self.srcdir, "utils/genrandconfig"), - "-o", outputdir, "-b", self.srcdir] + "-o", self.outputdir, "-b", self.srcdir] toolchains_csv = self.toolchains_csv if toolchains_csv: @@ -423,16 +426,15 @@ class Builder: installed, fallback to cmp """ - outputdir = os.path.join(self.idir, "output") - reproducible_results = os.path.join(outputdir, "results", "reproducible_results") + reproducible_results = os.path.join(self.outputdir, "results", "reproducible_results") # Using only tar images for now - build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") - build_2_image = os.path.join(outputdir, "images", "rootfs.tar") + build_1_image = os.path.join(self.outputdir, "images-1", "rootfs.tar") + build_2_image = os.path.join(self.outputdir, "images", "rootfs.tar") with open(reproducible_results, 'w') as diff: if self.sysinfo.has("diffoscope"): # Prefix to point diffoscope towards cross-tools - prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) + prefix = subprocess.check_output(["make", "O=%s" % self.outputdir, "-C", self.srcdir, "printvars", "VARS=TARGET_CROSS"]) # Remove TARGET_CROSS= and \n from the string prefix = prefix[13:-1] log_write(self.log, "INFO: running diffoscope on images") @@ -453,16 +455,11 @@ class Builder: def do_build(self): """Run the build itself""" - # We need the absolute path to use with O=, because the relative - # path to the output directory here is not relative to the - # Buildroot sources, but to the location of the autobuilder - # script. - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - f = open(os.path.join(outputdir, "logfile"), "w+") + f = open(os.path.join(self.outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), - "make", "O=%s" % outputdir, + "make", "O=%s" % self.outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() @@ -474,7 +471,7 @@ class Builder: build_monitor = Thread(target=self.stop_on_build_hang, args=(monitor_thread_hung_build_flag, monitor_thread_stop_flag, - sub, outputdir)) + sub, self.outputdir)) build_monitor.daemon = True build_monitor.start() @@ -495,7 +492,7 @@ class Builder: log_write(self.log, "INFO: build failed [%d]" % ret) return -1 - cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, + cmd = ["make", "O=%s" % self.outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) @@ -512,8 +509,6 @@ class Builder: perform the actual build. """ - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") ret = self.do_build() @@ -522,11 +517,11 @@ class Builder: return ret # First build has been built, move files and start build 2 - os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1")) + os.rename(os.path.join(self.outputdir, "images"), os.path.join(self.outputdir, "images-1")) # Clean up build 1 - f = open(os.path.join(outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) + f = open(os.path.join(self.outputdir, "logfile"), "w+") + subprocess.call(["make", "O=%s" % self.outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") @@ -547,19 +542,18 @@ class Builder: are available) or stores them locally as tarballs. """ - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - resultdir = os.path.join(outputdir, "results") + resultdir = os.path.join(self.outputdir, "results") - shutil.copyfile(os.path.join(outputdir, ".config"), + shutil.copyfile(os.path.join(self.outputdir, ".config"), os.path.join(resultdir, "config")) - shutil.copyfile(os.path.join(outputdir, "defconfig"), + shutil.copyfile(os.path.join(self.outputdir, "defconfig"), os.path.join(resultdir, "defconfig")) - shutil.copyfile(os.path.join(outputdir, "branch"), + shutil.copyfile(os.path.join(self.outputdir, "branch"), os.path.join(resultdir, "branch")) def copy_if_exists(directory, src, dst=None): - if os.path.exists(os.path.join(outputdir, directory, src)): - shutil.copyfile(os.path.join(outputdir, directory, src), + if os.path.exists(os.path.join(self.outputdir, directory, src)): + shutil.copyfile(os.path.join(self.outputdir, directory, src), os.path.join(resultdir, src if dst is None else dst)) copy_if_exists("build", "build-time.log") @@ -575,7 +569,7 @@ class Builder: # Return True if the result should be rejected, False otherwise def reject_results(): lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], stdout=subprocess.PIPE).communicate()[0]).splitlines() # Reject results where qemu-user refused to build @@ -592,7 +586,7 @@ class Builder: def get_failure_reason(): # Output is a tuple (package, version), or None. lastlines = decode_bytes(subprocess.Popen( - ["tail", "-n", "3", os.path.join(outputdir, "logfile")], + ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")], stdout=subprocess.PIPE).communicate()[0]).splitlines() regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') @@ -609,14 +603,14 @@ class Builder: def extract_last_500_lines(): subprocess.call(["tail -500 %s > %s" % \ - (os.path.join(outputdir, "logfile"), resultfile)], + (os.path.join(self.outputdir, "logfile"), resultfile)], shell=True) reason = get_failure_reason() if not reason: extract_last_500_lines() else: - f = open(os.path.join(outputdir, "logfile"), 'r') + f = open(os.path.join(self.outputdir, "logfile"), 'r') mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) mf.seek(0) # Search for first action on the failed package @@ -640,7 +634,7 @@ class Builder: if not reason: return - srcroot = os.path.join(outputdir, "build", '-'.join(reason)) + srcroot = os.path.join(self.outputdir, "build", '-'.join(reason)) destroot = os.path.join(resultdir, '-'.join(reason)) config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', 'CMakeOutput.log') @@ -671,7 +665,7 @@ class Builder: # Yes, shutil.make_archive() would be nice, but it doesn't exist # in Python 2.6. ret = subprocess.call(["tar", "cjf", "results.tar.bz2", "results"], - cwd=outputdir, stdout=self.log, stderr=self.log) + cwd=self.outputdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not make results tarball") sys.exit(1) @@ -683,7 +677,7 @@ class Builder: ret = subprocess.call(["curl", "-u", "%s:%s" % (self.http_login, self.http_password), "-H", "Expect:", - "-F", "uploadedfile=@%s" % os.path.join(outputdir, "results.tar.bz2"), + "-F", "uploadedfile=@%s" % os.path.join(self.outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", self.http_url], stdout=self.log, stderr=self.log) @@ -693,10 +687,10 @@ class Builder: log_write(self.log, "INFO: results were submitted successfully") else: # No http login/password, keep tarballs locally - with open(os.path.join(outputdir, "results.tar.bz2"), 'rb') as f: + with open(os.path.join(self.outputdir, "results.tar.bz2"), 'rb') as f: sha1 = hashlib.sha1(f.read()).hexdigest() resultfilename = "instance-%d-%s.tar.bz2" % (self.instance, sha1) - os.rename(os.path.join(outputdir, "results.tar.bz2"), resultfilename) + os.rename(os.path.join(self.outputdir, "results.tar.bz2"), resultfilename) log_write(self.log, "INFO: results saved as %s" % resultfilename) def run_instance(self): @@ -728,8 +722,7 @@ class Builder: continue # Check if the build test is supposed to be a reproducible test - outputdir = os.path.abspath(os.path.join(self.idir, "output")) - with open(os.path.join(outputdir, ".config"), "r") as fconf: + with open(os.path.join(self.outputdir, ".config"), "r") as fconf: reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() if reproducible: ret = self.do_reproducible_build() From patchwork Thu Aug 1 02:46:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140217 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="J+WNfgSh"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTZ4G5lz9sMr for ; Thu, 1 Aug 2019 12:47:46 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0B04C226AF; Thu, 1 Aug 2019 02:47:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 60QlskeRSbod; Thu, 1 Aug 2019 02:47:42 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 3B93C226B7; Thu, 1 Aug 2019 02:47:42 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 5CD5E1BF59D for ; Thu, 1 Aug 2019 02:47:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 57FDB86BA4 for ; Thu, 1 Aug 2019 02:47:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WtPQ37666osg for ; Thu, 1 Aug 2019 02:47:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by whitealder.osuosl.org (Postfix) with ESMTPS id C6AA486B94 for ; Thu, 1 Aug 2019 02:47:39 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id f17so29051238pfn.6 for ; Wed, 31 Jul 2019 19:47:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=h3MHZ4Z7iYwb1SSYjW/0XYW9hLEZvlPXuLL3ZF+QVs4=; b=J+WNfgShrVYvTxKwUPaQaw1jrJI5wgzlIffb5SiQdodoUfSZ9r+CRWqzBUP1NHLsuo 5SZmdsEmgAU8DqwUNhs/TQcKdT/AOjhi4e9r4KRAs9Dv7UHnU3/0SSOLfOeKNfXZ41ib fz+GazXw0I+yzPZZ7jGIRQpGoDRTii2jmhyqy76285IPPqGWtMrj8riQb74KWnkc+Swf TFdfXQBfyxGjdog90gqqRM3g2PrqYQ4KxXH/k4Utdj4MxgJzwpahHLMsaMcrzyw7eptT C34Z77PuYRvxL/krL5XJtiLg+TG2OBv0zO4WLDxtwrFwVOBUARKaO0f+31xk2J22wWF5 1URQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=h3MHZ4Z7iYwb1SSYjW/0XYW9hLEZvlPXuLL3ZF+QVs4=; b=FISia7l5hfwXsXndm+IbS/ilc25NDW8PIAXNyEzYTJ3kD28BsN/PWgd3Z9+0+ml0/t jdlAHh6l6zI3dVnPHyTE2AfXo7J3f8qZhcJf/bytReWZsvazrHK09M58GwnzDE/2AWMf RN4hxlhEUuTPKAWRPwvLq63rqSRPafjyHGxGCuDzID/V4ATK5/wwgO9woSQXf+sleknI bqpuaFNb80zY5AZiQAAcNRbJHGMFFCidcOz23Lo14RJaz10LgtsR4LlQmziwJb44TvKL bMw4pP74Mf1lEL6sPNFSFO3zGpnjVC2+Irehor+hLjZJkq2AieWzoeYcO9shnSqBX/yA 4AeA== X-Gm-Message-State: APjAAAUa6aGtDEcicuXaowgpj+ypupnJK7LlF78/yl9D9rHP6aSMYNOS CjBCJLs3I87Tv+tRVOPnXMH8mw8wjww= X-Google-Smtp-Source: APXvYqzTIcnnB8lffUaq2ParjwPXDhJSSp+LO02T/UAtpLDO66D8ereEhHBMLzrCCYzJYQ+5w5chGg== X-Received: by 2002:a63:d941:: with SMTP id e1mr82536589pgj.75.1564627658964; Wed, 31 Jul 2019 19:47:38 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.36 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:38 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:33 +0530 Message-Id: <20190801024643.11024-21-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 20/30] autobuild-run: define results directory as part of Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele Acked-by: Arnout Vandecappelle (Essensium/Mind) --- scripts/autobuild-run | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index e225211..8a0bec7 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -299,6 +299,7 @@ class Builder: # Buildroot sources, but to the location of the autobuilder # script. self.outputdir = os.path.abspath(os.path.join(self.idir, "output")) + self.resultdir = os.path.join(self.outputdir, "results") if self.debug: self.log = sys.stdout @@ -542,19 +543,17 @@ class Builder: are available) or stores them locally as tarballs. """ - resultdir = os.path.join(self.outputdir, "results") - shutil.copyfile(os.path.join(self.outputdir, ".config"), - os.path.join(resultdir, "config")) + os.path.join(self.resultdir, "config")) shutil.copyfile(os.path.join(self.outputdir, "defconfig"), - os.path.join(resultdir, "defconfig")) + os.path.join(self.resultdir, "defconfig")) shutil.copyfile(os.path.join(self.outputdir, "branch"), - os.path.join(resultdir, "branch")) + os.path.join(self.resultdir, "branch")) def copy_if_exists(directory, src, dst=None): if os.path.exists(os.path.join(self.outputdir, directory, src)): shutil.copyfile(os.path.join(self.outputdir, directory, src), - os.path.join(resultdir, src if dst is None else dst)) + os.path.join(self.resultdir, src if dst is None else dst)) copy_if_exists("build", "build-time.log") copy_if_exists("build", "packages-file-list.txt") @@ -563,7 +562,7 @@ class Builder: copy_if_exists("legal-info", "manifest.csv", "licenses-manifest.csv") subprocess.call(["git log -n 1 --pretty=format:%%H > %s" % \ - os.path.join(resultdir, "gitid")], + os.path.join(self.resultdir, "gitid")], shell=True, cwd=self.srcdir) # Return True if the result should be rejected, False otherwise @@ -625,7 +624,7 @@ class Builder: mf.close() f.close() - extract_end_log(os.path.join(resultdir, "build-end.log")) + extract_end_log(os.path.join(self.resultdir, "build-end.log")) def copy_config_log_files(): """Recursively copy any config.log files from the failing package""" @@ -635,7 +634,7 @@ class Builder: return srcroot = os.path.join(self.outputdir, "build", '-'.join(reason)) - destroot = os.path.join(resultdir, '-'.join(reason)) + destroot = os.path.join(self.resultdir, '-'.join(reason)) config_files = ('config.log', 'CMakeCache.txt', 'CMakeError.log', 'CMakeOutput.log') @@ -650,7 +649,7 @@ class Builder: copy_config_log_files() - resultf = open(os.path.join(resultdir, "status"), "w+") + resultf = open(os.path.join(self.resultdir, "status"), "w+") if result == 0: resultf.write("OK") elif result == -1: @@ -659,7 +658,7 @@ class Builder: resultf.write("TIMEOUT") resultf.close() - with open(os.path.join(resultdir, "submitter"), "w+") as submitterf: + with open(os.path.join(self.resultdir, "submitter"), "w+") as submitterf: submitterf.write(self.submitter) # Yes, shutil.make_archive() would be nice, but it doesn't exist @@ -713,8 +712,7 @@ class Builder: if ret != 0: continue - resultdir = os.path.join(self.idir, "output", "results") - os.mkdir(resultdir) + os.mkdir(self.resultdir) ret = self.gen_config() if ret != 0: From patchwork Thu Aug 1 02:46:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140218 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="bLbOcHIJ"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTb274vz9sNF for ; Thu, 1 Aug 2019 12:47:47 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 9BEA486CB4; Thu, 1 Aug 2019 02:47:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HtCRo3Inx9n1; Thu, 1 Aug 2019 02:47:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 4B13386BDB; Thu, 1 Aug 2019 02:47:44 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 88A981BF59D for ; Thu, 1 Aug 2019 02:47:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 859B185FD5 for ; Thu, 1 Aug 2019 02:47:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DMbFPhKDcUy0 for ; Thu, 1 Aug 2019 02:47:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 19DBE85FB9 for ; Thu, 1 Aug 2019 02:47:42 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id y15so33054054pfn.5 for ; Wed, 31 Jul 2019 19:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=o++Q7ln/GUoIHHganxrehuyBNxr07J6LxIzsYLfcSCA=; b=bLbOcHIJMdDz71FahqADayYlY2TbbzN08/nDPHCaO+fzz1SIZfdLdh2rn9gbdEzdhN wijFfkp+/fG/X9kmpHo70N04luG1NjtT3vSu+j/mOEm6cDjX/zR3TiQZrH4w2pGGuJUU BNhue+Z1G2qI5l4Jd298gM2jof/8fyuJX9jfb38dFUM1p7wGKgJ2rwp16+EPEcnr5xnV KeRWSm3z7/aXzrkEFvWB20hA+NIM6G7JqbVHaAhjHRIWstGvfgwQJwaS69Zd2poVuagm JLnMP/ABpU+sV839L8c0cHjnYkZdAhqYSRsq7YLXcXzi4bYrKfMUK+yoChsGnoT7lg0Z LaIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=o++Q7ln/GUoIHHganxrehuyBNxr07J6LxIzsYLfcSCA=; b=FoTT9fhwxKa1/gMnwKmxemVHwnI2QaIhUzdNy1SPAEL7HcRzTWMYFp+UDWeYZ4nN5u Qk923HEwMa+6ydE2gioTqcymmX+p4x7EDQ7CHifuq8WJwHLt2RFgIMJ781Sqo1D+WO8R mMVJBf61SAPiawfNxLEbNWMLTtoNS2+0wwFC/8EpH5Y7DYhHMfooBhxx+Uuu1aPcapL8 whC0Ryqb8jCEvlSBNHEcIinRYSopS09PcHqmny1HJlqbWmCur73Fa6Z5vzN5DttdRcR9 /rXHpRS3dxtcEPcULUkMmYQgtIJ2CHFHZIdQKb4QsOMb6zc2x3B4qfFXjoaDIFayekH8 Zdzg== X-Gm-Message-State: APjAAAU/tlAKHApq8mHHzNHcQSnPgbVix9XTclG1T9CFut1x0NpLXy3T 83uNYFwsdaH18FBUGJkiIM3Uz+g3Iwg= X-Google-Smtp-Source: APXvYqxnCfD0/xIXNqV+GpqfgNFj36AOFicq7AbckuggteoAuUexm9ruGMEeNmknvl2aSuEjfrseJA== X-Received: by 2002:a17:90a:bd0b:: with SMTP id y11mr5717006pjr.141.1564627661324; Wed, 31 Jul 2019 19:47:41 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.39 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:41 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:34 +0530 Message-Id: <20190801024643.11024-22-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 21/30] autobuild-run: move check_version() to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Upcoming patch from Thomas needs check_version() to use http_url. As that is inside Builder class and check_version() is only used inside Builder class, move it inside the class. Also remove the redundant call to check_version() from main(). We already call check_version() in run_instance(). Also the fact that we cannot call check_version() in main() since the builder class object is not defined at that point. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - New patch --- scripts/autobuild-run | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 8a0bec7..391c1dc 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -178,13 +178,6 @@ def log_write(logf, msg): logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg)) logf.flush() -def check_version(): - with urlopen_closing('http://autobuild.buildroot.org/version') as r: - version = int(decode_bytes(r.readline()).strip()) - if version > VERSION: - print("ERROR: script version too old, please upgrade.") - sys.exit(1) - def get_branch(): """Returns the branch that should be built. It fetches a CSV file from autobuild.buildroot.org that provides the list of branches to test @@ -306,6 +299,13 @@ class Builder: else: self.log = open(os.path.join(self.idir, "instance.log"), "a+") + def check_version(self): + with urlopen_closing('http://autobuild.buildroot.org/version') as r: + version = int(decode_bytes(r.readline()).strip()) + if version > VERSION: + print("ERROR: script version too old, please upgrade.") + sys.exit(1) + def prepare_build(self): """Prepare for the next build of the specified instance @@ -706,7 +706,7 @@ class Builder: log_write(self.log, "INFO: instance started") while True: - check_version() + self.check_version() ret = self.prepare_build() if ret != 0: @@ -765,7 +765,6 @@ def main(): # showing error messages in another language. os.environ['LC_ALL'] = 'C' - check_version() sysinfo = SystemInfo() args = docopt.docopt(doc, version=VERSION) From patchwork Thu Aug 1 02:46:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140219 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="EOHC1Jyi"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTh3yyXz9sMr for ; Thu, 1 Aug 2019 12:47:52 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 11C8F22703; Thu, 1 Aug 2019 02:47:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QUsu365dMI7o; Thu, 1 Aug 2019 02:47:46 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 8646D22735; Thu, 1 Aug 2019 02:47:46 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 53F781BF59D for ; Thu, 1 Aug 2019 02:47:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4F48787E65 for ; Thu, 1 Aug 2019 02:47:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dx7Khpko10lj for ; Thu, 1 Aug 2019 02:47:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by hemlock.osuosl.org (Postfix) with ESMTPS id B505587E8E for ; Thu, 1 Aug 2019 02:47:44 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id f17so29051345pfn.6 for ; Wed, 31 Jul 2019 19:47:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jv7WWoFIpG1b6byU6+O93J/WQpVnC24lvDCFz3bL/og=; b=EOHC1JyiMVf/3CRxFEuS3wjPBl9B+u478JmtHoJMJwE0dpQ2L31w3kzc4tD1gU3FlN ozpkAfvLiOTqMUp9536Bmm4+/cbLyqWCkqGzV5B+kPVAfwT+sEHYEUaQvg5hjbI4cKeZ z8yBW8bj3eJri4PsGT5R0RGakxUw/SFK6bombljTDWoiAMcclivx+TMWeqSATxl3Fpfk /O5+j1zU8oAmQBycJU/PN9T5IhOfDps0A0vbILv288KoTj7MB5U2eAiPxrNtqE6Ioy41 /MGwAZ+1ZPBRYzmp1BFOCiUOJVeiu57zPzfxbey2LV32Su+/wqKTEIyYLVV/lkan8pPF R1Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jv7WWoFIpG1b6byU6+O93J/WQpVnC24lvDCFz3bL/og=; b=Yg+VoRam9rHvu/XNQBpDfhM8WEOn+obHHEDJ8Jxk+eJl4miwyW7bJMN+p8sKADWdq9 ar2x3ImgCD41gLBzOABEEr8k8K8nZ4RbuydJl5iGbapKhlAT+WATxlv0Od3x8D7Yv6eo A58L/V5ZLzstcYR2y7DKEOHny6XfYwgycR9kSOjKjkKpkKgfSZBK8liKJqk7wHaLIZWc BQraKdIbOK6wRsvKFUG0F3Ch1hixdohI/1rFTQQxckTJ0/gpU2/KfxtScMfi2ewjDOJq 9D0vtOoaF1vmy1Qkr/T26e/YleKCxzGYOotxexid4Uxw3CieACVPg2XcqdnhJ0ZLsE1l go4A== X-Gm-Message-State: APjAAAXhS09zaJkQKMKP9R2aO7V4ZiGYAKiE1yFHFLh85RLXjE24uya5 DAj7HNlpEmX6kPH5VhHlX2WY5lApVnQ= X-Google-Smtp-Source: APXvYqz9fxD4zVhaSZIkrP35L8iak2Mcb4Ng2KET8pMRilZvLm8iw27DE2TTPNCfm8k31kLg9khh1Q== X-Received: by 2002:a65:6256:: with SMTP id q22mr116304882pgv.408.1564627663619; Wed, 31 Jul 2019 19:47:43 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.41 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:43 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:35 +0530 Message-Id: <20190801024643.11024-23-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 22/30] autobuild-run: move get_branch() to Builder class X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" get_branch() requires http_url and is only called inside Builder class. Moving it to builder class makes http_url available to it and makes the Builder class cohesive. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - New patch --- scripts/autobuild-run | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 391c1dc..ccfa9c4 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -178,23 +178,6 @@ def log_write(logf, msg): logf.write("[%s] %s\n" % (strftime("%a, %d %b %Y %H:%M:%S", localtime()), msg)) logf.flush() -def get_branch(): - """Returns the branch that should be built. It fetches a CSV file from - autobuild.buildroot.org that provides the list of branches to test - (first field) and their weight (second field). We build a list of - branches, where the branch name is repeated N times, with N being - the weight of the branch. We then pick a random branch in this - list. This way, branches with a higher weight are more likely to - be selected. - """ - with urlopen_closing('http://autobuild.buildroot.org/branches') as r: - csv_branches = r.readlines() - branches = [] - for branch in csv.reader(csv_branches): - branches += [branch[0]] * int(branch[1]) - - return branches[randint(0, len(branches) - 1)] - class SystemInfo: DEFAULT_NEEDED_PROGS = ["make", "git", "gcc"] DEFAULT_OPTIONAL_PROGS = ["bzr", "diffoscope", "java", "javac", "jar"] @@ -306,6 +289,23 @@ class Builder: print("ERROR: script version too old, please upgrade.") sys.exit(1) + def get_branch(self): + """Returns the branch that should be built. It fetches a CSV file from + autobuild.buildroot.org that provides the list of branches to test + (first field) and their weight (second field). We build a list of + branches, where the branch name is repeated N times, with N being + the weight of the branch. We then pick a random branch in this + list. This way, branches with a higher weight are more likely to + be selected. + """ + with urlopen_closing('http://autobuild.buildroot.org/branches') as r: + csv_branches = r.readlines() + branches = [] + for branch in csv.reader(csv_branches): + branches += [branch[0]] * int(branch[1]) + + return branches[randint(0, len(branches) - 1)] + def prepare_build(self): """Prepare for the next build of the specified instance @@ -346,7 +346,7 @@ class Builder: os.path.relpath(f, self.dldir)) os.remove(f) - branch = get_branch() + branch = self.get_branch() log_write(self.log, "INFO: testing branch '%s'" % branch) # Clone Buildroot. This only happens if the source directory From patchwork Thu Aug 1 02:46:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140223 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="cPczG1c+"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTw4Rkxz9sMr for ; Thu, 1 Aug 2019 12:48:04 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id E4B0686004; Thu, 1 Aug 2019 02:48:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8AaSOLOxfZWx; Thu, 1 Aug 2019 02:47:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id ABB8185FDB; Thu, 1 Aug 2019 02:47:52 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id B41591BF59D for ; Thu, 1 Aug 2019 02:47:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A4A8A22731 for ; Thu, 1 Aug 2019 02:47:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id T7pWl2OM02qC for ; Thu, 1 Aug 2019 02:47:47 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f195.google.com (mail-pg1-f195.google.com [209.85.215.195]) by silver.osuosl.org (Postfix) with ESMTPS id 286912274A for ; Thu, 1 Aug 2019 02:47:47 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id u17so33183880pgi.6 for ; Wed, 31 Jul 2019 19:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Qa9i9ZHSAtB4trGEnxaO3r8utSZeromPfcjGLjBeKJk=; b=cPczG1c+Cs2kWfLVysmYC3lWxYtJfsU5H9cR8VJaJUV8UCrEp782E7JBiYmpAoD/wJ PwiXWTKUesjZ5yOgRBg4nvSH6Ulos1HcwCEZ2aZni9h0D/QB+5qxJE5YMOQEYW3ownmS VjAoGrRqp0wG2f2pkBZbj9+qyowPbfNss1qUCxr0K6cW+VHthi4IZCELdc4M9kDDiCjS k7UgdlpfG+2+AqGf2qP9pOXxxpbHeGS5+FNSlEu4OgecVKAdO19RyRatlPzQNMBprk+R xI99FVtR371LnUMLOtpHAX1O8ogYtfOB9HfPMQ26M6sezoBrLLU8ZNq6S413ZKuTrsuz Y9RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Qa9i9ZHSAtB4trGEnxaO3r8utSZeromPfcjGLjBeKJk=; b=S8RJ6A96jICo7BXA4v+J43pAECFsO5tA2ONHZiqol8dhK0wYgbyCgWdW2dJra4vR7o uaKunlZ6FkRcF92LUPVLPsCD7waRsNBjvfEPzpS2JtzOmn2dOze0SZ5w3j5IDVGNcOVa qH+SeMQl51EE/sOvQJ2HE2HbJ9uk9EGU+lhbfvt1awmD3gSC9cg6ReSIEvi6nGAgsLC5 lWiDVHwp7NsPOqu86at4j0JCy+1RZsesS2UILydJpYkvf9rYw4WiQypJDCUh3lM1MZaU 5yc8mcE4PK7doS0YWzDADvzKHwUQeTMJyynriaUFXWb8l5CbboE+MreP8dz19mLXxaB0 7Myw== X-Gm-Message-State: APjAAAWRA4I86oNtjxw59huEj5tVoK2p9qyPEvJW2r0lkYxuk53Oyh0v iBNhSwUJpknyH0sov2BwhBP9bhKYtJY= X-Google-Smtp-Source: APXvYqwrmJaVs+uC2uvmim4q53QHN6FYHJoVM0wkd7URB00G5tqbazoEwa7LEqwNRptSwRGQB6sL+g== X-Received: by 2002:a65:47c1:: with SMTP id f1mr114984487pgs.169.1564627666313; Wed, 31 Jul 2019 19:47:46 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.44 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:46 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:36 +0530 Message-Id: <20190801024643.11024-24-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 23/30] scripts/autobuild-run: make the HTTP URL really configurable X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas Petazzoni The --http-url option allowed to customize the URL at which build results are submitted. However, there were two other places in the script where autobuild.buildroot.org was hardcoded: when checking the script version, and when getting the list of branches to build. This commit changes the --http-url to be the base URL of the autobuild server, and it is used everywhere instead of hardcoding autobuild.buildroot.org. Note: users of autobuild-run that were passing a custom --http-url option such as http://foo.com/submit/ should change it to just http://foo.com/ Signed-off-by: Thomas Petazzoni [Atharva: Adapted Thomas' patches to work with Builder class] Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Incorporate Thomas' patch into Builder class series - Move patch for appropriate order of patches --- scripts/autobuild-run | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index ccfa9c4..678bac5 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -66,7 +66,7 @@ defaults = { '--make-opts': '', '--nice': 0, '--pid-file': '/tmp/buildroot-autobuild.pid', - '--http-url': 'http://autobuild.buildroot.org/submit/', + '--http-url': 'http://autobuild.buildroot.org/', '--toolchains-csv': 'support/config-fragments/autobuild/toolchain-configs.csv', '--repo': 'https://github.com/buildroot/buildroot.git', } @@ -145,6 +145,7 @@ from distutils.version import StrictVersion import platform from threading import Thread, Event import datetime +import urlparse if sys.hexversion >= 0x3000000: import configparser @@ -283,7 +284,7 @@ class Builder: self.log = open(os.path.join(self.idir, "instance.log"), "a+") def check_version(self): - with urlopen_closing('http://autobuild.buildroot.org/version') as r: + with urlopen_closing(urlparse.urljoin(self.http_url, 'version')) as r: version = int(decode_bytes(r.readline()).strip()) if version > VERSION: print("ERROR: script version too old, please upgrade.") @@ -291,14 +292,14 @@ class Builder: def get_branch(self): """Returns the branch that should be built. It fetches a CSV file from - autobuild.buildroot.org that provides the list of branches to test + the autobuild server that provides the list of branches to test (first field) and their weight (second field). We build a list of branches, where the branch name is repeated N times, with N being the weight of the branch. We then pick a random branch in this list. This way, branches with a higher weight are more likely to be selected. """ - with urlopen_closing('http://autobuild.buildroot.org/branches') as r: + with urlopen_closing(urlparse.urljoin(self.http_url, 'branches')) as r: csv_branches = r.readlines() branches = [] for branch in csv.reader(csv_branches): @@ -678,7 +679,7 @@ class Builder: "-H", "Expect:", "-F", "uploadedfile=@%s" % os.path.join(self.outputdir, "results.tar.bz2"), "-F", "uploadsubmit=1", - self.http_url], + urlparse.urljoin(self.http_url, 'submit/')], stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "INFO: results could not be submitted, %d" % ret) From patchwork Thu Aug 1 02:46:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140220 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="k89f/IKJ"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTj4YzJz9sNF for ; Thu, 1 Aug 2019 12:47:53 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E6D7487E9B; Thu, 1 Aug 2019 02:47:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vZW23hzhDymt; Thu, 1 Aug 2019 02:47:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 4F8C687E9C; Thu, 1 Aug 2019 02:47:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 00E101BF59D for ; Thu, 1 Aug 2019 02:47:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id EEB4187E9C for ; Thu, 1 Aug 2019 02:47:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cC7i7FMkiamy for ; Thu, 1 Aug 2019 02:47:49 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 7FB1987E91 for ; Thu, 1 Aug 2019 02:47:49 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id q10so33031016pff.9 for ; Wed, 31 Jul 2019 19:47:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lNHBTMFIGwb9H7hNx4sCqafyG+kux1lxRqUDS1oIKtQ=; b=k89f/IKJsNBzt6LWz2Rg0xckHB3N9FqxA04D+7O1AdpzG7pTfNCcTek4riXV8BhR4o j1z0iXknixGIDr4ZaOJRBkMu0xmzC3CetvkdzsTZ2igCVHvAji2sxGS4N1Y1jvxtRXE1 sxoUPmcLW2W+iWjK5bp1NQ39GR1R3bAJY14OvVg3Ce0PBHcfFvFQ5IE8VhsR1Wox49Py unCoIm0R9KoshSyIhVCD/LGZhazCPRQBxQDh1CIRZ9QWx0If8AuhOi6tDr/pyKoBNVfr kefpsgdxZY5DOqw9oKrtoNR7lifhEk6Q/cfKjFb5i5QaB4W7zdH8z9Ywrv7dTPU1tZcb NMaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lNHBTMFIGwb9H7hNx4sCqafyG+kux1lxRqUDS1oIKtQ=; b=joAsuOimkBC4Ev4pVrk8Z2CTdgAw7/4/ljmqcHKLEzqd2MoO+2bRd7a4w0P8tXt5WL YLpZdvPb5PB1tLwmJYFrWwtyARxWB5x5eC6hGCFxf8nmefo5oJZ+Favih9vGmpkdnkeG 0LHOUn2W4497UfaAa9F6G51eIHNjb6rSEDqk05EFYKYrKQ1d5ujuGtYLCBCYaSTHoj2d Huk//DlzxRf0wnXbKiVu1FdsKhQzoTocLTuD5kqFZDIIi+EpQ56zZsCE/3bpJCtTEgp3 ZZxl4Sf3XfGD86gI6wmUSHmkBs8aFPw92GuxmLeSsQo3BNi49VpYrgOid3OVZwAW2FJi LwGg== X-Gm-Message-State: APjAAAUGVRuvtLi8p+CRCmbNA+TxJW3aNm8K6DttlmYgxeMn14+rH9hW rnxGoyLhXSvm5Ig/F+I5EQXavEVvWtQ= X-Google-Smtp-Source: APXvYqyphayJQMkuN0OPzMZcdTjkEj55/cyPFyZTJq9eER8edssIv5/CIHY69JCzrVSZTo6iFs/+ZQ== X-Received: by 2002:a17:90a:20a2:: with SMTP id f31mr5962886pjg.90.1564627668710; Wed, 31 Jul 2019 19:47:48 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.46 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:48 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:37 +0530 Message-Id: <20190801024643.11024-25-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 24/30] autobuild-run: create reason file on build failures X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" When a build fails, we calculate the reason of failure on the server side as well as client side. To solve this redundancy we only calculate on the client side and submit a file which contains the reason of failure. Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes from reason-file-v2 series: - Move to builder-class series --- scripts/autobuild-run | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 678bac5..8f60e89 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -598,6 +598,11 @@ class Builder: # not found return None + reason = get_failure_reason() + if reason: + with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf: + reasonf.write("-".join(reason)) + def extract_end_log(resultfile): """Save the last part of the build log, starting from the failed package""" @@ -606,7 +611,6 @@ class Builder: (os.path.join(self.outputdir, "logfile"), resultfile)], shell=True) - reason = get_failure_reason() if not reason: extract_last_500_lines() else: @@ -630,7 +634,6 @@ class Builder: def copy_config_log_files(): """Recursively copy any config.log files from the failing package""" - reason = get_failure_reason() if not reason: return From patchwork Thu Aug 1 02:46:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140221 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="MrBzxz/1"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTm5VZWz9sMr for ; Thu, 1 Aug 2019 12:47:56 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 454A687E8A; Thu, 1 Aug 2019 02:47:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j1j3HWvYpXvX; Thu, 1 Aug 2019 02:47:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 70ECA87E97; Thu, 1 Aug 2019 02:47:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 1D9DA1BF59D for ; Thu, 1 Aug 2019 02:47:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1976187EA1 for ; Thu, 1 Aug 2019 02:47:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EIVKy42W6H8B for ; Thu, 1 Aug 2019 02:47:52 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by hemlock.osuosl.org (Postfix) with ESMTPS id D1E5487EA0 for ; Thu, 1 Aug 2019 02:47:51 +0000 (UTC) Received: by mail-pf1-f172.google.com with SMTP id i189so33048413pfg.10 for ; Wed, 31 Jul 2019 19:47:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Zk2FiuLaLs/di/XgoD6jTxFuScCyTPRxSsKeZt9wdRg=; b=MrBzxz/1q4uIrn8q19jsmbD+dAXfyU1QHmtCGcmF1un2cND9EufTWjLuHxAbyVMFKk Wvlvy47hZmR1YLr4aESAorJ/vSIXXWa78rTt3Gk/wOW5FdrJiv7fHh2/Mc3cqUSIRXVs 6Eixv0bKZiaLHtRHXttei3q5pJm6nbbLdK0kyIZyaQa5FehSwa3RyRA6/zcCJLOB4FuI OBBNAp6u2OprePGHFCMX/aTjuZl369a7EyfAVesGhlbSvuhzCWeEeWjt03SNwyfXCdJV cYCkz+YMO/sDJpcJ+rnG3vYyGAzdd0CiuJcJ2q5LLTgS7UmqUAbs3McgD5j5HDQWx/CM JKpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Zk2FiuLaLs/di/XgoD6jTxFuScCyTPRxSsKeZt9wdRg=; b=imb8zKkiOs8qc/OzpwHSpaAIfLgGN1AD37s0GVWN6FZG6PVXfoleSd2jKI3jXW6dr5 iVMtctKO0plO9zyoGJGTGW7CWrBZqSTPP8I1IDY+fgA9FOQsHiQqJJGZhcHBsixOpzX2 yU0Zih0gwsEKOH3J3lNYNet4VBePFhP0q7Tj2oU/KO0eHPpb7mLdXGKlOas/RNeLAipc OUuw8KhBWu2skoVLghRfTAANT4R0JTZD3km8++B66aBT7mT4mk/skEJVWtPL2nGZrpDI 32Ef3QpSVLFheTVb0DUt/swj3vrNFExaFO1ZIzlo5k7DAf6ATW/tFHnCs/9xZ1mLBUET diSA== X-Gm-Message-State: APjAAAU7ohgykBEwErLRR5+S1ymiqlevbHnWADiSs0Z/7CNaWjpMf8K4 nkCg+wbG3bj96qTV54rHcAFOz1ovQi0= X-Google-Smtp-Source: APXvYqwSaBC1eigFYZgn9s81tRYi5+3P+GjzgQ0c7qdJVdBAPQoACC/dhQDzb8i532ynO4NFhBFkoA== X-Received: by 2002:a17:90a:3247:: with SMTP id k65mr5998461pjb.49.1564627671102; Wed, 31 Jul 2019 19:47:51 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.48 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:50 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:38 +0530 Message-Id: <20190801024643.11024-26-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 25/30] web/import.inc.php: support reading failure reason from reason file X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- NOTE: This patch is untested. Changes from reason-file-v2 series: - Move to builder-class series --- web/import.inc.php | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/web/import.inc.php b/web/import.inc.php index a9b368f..6f878e0 100644 --- a/web/import.inc.php +++ b/web/import.inc.php @@ -230,17 +230,21 @@ function import_result($buildid, $filename) if ($status == 0) $reason = "none"; else { - $tmp = Array(); - exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp); - if (trim($tmp[0])) - $reason = $tmp[0]; - else { - exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp); - if (trim($tmp[0])) - $reason = trim($tmp[0]); - else - $reason = "unknown"; - } + if (file_exists($thisbuildfinaldir . "reason")) + $reason = trim(file_get_contents($thisbuildfinaldir . "reason", "r")); + else { + $tmp = Array(); + exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp); + if (trim($tmp[0])) + $reason = $tmp[0]; + else { + exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp); + if (trim($tmp[0])) + $reason = trim($tmp[0]); + else + $reason = "unknown"; + } + } } /* Compress files that are typically too large and infrequently From patchwork Thu Aug 1 02:46:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140222 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="V/UBrzAh"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTs4t9Nz9sMr for ; Thu, 1 Aug 2019 12:48:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 07E0D226CF; Thu, 1 Aug 2019 02:48:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dUNo2R-6gRJY; Thu, 1 Aug 2019 02:47:57 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 4BE4D226D4; Thu, 1 Aug 2019 02:47:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id AE0841BF59D for ; Thu, 1 Aug 2019 02:47:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id AB4EC87EA0 for ; Thu, 1 Aug 2019 02:47:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id k+CHzGoJApiY for ; Thu, 1 Aug 2019 02:47:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f196.google.com (mail-pg1-f196.google.com [209.85.215.196]) by hemlock.osuosl.org (Postfix) with ESMTPS id 27D9887E7A for ; Thu, 1 Aug 2019 02:47:54 +0000 (UTC) Received: by mail-pg1-f196.google.com with SMTP id d1so364460pgp.4 for ; Wed, 31 Jul 2019 19:47:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fH3RyXTiLu45+Y7g6upszF37w6Dsr+UHPGvum6frPTo=; b=V/UBrzAhi/EoMbDatZjsMC3MgdKAWZF6bLPhlXI8RD+b7oW4Cl09eJGHNf09pC/ZQG s63nrz0tdt5bp2tf4LEUQPfUoZya5zWVy6AnSf6cgi5fGqOC9zvu+mDEIGdurIK5O/+2 kp6MQ4tN3O8o7EnCweB6iLulKjSfJoqEEEZFsESnbh4ur7/ujfuoHf5baHGJ4KQc/r7F R6XpJp4yhfVXtC7wHG4gNdvgnSBTaTp3G4N7QzPbcV/+gA1DxxPGXgvhEE129OQJOcXg KBR5iWQyBwZagMBZ73TreEIftPIHFc1fWPjBAXADS9HJb9vS/8RpqA4lOnTSUdXTmbJ5 +ZbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fH3RyXTiLu45+Y7g6upszF37w6Dsr+UHPGvum6frPTo=; b=WL+WfFX8JaP/nlvJO+q0p+k/woEAnp5MYMurO0iaAF/4ul5uGOBx1obVNpEwOlO/Zy /vngT+hm7GJwGeWslq4/HM1SU/nu5jFYpIXHeqtC6yQhhauCdwqgTJAZ6o4xBCXR9j3F r5s4uaMU6mtwkDw6kVzlzXEs8q41p/OkUfL+uG5gaRNTKPQciKOV5N71CQmX2FJ8cty6 acdf9ADhT7LHSvCkWw/ydO4Ogy9gvvDRLLuz4C0za5Dx+Z8frHtWQitEss6VWXIAmGWQ JLRMnqH8pFKSmBYDPCp1lBh29uGvAiIcz5Jr1V6jYxx3C9S0p3y+O0TVZEqh0Qf/Ka0d DR2Q== X-Gm-Message-State: APjAAAXIUdBNrU36lkbb33vrWmTUg0UDSaSua6roLYJtc0lRhD9+OZkx NJidW4vJ3wNpx7De+2uEexfQu7NVcL0= X-Google-Smtp-Source: APXvYqwaSOeNwp3Dwf0vk9tG/R/BnMy9RkIziexB85FtpymoNX7DvDog6OWn5tuXagIp0VNfBqhXVA== X-Received: by 2002:a17:90a:ba93:: with SMTP id t19mr5883961pjr.139.1564627673385; Wed, 31 Jul 2019 19:47:53 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.51 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:53 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:39 +0530 Message-Id: <20190801024643.11024-27-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 26/30] scripts/autobuild-run: support changing repo X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas Petazzoni The current logic in prepare_build() assumes that the "origin" repo never changes. However, if one regularly changes his autobuild-run configuration, switching being repository, this is not true. Currently, it requires manually wiping out the Buildroot clone in every autobuild instance when changing the repository to pull from. So instead, use: git fetch git checkout FETCH_HEAD which will easily allow switching from one repo to the other. Signed-off-by: Thomas Petazzoni [Atharva: Adapted Thomas' patches to work with Builder class] Signed-off-by: Atharva Lele Reviewed-by: Arnout Vandecappelle (Essensium/Mind) --- Changes v1 -> v2: - Incorporate Thomas' patch into Builder class series --- scripts/autobuild-run | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 8f60e89..09dd520 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -361,12 +361,12 @@ class Builder: # Update the Buildroot sources. abssrcdir = os.path.abspath(self.srcdir) - ret = subprocess.call(["git", "fetch", "origin"], cwd=abssrcdir, stdout=self.log, stderr=self.log) + ret = subprocess.call(["git", "fetch", self.repo, branch], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not fetch Buildroot sources") return -1 - ret = subprocess.call(["git", "checkout", "--detach", "origin/%s" % branch], cwd=abssrcdir, stdout=self.log, stderr=self.log) + ret = subprocess.call(["git", "checkout", "FETCH_HEAD"], cwd=abssrcdir, stdout=self.log, stderr=self.log) if ret != 0: log_write(self.log, "ERROR: could not check out Buildroot sources") return -1 From patchwork Thu Aug 1 02:46:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140224 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="YUFpcvuI"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTw5fBhz9sNF for ; Thu, 1 Aug 2019 12:48:04 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 55E1F87E68; Thu, 1 Aug 2019 02:48:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id og9IeN6iFAmG; Thu, 1 Aug 2019 02:47:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 436CC87EAE; Thu, 1 Aug 2019 02:47:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id F1FBE1BF59D for ; Thu, 1 Aug 2019 02:47:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id EC9242274A for ; Thu, 1 Aug 2019 02:47:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XyXtZh4fN07M for ; Thu, 1 Aug 2019 02:47:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by silver.osuosl.org (Postfix) with ESMTPS id 78976226A0 for ; Thu, 1 Aug 2019 02:47:56 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id r22so5578860pgk.10 for ; Wed, 31 Jul 2019 19:47:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XbdRlm2M3W+idPE9sz5KHhFCWstCTi82EKTIG6Y3Dlo=; b=YUFpcvuIJuQEGHdHUx7MJk6J/hlIKUnIRwcp2uXl0vqvRlIraqS8oDLrNK0Up1oDZc LKjm4dtTabcqIs1bwV3hmj/B82zku8Q623rf5VZKoJFZHsY4dl08we121SgHtmSUWJ3G r/a8a5m7J9qJoiYw/1bnnn8TYZUNsXxBVylzddBxfdZ3euINT+LTg4regPdm7s7kZQax z1gmpjUQ8BNZodbysHHHefZP+ie9EpupYYuqVei2xyttLOlvbmf1VvyhDrlchCa78yYH 7AAu7f8YAl/2Jhi8uwQ3I0EsPdwWaJ/RIoOSM0ytw7SznUkpXcOCBi9E8CNS8dIfkOsK SkiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XbdRlm2M3W+idPE9sz5KHhFCWstCTi82EKTIG6Y3Dlo=; b=BwE7sMDgJMsVo/A6jhcA9O0fiCD1N2U/paFAFS0T3Pil50BZxgO43TWFJak8v4Hr9X QwBlwUOy5I48T8p/7caAmyyJuhkhvEhf42rK0IhXx0YoEQNFRnuuCytk/JNKyP2a3DDt ue/ASGiH8Dn6HCjFojDGiFlSRw3eV1FzdvTEeUjOwIHFJgoIr7PuCbfmmO8n2fqNh7PX wbQsqogbT5JFiFU5yHHNNOPRvVOIQZGzRM0Ir25VDokGlipqZIWXDuSiarljYANYc0N5 cjgySA11KlD2y50zjJWhrJ9WpFQK6MfIs5QTcavARTLTYGEaVIz+xFMzfwFTjOT6dgmz gytQ== X-Gm-Message-State: APjAAAUlv298DDnl96dKFfnZfAbDshaGCNjqImcKaUEOHafX7NBQBWK6 2mP/6v2WYSI6kJyztedoaBBm1HB8qJ4= X-Google-Smtp-Source: APXvYqwyxLQokqmLp/O3AFPMspW6Avbhem6V4lPA+D60LbATudFELMSNKmxGkeGy7Cif/4DBm4UR0A== X-Received: by 2002:a17:90a:f98a:: with SMTP id cq10mr6060908pjb.43.1564627675661; Wed, 31 Jul 2019 19:47:55 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.53 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:55 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:40 +0530 Message-Id: <20190801024643.11024-28-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 27/30] autobuild-run: modify do_build() to accept outputdir as argument X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Pass outputdir to do_build() so in future patches we can pass different output directories to increase variance in case of reproducibility testing. Signed-off-by: Atharva Lele --- Changes v3: - New patch --- scripts/autobuild-run | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 09dd520..7c05234 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -454,14 +454,14 @@ class Builder: log_write(self.log, "INFO: Build is reproducible!") return 0 - def do_build(self): + def do_build(self, outputdir): """Run the build itself""" - f = open(os.path.join(self.outputdir, "logfile"), "w+") + f = open(os.path.join(outputdir, "logfile"), "w+") log_write(self.log, "INFO: build started") cmd = ["nice", "-n", str(self.nice), - "make", "O=%s" % self.outputdir, + "make", "O=%s" % outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "BR2_JLEVEL=%s" % self.njobs] \ + self.make_opts.split() @@ -473,7 +473,7 @@ class Builder: build_monitor = Thread(target=self.stop_on_build_hang, args=(monitor_thread_hung_build_flag, monitor_thread_stop_flag, - sub, self.outputdir)) + sub, outputdir)) build_monitor.daemon = True build_monitor.start() @@ -494,7 +494,7 @@ class Builder: log_write(self.log, "INFO: build failed [%d]" % ret) return -1 - cmd = ["make", "O=%s" % self.outputdir, "-C", self.srcdir, + cmd = ["make", "O=%s" % outputdir, "-C", self.srcdir, "BR2_DL_DIR=%s" % self.dldir, "legal-info"] \ + self.make_opts.split() ret = subprocess.call(cmd, stdout=f, stderr=f) @@ -513,7 +513,7 @@ class Builder: # Start the first build log_write(self.log, "INFO: Reproducible Build Test, starting build 1") - ret = self.do_build() + ret = self.do_build(self.outputdir) if ret != 0: log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret @@ -527,7 +527,7 @@ class Builder: # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") - ret = self.do_build() + ret = self.do_build(self.outputdir) if ret != 0: log_write(self.log, "INFO: build 2 failed") return ret From patchwork Thu Aug 1 02:46:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140225 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="YGPwEaab"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTx6t6kz9sNf for ; Thu, 1 Aug 2019 12:48:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 67B9E226A9; Thu, 1 Aug 2019 02:48:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Vdm-EuNhdKNS; Thu, 1 Aug 2019 02:48:01 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 5461922785; Thu, 1 Aug 2019 02:48:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 1DEF11BF59D for ; Thu, 1 Aug 2019 02:47:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 1AF8986BE0 for ; Thu, 1 Aug 2019 02:47:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nnosl0vQr9le for ; Thu, 1 Aug 2019 02:47:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id AF28286BC9 for ; Thu, 1 Aug 2019 02:47:58 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id t132so33172705pgb.9 for ; Wed, 31 Jul 2019 19:47:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Wn0p5oYi314qIumTHuF/a6f3g7Fb/bQwysuZsRrbd+0=; b=YGPwEaab1G42AVVsyLcAaVNk/UNRsj/dWfTWWggF/EZk6f49XHPPvGws9hhzv+M2rW xJ3kICENKgDEen63tRlJmDJ6c2NtpozkdxS+3KzC9v/iUmEduFkDAbDQMY6MucyHULiO +ah/wpJZcSyddN57fjXsT6SOYx8QXU0DB0Asad3IKviQtRBbx3ki+8q0H9/qK/JAFAId hbMGSzEBocjCSgp2l9SCmg+C/oXuGE/3RTTDAEgC8HBKzhLBwW+jE64vjTYPDDPYMXgQ ijBuItosIh8U+VZvuCo3h0q69uVpfDhccG2lEYmaNQXTwspxz1f/BFVO8iSDgXJ5dNQh qssw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Wn0p5oYi314qIumTHuF/a6f3g7Fb/bQwysuZsRrbd+0=; b=XxwZh7mozuZODCpFsmdT2DoKDJvPzFC1k0mvCh6tbqF4/RTp83QY4OcSc+OE0jUBIg 8ubeTunPoNv0HrEQ14pb/p606t1TAYJKYhRwpRGQFuMDvEk3zxVFmkGZD3La5FynS70b MhnZzE1G9MPo3oJy09vvceylNSh352gqhDoaqUFIJX7Kmo2CCBNtyhXD/lurmk2cobyP cRMTcJuKasp67RjAF0XvdY4Vfv3VMka6/5spxeH48qmyC/zCx8+jRpm6vxiuygBmLH1Y 6tgzoF5nZb30iE+2TtSHuid7YD1+oGsWuXJH/SOHaiCVEEMHbNnyTLIhSxyo1p4KKKqN 8eiA== X-Gm-Message-State: APjAAAXsV76wBBVRziYH9Q0jzSlGd3iyTPby1Bzy7EmcqSlJcNCv6xL+ tm+zBLnBBNCK+9nNPOQxaoCfaXWev68= X-Google-Smtp-Source: APXvYqxoDcME9tBVfpIxDEGdFhAtXDh/SPTKG+cCghsAv/FGP0l0xJ8afSd6CV4s58+u8PYwuaZS2Q== X-Received: by 2002:a63:24a:: with SMTP id 71mr7294974pgc.273.1564627677976; Wed, 31 Jul 2019 19:47:57 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.55 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:47:57 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:41 +0530 Message-Id: <20190801024643.11024-29-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 28/30] autobuild-run: define different output directory for reproducible builds X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Having different output directories will increase variance. Subsequent patches will modify other parts of the scripts to use this directory. Signed-off-by: Atharva Lele --- Changes v3: - New patch --- scripts/autobuild-run | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 7c05234..a96840f 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -278,6 +278,10 @@ class Builder: self.outputdir = os.path.abspath(os.path.join(self.idir, "output")) self.resultdir = os.path.join(self.outputdir, "results") + # We use a different output directory for more variance in reproducibility + # testing. + self.outputdir_1 = os.path.abspath(os.path.join(self.idir, "output-1")) + if self.debug: self.log = sys.stdout else: From patchwork Thu Aug 1 02:46:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140226 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="W/yYK7sx"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZTz5yWFz9sMr for ; Thu, 1 Aug 2019 12:48:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5D09387E55; Thu, 1 Aug 2019 02:48:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ssqa5ja0ZxmN; Thu, 1 Aug 2019 02:48:05 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id B89BC87E5C; Thu, 1 Aug 2019 02:48:05 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id C94F11BF59D for ; Thu, 1 Aug 2019 02:48:04 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C497322767 for ; Thu, 1 Aug 2019 02:48:04 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jaCZPGI6a122 for ; Thu, 1 Aug 2019 02:48:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by silver.osuosl.org (Postfix) with ESMTPS id 2E24C2277A for ; Thu, 1 Aug 2019 02:48:01 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id f5so24364911pgu.5 for ; Wed, 31 Jul 2019 19:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=igjM7kGcRvpkrWSL50nao0po4tx6jp+23JtJw5qRQis=; b=W/yYK7sxau4sKpkAJQ+oqIhfKqa8mFFTsAe5WfTsTqy0cR8Ze+5qPrmBHQ0nNicyxF 9UMFUOA5uQjlfS4WVF0dvB7zDFbafUEhEIpSH4E5zkgD83P0BhnrqrwRf0HpSHeaIFxS vf/NZ2AQirMeN+uHi8HCUCUDL/WkmQtnpAmPuFwzoYCoJ8mxe5/Cj2ktORwJCp5MRDYc /Cp/BhWgXnhf2LD1aYxQLSMV4El7GjPI8SjA5ugi52+ABkHAo0BA6F0PDLNYG4sJeMuh 8TR5h08LNes5gXepgQOyKnP9DUZMhanC6TYIn/D9ycgKUncFx0NN9HO+/NEPxsnmfmzX zvQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=igjM7kGcRvpkrWSL50nao0po4tx6jp+23JtJw5qRQis=; b=f1uR1w45BWJNNU2RMTevlsmDqinsVRXrgUk+vD2ceEkmlFsjBOxf9xKRpv8g3KKtHD bYpB2EXVarMm2ElUHMbcZ++kVlwXgYtL7S1WAHyiIpZWFrQyWuVr/KrXKs2qzNduLMKj gAq3Nt9r3JMzwPzDTPcC+Shp0y3HW/BjTHKpNM2b6+hEd/Z/GqdM72TESyggrOxxGUPE 9/NeK+y8Qp4vQWXZ9W6Zf9Fj/oNA/+35Q/X/FmhCPtfgH8tXLVxqX5PiIhThQXRSsMKd TsRzRTZZXqLg1NwyQ/3MvXZwULE34WnKd5ZajFq0nXt3CBz4kMEdigigPKyQ3PlKBD3h A01A== X-Gm-Message-State: APjAAAW2fdj6rKgbDwSH81DAJZLEP7MjetRnja4Qn7ncE4Pn6su1CjE+ eR2Q+dU6DHSZ6CSw687fHDjHTcJy5xI= X-Google-Smtp-Source: APXvYqzcGadmrn1gDv00/jNIvCqf6N48rgD9IblCqW54vkWcwDXXpwt7M/+C3D0QAb+z4VL1grEZow== X-Received: by 2002:a65:5cca:: with SMTP id b10mr119546491pgt.365.1564627680337; Wed, 31 Jul 2019 19:48:00 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.47.58 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:48:00 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:42 +0530 Message-Id: <20190801024643.11024-30-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 29/30] autobuild-run: use different output directories for reproducible builds testing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- Changes v3: - New patch --- scripts/autobuild-run | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index a96840f..b32cd5b 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -434,8 +434,8 @@ class Builder: reproducible_results = os.path.join(self.outputdir, "results", "reproducible_results") # Using only tar images for now - build_1_image = os.path.join(self.outputdir, "images-1", "rootfs.tar") - build_2_image = os.path.join(self.outputdir, "images", "rootfs.tar") + build_1_image = os.path.join(self.outputdir, "images", "rootfs.tar") + build_2_image = os.path.join(self.outputdir_1, "images", "rootfs.tar") with open(reproducible_results, 'w') as diff: if self.sysinfo.has("diffoscope"): @@ -522,16 +522,17 @@ class Builder: log_write(self.log, "INFO: build 1 failed, skipping build 2") return ret - # First build has been built, move files and start build 2 - os.rename(os.path.join(self.outputdir, "images"), os.path.join(self.outputdir, "images-1")) + # Create the second output directory if it does not exist + if not os.path.exists(self.outputdir_1): + os.mkdir(self.outputdir_1) - # Clean up build 1 - f = open(os.path.join(self.outputdir, "logfile"), "w+") - subprocess.call(["make", "O=%s" % self.outputdir, "-C", self.srcdir, "clean"], stdout=f, stderr=f) + # Copy .config to the other output directory + shutil.copyfile(os.path.join(self.outputdir, ".config"), + os.path.join(self.outputdir_1, ".config")) # Start the second build log_write(self.log, "INFO: Reproducible Build Test, starting build 2") - ret = self.do_build(self.outputdir) + ret = self.do_build(self.outputdir_1) if ret != 0: log_write(self.log, "INFO: build 2 failed") return ret From patchwork Thu Aug 1 02:46:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Atharva Lele X-Patchwork-Id: 1140227 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="OOJ8TxYd"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45zZV71dr5z9sMr for ; Thu, 1 Aug 2019 12:48:15 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 49B7187E76; Thu, 1 Aug 2019 02:48:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x8MLQyJ9MqKf; Thu, 1 Aug 2019 02:48:07 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id DAF5887E7D; Thu, 1 Aug 2019 02:48:07 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 7A6161BF59D for ; Thu, 1 Aug 2019 02:48:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 73E302275F for ; Thu, 1 Aug 2019 02:48:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id e97O8EuJH9Xa for ; Thu, 1 Aug 2019 02:48:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by silver.osuosl.org (Postfix) with ESMTPS id 9EBAA2274A for ; Thu, 1 Aug 2019 02:48:03 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id a93so31366669pla.7 for ; Wed, 31 Jul 2019 19:48:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lFEkal5RWVtSpZcGOvMDSOh9991br9vxHdhHcAmHv40=; b=OOJ8TxYd/Jq2OhNFYN0HANx8et/Lk1si0uyipuj3kHYB3GnwAnOidYYGljB+py6+KG ORpMhYyYk3ft8/DEz7BccTFPHg9xxO+pdBypqePmlqzboouNUNELeTm/GJFDHJ+gpMOk YY8A3GFE+Ahd02Qd0jhVMuJYJE2orHBZBZsEqsDpZ7t5ocbx0UtFknnryDQRBlw0cJ9k geKo4EKBOtgk3LkSdeqV/RrwxhM21hrzXNRH7BQjQ2lDR+ULGykpj19PvmAQFq+RDmof EiLq1MfA/tTR2yxEu6NwU8fooHKJRsaexLsyEzk6njDVE2YS8ND5o/M1rRmiU3hAGGay 0Sgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lFEkal5RWVtSpZcGOvMDSOh9991br9vxHdhHcAmHv40=; b=t5LuNIkTj1NCK465adzBkzieLCvIovhTswKLn0wem/ARc3ipRGH9/8nvys4lwwJhx0 QX0fa1TTPawQvfz6xFsPxswgdfBUB0JXOo7TN4eVIU6Tbed0o4mQPm7OxFjjgrsZDz2G TW2RheufthFCnZPX6S0cM5Oq9pFEzFqEq+d8eHL9U0rRaZ1brEuwAH8XBzwmnanBnd4f 92tAAr4hfqtoqU1rmE65E60K2MijTgxU3dcV1KqchdrWFkaldilWsE3aQ8DTJkuHkJYA ySczMT7VXXtIGBg6m313uDlvRt8y9SwKXHIc2MXLTAWJifFMXZg9sIAdDQNVRCedN6Y6 WVmQ== X-Gm-Message-State: APjAAAWGj2TEBR/n81qH21vmtDwVb2p3okJp0ehOb8yydij6cQZFMPZH js43fDTsRSznTLd/menGARJppkkFR28= X-Google-Smtp-Source: APXvYqwcLcmQz9lRcugV4i/68nGpQPeXEBog4DiKz6nfabGaPpLw5o3IZ7qY0X4BGkx/TOcIStZzyA== X-Received: by 2002:a17:902:934a:: with SMTP id g10mr125654208plp.18.1564627682783; Wed, 31 Jul 2019 19:48:02 -0700 (PDT) Received: from localhost.localdomain ([123.201.36.76]) by smtp.gmail.com with ESMTPSA id 185sm79857669pfa.170.2019.07.31.19.48.00 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 19:48:02 -0700 (PDT) From: Atharva Lele To: buildroot@buildroot.org Date: Thu, 1 Aug 2019 08:16:43 +0530 Message-Id: <20190801024643.11024-31-itsatharva@gmail.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190801024643.11024-1-itsatharva@gmail.com> References: <20190801024643.11024-1-itsatharva@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 30/30] autobuild-run: make prepare_build() clean the output directory used for reproducibility testing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Atharva Lele , yann.morin.1998@free.fr, thomas.petazzoni@bootlin.com Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Atharva Lele --- Changes v3: - New patch --- scripts/autobuild-run | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index b32cd5b..7fcbf2a 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -380,6 +380,10 @@ class Builder: # shutil.rmtree doesn't remove write-protected files subprocess.call(["rm", "-rf", self.outputdir]) os.mkdir(self.outputdir) + + # If it exists, remove the other output directory used for reproducibility testing + if os.path.exists(self.outputdir_1): + subprocess.call(["rm", "-rf", self.outputdir_1]) with open(os.path.join(self.outputdir, "branch"), "w") as branchf: branchf.write(branch)