From patchwork Fri Feb 2 20:57:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 868783 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zY8SN1Bmdz9t20 for ; Sat, 3 Feb 2018 07:57:24 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A599D88DC1; Fri, 2 Feb 2018 20:57:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8Rkq6Ytc6z9k; Fri, 2 Feb 2018 20:57:21 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id D16D388DB4; Fri, 2 Feb 2018 20:57:21 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 93AA21C28CD for ; Fri, 2 Feb 2018 20:57:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8E12D88DB1 for ; Fri, 2 Feb 2018 20:57:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9xLr0SfmgQwe for ; Fri, 2 Feb 2018 20:57:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from secvs01.rockwellcollins.com (secvs01.rockwellcollins.com [205.175.225.240]) by fraxinus.osuosl.org (Postfix) with ESMTPS id C77E688DB4 for ; Fri, 2 Feb 2018 20:57:17 +0000 (UTC) Received: from ofwgwc03.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.225.12]) by secvs01.rockwellcollins.com with ESMTP; 02 Feb 2018 14:57:16 -0600 X-Received: from bacon.rockwellcollins.com (unknown [192.168.6.146]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id 4C83560191; Fri, 2 Feb 2018 14:57:16 -0600 (CST) From: Matt Weber To: buildroot@buildroot.org Date: Fri, 2 Feb 2018 14:57:11 -0600 Message-Id: <20180202205713.27191-3-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 2.14.2 In-Reply-To: <20180202205713.27191-1-matthew.weber@rockwellcollins.com> References: <20180202205713.27191-1-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v3 1/3] autobuilder: hung build: convert to monitor thread X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Check the build-time.log and monitor for modifications to determine if the build has hung for at most #mins before killing the build and reporting a timeout. This allows infinite sized builds as we get to a lower number of autobr fails. Less failures means we start to see false timeout failures when we hit the boundary of the old MAX_DURATION ~8hrs. Signed-off-by: Matthew Weber --- Change Log v2 -> v3 - Adjust hung timeout to 2hrs as minic, gst-ffmpeg and qt5webkit could go beyond 60mins on a minimal 2-4core machine while processing a single build step written v1->v2 [Thomas P. - Use mtime vs reading file - Use datetime for hung delta check - Removed camel case - Added hung build event to sync hand-off back to main thread --- scripts/autobuild-run | 58 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 9 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 2949417..04dffcb 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -143,6 +143,8 @@ import sys from time import localtime, strftime from distutils.version import StrictVersion import platform +from threading import Thread, Event +import datetime if sys.hexversion >= 0x3000000: import configparser @@ -167,7 +169,12 @@ else: decode_bytes = _identity encode_str = _identity -MAX_DURATION = 60 * 60 * 8 +# The following pkgs can be > 60mins of build time +# gst-ffmpeg - http://autobuild.buildroot.net/results/5f7/5f7d1847ebd65f221bf18095decaa3383d24a89c/ +# qt5webkit - http://autobuild.buildroot.net/results/195/195dc9ad4b21f6e7675bed277209ba2480337d54/ +# mimic - http://autobuild.buildroot.net/results/ae6/ae66d86988d8c7c0ae19597fed9dee4fafd48f90/ +# +HUNG_BUILD_TIMEOUT = 120 # mins VERSION = 1 def log_write(logf, msg): @@ -199,7 +206,7 @@ def get_branch(): return branches[randint(0, len(branches) - 1)] class SystemInfo: - DEFAULT_NEEDED_PROGS = ["make", "git", "gcc", "timeout"] + DEFAULT_NEEDED_PROGS = ["make", "git", "gcc"] DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"] def __init__(self): @@ -358,6 +365,24 @@ def gen_config(**kwargs): ret = subprocess.call(args, stdout=devnull, stderr=log) return ret +def stop_on_build_hang(monitor_thread_hung_build_flag, + monitor_thread_stop_flag, + sub_proc, outputdir, log): + build_time_logfile = os.path.join(outputdir, "build/build-time.log") + while True: + if monitor_thread_stop_flag.is_set(): + return + if os.path.exists(build_time_logfile): + mtime = datetime.datetime.fromtimestamp(os.stat(build_time_logfile).st_mtime) + + if mtime < datetime.datetime.now() - datetime.timedelta(minutes=HUNG_BUILD_TIMEOUT): + if sub_proc.poll() is None: + monitor_thread_hung_build_flag.set() # Used by do_build() to determine build hang + log_write(log, "INFO: build hung") + sub_proc.kill() + break + monitor_thread_stop_flag.wait(30) + def do_build(**kwargs): """Run the build itself""" @@ -375,25 +400,40 @@ def do_build(**kwargs): f = open(os.path.join(outputdir, "logfile"), "w+") log_write(log, "INFO: build started") - cmd = ["timeout", str(MAX_DURATION), - "nice", "-n", str(nice), + cmd = ["nice", "-n", str(nice), "make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "BR2_JLEVEL=%s" % kwargs['njobs']] \ + kwargs['make_opts'].split() sub = subprocess.Popen(cmd, stdout=f, stderr=f) + + # Setup hung build monitoring thread + monitor_thread_hung_build_flag = Event() + monitor_thread_stop_flag = Event() + build_monitor = Thread(target=stop_on_build_hang, + args=(monitor_thread_hung_build_flag, + monitor_thread_stop_flag, + sub, outputdir, log)) + build_monitor.daemon = True + build_monitor.start() + kwargs['buildpid'][kwargs['instance']] = sub.pid ret = sub.wait() kwargs['buildpid'][kwargs['instance']] = 0 - # 124 is a special error code that indicates we have reached the - # timeout - if ret == 124: - log_write(log, "INFO: build timed out") + # If build failed, monitor thread would have exited at this point + if monitor_thread_hung_build_flag.is_set(): + log_write(log, "INFO: build timed out [%d]" % ret) return -2 + else: + # Stop monitor thread as this build didn't timeout + monitor_thread_stop_flag.set() + # Monitor thread should be exiting around this point + if ret != 0: - log_write(log, "INFO: build failed") + log_write(log, "INFO: build failed [%d]" % ret) return -1 + cmd = ["make", "O=%s" % outputdir, "-C", srcdir, "BR2_DL_DIR=%s" % dldir, "legal-info"] \ + kwargs['make_opts'].split() From patchwork Fri Feb 2 20:57:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 868786 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zY8Sf3q3Tz9t20 for ; Sat, 3 Feb 2018 07:57:38 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9E22E30529; Fri, 2 Feb 2018 20:57:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Bav1kIU3TTHI; Fri, 2 Feb 2018 20:57:32 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id A4BAA306A1; Fri, 2 Feb 2018 20:57:32 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 7D4571C28CD for ; Fri, 2 Feb 2018 20:57:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 7A01C30529 for ; Fri, 2 Feb 2018 20:57:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IYG7f6ewMsGm for ; Fri, 2 Feb 2018 20:57:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs01.rockwellcollins.com (smtpimr.rockwellcollins.com [205.175.226.27]) by silver.osuosl.org (Postfix) with ESMTPS id B0BE2306A1 for ; Fri, 2 Feb 2018 20:57:30 +0000 (UTC) Received: from ofwch3n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs01.rockwellcollins.com with ESMTP; 02 Feb 2018 14:57:16 -0600 X-Received: from bacon.rockwellcollins.com (unknown [192.168.6.146]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id 524E8603A1; Fri, 2 Feb 2018 14:57:16 -0600 (CST) From: Matt Weber To: buildroot@buildroot.org Date: Fri, 2 Feb 2018 14:57:12 -0600 Message-Id: <20180202205713.27191-4-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 2.14.2 In-Reply-To: <20180202205713.27191-1-matthew.weber@rockwellcollins.com> References: <20180202205713.27191-1-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v3 2/3] autobuilder: schema update for branch varchar X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Matthew Weber --- web/schema.sql | 1 + 1 file changed, 1 insertion(+) diff --git a/web/schema.sql b/web/schema.sql index 929ceed..e496283 100644 --- a/web/schema.sql +++ b/web/schema.sql @@ -11,6 +11,7 @@ CREATE TABLE `results` ( `static` tinyint(1) NOT NULL default '0', `subarch` varchar(64) NOT NULL DEFAULT '', `duration` int(11) NOT NULL DEFAULT '0', + `branch` varchar(64) NOT NULL DEFAULT '', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1; From patchwork Fri Feb 2 20:57:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 868784 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zY8SQ0jlQz9t20 for ; Sat, 3 Feb 2018 07:57:25 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id CA9AD8A3C1; Fri, 2 Feb 2018 20:57:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ijWsht+d7e+f; Fri, 2 Feb 2018 20:57:20 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id F12458A3C2; Fri, 2 Feb 2018 20:57:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id A40D01C28CD for ; Fri, 2 Feb 2018 20:57:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A038D306A1 for ; Fri, 2 Feb 2018 20:57:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gm2rAS4dQIos for ; Fri, 2 Feb 2018 20:57:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from da1vs02.rockwellcollins.com (smtpimr.rockwellcollins.com [205.175.227.29]) by silver.osuosl.org (Postfix) with ESMTPS id C7EE630529 for ; Fri, 2 Feb 2018 20:57:17 +0000 (UTC) Received: from ofwda1n02.rockwellcollins.com (HELO crulimr02.rockwellcollins.com) ([205.175.227.14]) by da1vs02.rockwellcollins.com with ESMTP; 02 Feb 2018 14:57:17 -0600 X-Received: from bacon.rockwellcollins.com (unknown [192.168.6.146]) by crulimr02.rockwellcollins.com (Postfix) with ESMTP id 5FAA460512; Fri, 2 Feb 2018 14:57:16 -0600 (CST) From: Matt Weber To: buildroot@buildroot.org Date: Fri, 2 Feb 2018 14:57:13 -0600 Message-Id: <20180202205713.27191-5-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 2.14.2 In-Reply-To: <20180202205713.27191-1-matthew.weber@rockwellcollins.com> References: <20180202205713.27191-1-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [PATCH v3 3/3] autobuilder: failure reason use build-time.log X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" When the make output doesn't parse to provide a result and unknown is assumed, check the build-time.log and see if the last line was a :start:. IF so, use this pkgname instead before declaring unknown. This was tested against the ti-cgt-pru hang. http://autobuild.buildroot.net/results/60e/60e11a3bb90b9f41259e4a970081b72d8b8d100b// Signed-off-by: Matthew Weber --- web/import.inc.php | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/web/import.inc.php b/web/import.inc.php index 8eb3066..081a1cb 100644 --- a/web/import.inc.php +++ b/web/import.inc.php @@ -235,7 +235,12 @@ function import_result($buildid, $filename) if (trim($tmp[0])) $reason = $tmp[0]; else - $reason = "unknown"; + exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp); + if (trim($tmp[0])) + print "Using build-time.log for reason[".trim($tmp[0])."]"; + $reason = trim($tmp[0]); + else + $reason = "unknown"; } $db = new db();