From patchwork Mon Oct 20 19:39:26 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 401233 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from silver.osuosl.org (silver.osuosl.org [140.211.166.136]) by ozlabs.org (Postfix) with ESMTP id 7458514008C for ; Tue, 21 Oct 2014 06:40:20 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id ADA9D33431; Mon, 20 Oct 2014 19:40:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tUWe8C4-Nk6o; Mon, 20 Oct 2014 19:40:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 2BFE927291; Mon, 20 Oct 2014 19:40:02 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (hemlock.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 3B1651C2250 for ; Mon, 20 Oct 2014 19:39:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1217B8896A for ; Mon, 20 Oct 2014 19:39:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nWGIP3zVYdIT for ; Mon, 20 Oct 2014 19:39:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-lb0-f178.google.com (mail-lb0-f178.google.com [209.85.217.178]) by hemlock.osuosl.org (Postfix) with ESMTPS id 467E681C08 for ; Mon, 20 Oct 2014 19:39:46 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id w7so4572570lbi.9 for ; Mon, 20 Oct 2014 12:39:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NqI6sY5+w/gJOeeOWfZtT8Y3aLOrWPeVyyoDgwoz5M8=; b=whySsqjipYZOwyIJgIYzgCyIHJg6+vi/rLXZ/SZmo8aWtzO9aR+teCOoTyyroP5LEu j13uuoUnJrkVCONktYcFwY7vCgTDsQ53RsXHL8JKiiCzRW9X7YXrbtN4ZkDY96CpsbI4 vDuvLbn3jO6Wj4yzWn5IigT8oIlvVMaiwCksHJu+D3HMvDmrblPZf8y3lL2I9Ijeb1yr YacHmlS/91W+0nOD8u4J3H4up5XlL7odLa5D3SNSfX1I5STTVOCa23Tco7EvjC5DC3yx mdrpn9SYgGjUHMbsQY+lz+j87bnKQZxDMqz1JzzoyNBMYrJIeYuw/g0SzpMjRAHbpHGI VDCQ== X-Received: by 10.152.115.131 with SMTP id jo3mr29670588lab.20.1413833984498; Mon, 20 Oct 2014 12:39:44 -0700 (PDT) Received: from localhost.localdomain (d54C62EEB.access.telenet.be. [84.198.46.235]) by mx.google.com with ESMTPSA id nx6sm2663061lbb.20.2014.10.20.12.39.43 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 20 Oct 2014 12:39:43 -0700 (PDT) From: Thomas De Schampheleire To: thomas.petazzoni@free-electrons.com Date: Mon, 20 Oct 2014 21:39:26 +0200 Message-Id: <1413833968-11808-8-git-send-email-patrickdepinguin@gmail.com> X-Mailer: git-send-email 1.8.5.1 In-Reply-To: <1413833968-11808-1-git-send-email-patrickdepinguin@gmail.com> References: <1413833968-11808-1-git-send-email-patrickdepinguin@gmail.com> Cc: Thomas De Schampheleire , buildroot@buildroot.org Subject: [Buildroot] [PATCHv3 buildroot-test 7/9] autobuild-run: improve the logic to generate build-end.log X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire This patch makes the generation of build-end.log more intelligent, by not simply adding the 500 last lines, but instead capture the entire output of the failed package. If no failure reason can be found (possibly a good run) or the start for that package cannot be found, use the 500 last lines as fallback. The logic to find the failed package is taken from the result import script web/import.inc.php. If needed, the search range of the last 3 lines could be extended. Note that the package-version string extracted from the build log is split on the last hyphen into a tuple (package, version). This is needed to find back the beginning of that package's build (where the package name is separated from the version with a space, not a hyphen). However, this logic will fail to work when the version also contains a hyphen, for example lmbench-3.0-a9. In this case, the resulting tuple will be (lmbench-3.0,a9) and the text searched in the build log '>>> lmbench-3.0 a9' (which will not be found). In this case, the fallback of the last 500 lines will be used. Signed-off-by: Thomas De Schampheleire --- scripts/autobuild-run | 53 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 46 insertions(+), 7 deletions(-) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 299a931..7b88c20 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -44,10 +44,6 @@ # # TODO: # -# - Improve the logic that generates the 'build-end.log' file. Instead -# of just using the last 500 lines of the build log, search the -# start of the build of the failing package. -# # - Include the config.log file (when it exists) in the tarball for # failed builds when the failure occurs on an autotools package. # @@ -485,9 +481,52 @@ def send_results(result, **kwargs): subprocess.call(["git log master -n 1 --pretty=format:%%H > %s" % \ os.path.join(resultdir, "gitid")], shell=True, cwd=srcdir) - subprocess.call(["tail -500 %s > %s" % \ - (os.path.join(outputdir, "logfile"), os.path.join(resultdir, "build-end.log"))], - shell=True) + + def get_failure_reason(): + # Output is a tuple (package, version), or None. + + lastlines = subprocess.check_output(["tail", "-n", "3", + os.path.join(outputdir, "logfile")]).splitlines() + + import re + regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/') + for line in lastlines: + m = regexp.search(line) + if m: + return m.group(1).rsplit('-', 1) + + # not found + return None + + def extract_end_log(resultfile): + """Save the last part of the build log, starting from the failed package""" + + def extract_last_500_lines(): + subprocess.call(["tail -500 %s > %s" % \ + (os.path.join(outputdir, "logfile"), resultfile)], + shell=True) + + reason = get_failure_reason() + if not reason: + extract_last_500_lines() + else: + import mmap + f = open(os.path.join(outputdir, "logfile"), 'r') + mf = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) + mf.seek(0) + # Search for first action on the failed package + offset = mf.find('>>> %s' % ' '.join(reason)) + if offset != -1: + with open(resultfile, "w") as endlog: + endlog.write(mf[offset:]) + else: + # not found, use last 500 lines as fallback + extract_last_500_lines() + + mf.close() + f.close() + + extract_end_log(os.path.join(resultdir, "build-end.log")) resultf = open(os.path.join(resultdir, "status"), "w+") if result == 0: