From patchwork Sun Apr 11 04:27:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1464742 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=M/XSGSPT; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FHzQX3Qv9z9sVt for ; Sun, 11 Apr 2021 14:28:52 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 4BD0E804CB; Sun, 11 Apr 2021 06:28:35 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="M/XSGSPT"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 365B18051A; Sun, 11 Apr 2021 06:28:30 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id F0F8D80202 for ; Sun, 11 Apr 2021 06:28:22 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-pj1-x1036.google.com with SMTP id nm3-20020a17090b19c3b029014e1bbf6c60so1043740pjb.4 for ; Sat, 10 Apr 2021 21:28:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4dSaQl2GZFLvUOe0ftzsxxXMxrEhIVTjL4ww8APyxwk=; b=M/XSGSPTRKe6SLWY4bC1uqnJuoGC4BrQhXxfvGtO3DYQguDFw4kDT2hmHLg7i4GPeV Cd05eHMbFyQhzguZBxLsp5wPxucnvYU8xLmv3s1S3ZXcRogoVQPTQU1L+SSEeUy4nGyg FULDS8BoF4QVSGsHe+wXGW16vs+ULAJuq2F6A= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4dSaQl2GZFLvUOe0ftzsxxXMxrEhIVTjL4ww8APyxwk=; b=NU8fjb3A95N/tkaL2SEUpPjYUpk10THFa9zSheAPhgqcOjiddVWHWuzUixTE9ODcg2 cYz5vZV1C+Wiu4+7uTZtbvQfnIq640UMIe2hzDFKT2xGhF8i+I2zF9xo8hz8/HLnYMfl 5a1zMcKzcvNsGCpnj7JrIgA/GiQg/0CmKfFz/Y/DgE2whm1dpSiuhpxsCk6+Lhe9zRj2 nKXLDWLh5+mdDFH1kq232AdxfN/gq4mwDEQp0+2p0tV20r7dS5glaamWhyDHMMdgBCCG jrjXm+LH1RCFelFpSVgw3TkUAvvS6k/Z3Hyp4sNV364iNEY4nnXISkzzMyiFrmRmPmmd aWBQ== X-Gm-Message-State: AOAM5309bv9DcPwRotu2YAPGNOiS4PFfEdczaJSdPtdOlG5wYv5o1p89 TtnLHKd6AuPDGJNzImU8YMtGF6cIWkH21nja X-Google-Smtp-Source: ABdhPJwPLmnJkNDQ10O+wSZzzlBmLglecK0lmTVj7JLWVwTdvsAl9pVdwEQ23Izk28XGUx0c43midw== X-Received: by 2002:a17:90a:288:: with SMTP id w8mr878759pja.163.1618115301240; Sat, 10 Apr 2021 21:28:21 -0700 (PDT) Received: from localhost.localdomain (203-173-153-56.dsl.dyn.ihug.co.nz. [203.173.153.56]) by smtp.gmail.com with ESMTPSA id 186sm2856589pfe.212.2021.04.10.21.28.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 21:28:20 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Marek Vasut , Simon Glass Subject: [PATCH 1/4] buildman: Tidy up a few comments Date: Sun, 11 Apr 2021 16:27:25 +1200 Message-Id: <20210411162720.1.If1823007e3501061cb1d14cf577723996d1c871c@changeid> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog In-Reply-To: <20210411042728.3549261-1-sjg@chromium.org> References: <20210411042728.3549261-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean Add some function comments which are missing, or missing arguments. Signed-off-by: Simon Glass Signed-off-by: Simon Glass --- tools/buildman/builderthread.py | 10 +++++++--- tools/buildman/control.py | 2 ++ tools/buildman/func_test.py | 13 +++++++++++-- 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index 06ed27203a2..6f08ff2da0c 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -89,10 +89,14 @@ class BuilderThread(threading.Thread): Members: builder: The builder which contains information we might need thread_num: Our thread number (0-n-1), used to decide on a - temporary directory. If this is -1 then there are no threads - and we are the (only) main process + temporary directory. If this is -1 then there are no threads + and we are the (only) main process + mrproper: Use 'make mrproper' before each reconfigure + per_board_out_dir: True to build in a separate persistent directory per + board rather than a thread-specific directory + test_exception: Used for testing; True to raise an exception instead of + reporting the build result """ - def __init__(self, builder, thread_num, mrproper, per_board_out_dir): """Set up a new builder thread""" threading.Thread.__init__(self) self.builder = builder diff --git a/tools/buildman/control.py b/tools/buildman/control.py index a7675701466..5fcfba7ca36 100644 --- a/tools/buildman/control.py +++ b/tools/buildman/control.py @@ -124,6 +124,8 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, arguments. This setting is useful for tests. board: Boards() object to use, containing a list of available boards. If this is None it will be created and scanned. + clean_dir: Used for tests only, indicates that the existing output_dir + should be removed before starting the build """ global builder diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py index 3dd2e6ee5bb..c6997d1ee25 100644 --- a/tools/buildman/func_test.py +++ b/tools/buildman/func_test.py @@ -219,12 +219,21 @@ class TestFunctional(unittest.TestCase): return command.RunPipe([[self._buildman_pathname] + list(args)], capture=True, capture_stderr=True) - def _RunControl(self, *args, clean_dir=False, boards=None): + """Run buildman + + Args: + args: List of arguments to pass + boards: + clean_dir: Used for tests only, indicates that the existing output_dir + should be removed before starting the build + + Returns: + result code from buildman + """ sys.argv = [sys.argv[0]] + list(args) options, args = cmdline.ParseArgs() result = control.DoBuildman(options, args, toolchains=self._toolchains, make_func=self._HandleMake, boards=boards or self._boards, - clean_dir=clean_dir) self._builder = control.builder return result From patchwork Sun Apr 11 04:27:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1464743 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=S63rUs24; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FHzQs498nz9sVt for ; Sun, 11 Apr 2021 14:29:09 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id E40B48068F; Sun, 11 Apr 2021 06:28:43 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="S63rUs24"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id EA8C38051A; Sun, 11 Apr 2021 06:28:34 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-pf1-x432.google.com (mail-pf1-x432.google.com [IPv6:2607:f8b0:4864:20::432]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 03E7A804B3 for ; Sun, 11 Apr 2021 06:28:27 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-pf1-x432.google.com with SMTP id w8so3484833pfn.9 for ; Sat, 10 Apr 2021 21:28:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=j7FQOsCu6i9K59QEGKc96pEa2LLotd8c5mPG6YISDq4=; b=S63rUs24cLVB1xuWfCGWZ3HZ2k404D3yNZ+Nq/4Qm7sb6dIoOwxNc1JZvDrOxOAye8 24l7V62RWv2YaqxX8D7QFXRfVBj5MN/qIqCDolWNaj2jQSCO/wsiBIsTEphp/2jP5KHW w3/ST7f2/NIWSGmZijL+3mECKI5ZBIeVeh6y4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=j7FQOsCu6i9K59QEGKc96pEa2LLotd8c5mPG6YISDq4=; b=Ya+l2Jdrz163CZsz6Ge5gpLr/Fg5rs0q7vR21ad7i/g5nRNLFsZlaR8ydhUkWdydHb b4T5+f9uJAbTKYiOAMonGmqwUrY9Ep4xfy5y721HUw5+8oK/nV1GsohaIO47ob779bzH JkXGEyX5kGb8TyH7r2ceGvdWS6BCDWCcUu+KSapeBw+SLIkMD2foTIyATzG+oWyASsMd UW8l2lkWEBP6us7GR0csbj3uudTbYCElbwaNSdA5saBjLsVijgQ2KmhiT/OIDiwynssa 3NvS5BzXFozeOy3jO/eruLOoO7Gbs8CFseQCsJiu2kdTQGQSTo7tJ8za5RoJ8AhauqMf scPg== X-Gm-Message-State: AOAM5325KnaEh7uGecCmTolwFLYwKwX6OQiwOplN2k8BTrMxpb37akvY a2TJGk5tdMs9CpBoWWn9jExZlzcTXm5cFnZS X-Google-Smtp-Source: ABdhPJz9abwpUydlA3rkcB38wYOg+PNBCuyXhZqohHe+25ThRD57cYUGJ8XAjQf966XvSXN35CGLaQ== X-Received: by 2002:a62:2a83:0:b029:21a:d3a4:80f2 with SMTP id q125-20020a622a830000b029021ad3a480f2mr19212495pfq.47.1618115304099; Sat, 10 Apr 2021 21:28:24 -0700 (PDT) Received: from localhost.localdomain (203-173-153-56.dsl.dyn.ihug.co.nz. [203.173.153.56]) by smtp.gmail.com with ESMTPSA id 186sm2856589pfe.212.2021.04.10.21.28.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 21:28:23 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Marek Vasut , Simon Glass Subject: [PATCH 2/4] buildman: Use common code to send an result Date: Sun, 11 Apr 2021 16:27:26 +1200 Message-Id: <20210411162720.2.Iac247ac00ba5db2becd0b4f9cbaee2da9f43b3b1@changeid> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog In-Reply-To: <20210411042728.3549261-1-sjg@chromium.org> References: <20210411042728.3549261-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean At present the code to report a build result is duplicated. Put it in a common function to avoid this. Signed-off-by: Simon Glass Signed-off-by: Simon Glass --- tools/buildman/builderthread.py | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index 6f08ff2da0c..76ffbb66be6 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -444,6 +444,17 @@ class BuilderThread(threading.Thread): target = '%s-%s%s' % (base, dirname, ext) shutil.copy(fname, os.path.join(build_dir, target)) + def _SendResult(self, result): + """Send a result to the builder for processing + + Args: + result: CommandResult object containing the results of the build + """ + if self.thread_num != -1: + self.builder.out_queue.put(result) + else: + self.builder.ProcessResult(result) + def RunJob(self, job): """Run a single job @@ -517,10 +528,7 @@ class BuilderThread(threading.Thread): # We have the build results, so output the result self._WriteResult(result, job.keep_outputs, job.work_in_output) - if self.thread_num != -1: - self.builder.out_queue.put(result) - else: - self.builder.ProcessResult(result) + self._SendResult(result) else: # Just build the currently checked-out build result, request_config = self.RunCommit(None, brd, work_dir, True, @@ -529,10 +537,7 @@ class BuilderThread(threading.Thread): work_in_output=job.work_in_output) result.commit_upto = 0 self._WriteResult(result, job.keep_outputs, job.work_in_output) - if self.thread_num != -1: - self.builder.out_queue.put(result) - else: - self.builder.ProcessResult(result) + self._SendResult(result) def run(self): """Our thread's run function From patchwork Sun Apr 11 04:27:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1464744 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=nmc6Ly1z; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FHzRH0swYz9sVt for ; Sun, 11 Apr 2021 14:29:31 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 910CE80515; Sun, 11 Apr 2021 06:28:48 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="nmc6Ly1z"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id EA8EF80590; Sun, 11 Apr 2021 06:28:38 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id DF5FD804E7 for ; Sun, 11 Apr 2021 06:28:28 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-pf1-x435.google.com with SMTP id s11so6975503pfm.1 for ; Sat, 10 Apr 2021 21:28:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SC9yP8YmafqZlhVz0zyVdLec2RHglRGH/DkpK+cVJsQ=; b=nmc6Ly1zlDHWSEK/OHdMs+bHlhzN1HTUtL1F1kq/vrYSRTndj+o4i/4cqFzQhprGaW 2l7fxcQWeL57g1C7iDZ3nRBRiE0fuIGHbe9prnfcYfi69H4+7P9P4ALFTI2RCTYRxgBA BKWVAoGsrumYlR8wOS5AEiQw81zZqC+twVQ+E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SC9yP8YmafqZlhVz0zyVdLec2RHglRGH/DkpK+cVJsQ=; b=lG7QMyzf89Oacv567G+piMmE1rrJe0p1wQXNsrUnnE4N6tD/uz5n+huosHZmL/Kdac bkZwffNnpbq9kLLp62CKWE7UREfgD+1cHFs1JHvE5w6Ektu7K4Sy4OkBwWBoeTy8q68P kkKfUnP+leBRKDKUXX2ORaqumdeusJ0nfyQA5+jDhT/555OHQLG9l8BiFKPztL89Tqdx FA3b7qcJnzRPjt62bNRhK6e9VdwpzDDAWzOVW9h8HV5E/FzXF4mG+VKkzMZc60PCdwZc u3BjNEN1pTa0vA9AWMa9TjMZ3YFhImUnXCfDtqbdtZ3K5dQ/BTwz1bdIyeaP/QUXblFM 6OAQ== X-Gm-Message-State: AOAM530rJ34hw0Luo6cGMlHbE82GVZ8GlM4tDwV4tI0iP8i+mj8KlZEI Nwt0l1RB9EwEL/ht5sJTY3j/OypOz+oOOaYr X-Google-Smtp-Source: ABdhPJw7lJcRwYFCZ/e9Vw2gVLJUddHBJ92AKfLR2Z2Yqy68YM7u6Eid9RrS9NyTvOsskCf3fjY9MA== X-Received: by 2002:a63:3189:: with SMTP id x131mr20348171pgx.430.1618115307058; Sat, 10 Apr 2021 21:28:27 -0700 (PDT) Received: from localhost.localdomain (203-173-153-56.dsl.dyn.ihug.co.nz. [203.173.153.56]) by smtp.gmail.com with ESMTPSA id 186sm2856589pfe.212.2021.04.10.21.28.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 21:28:26 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Marek Vasut , Simon Glass Subject: [PATCH 3/4] buildman: Handle exceptions in threads gracefully Date: Sun, 11 Apr 2021 16:27:27 +1200 Message-Id: <20210411162720.3.I473fd4e60936c695b1387b5a63437577ab9930aa@changeid> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog In-Reply-To: <20210411042728.3549261-1-sjg@chromium.org> References: <20210411042728.3549261-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean There have been at least a few cases where an exception has occurred in a thread and resulted in buildman hanging: running out of disk space and getting a unicode error. Handle these by collecting a list of exceptions, printing them out and reporting failure if any are found. Add a test for this. Signed-off-by: Simon Glass Signed-off-by: Simon Glass --- tools/buildman/builder.py | 22 +++++++++++++++++----- tools/buildman/builderthread.py | 14 +++++++++++++- tools/buildman/control.py | 16 +++++++++++----- tools/buildman/func_test.py | 15 +++++++++++++++ 4 files changed, 56 insertions(+), 11 deletions(-) diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index be8a8fa13a6..ce852eb03a3 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -182,6 +182,7 @@ class Builder: only useful for testing in-tree builds. work_in_output: Use the output directory as the work directory and don't write to a separate output directory. + thread_exceptions: List of exceptions raised by thread jobs Private members: _base_board_dict: Last-summarised Dict of boards @@ -235,7 +236,8 @@ class Builder: no_subdirs=False, full_path=False, verbose_build=False, mrproper=False, per_board_out_dir=False, config_only=False, squash_config_y=False, - warnings_as_errors=False, work_in_output=False): + warnings_as_errors=False, work_in_output=False, + test_thread_exceptions=False): """Create a new Builder object Args: @@ -262,6 +264,9 @@ class Builder: warnings_as_errors: Treat all compiler warnings as errors work_in_output: Use the output directory as the work directory and don't write to a separate output directory. + test_thread_exceptions: Uses for tests only, True to make the + threads raise an exception instead of reporting their result. + This simulates a failure in the code somewhere """ self.toolchains = toolchains self.base_dir = base_dir @@ -311,13 +316,16 @@ class Builder: self._re_migration_warning = re.compile(r'^={21} WARNING ={22}\n.*\n=+\n', re.MULTILINE | re.DOTALL) + self.thread_exceptions = [] + self.test_thread_exceptions = test_thread_exceptions if self.num_threads: self._single_builder = None self.queue = queue.Queue() self.out_queue = queue.Queue() for i in range(self.num_threads): - t = builderthread.BuilderThread(self, i, mrproper, - per_board_out_dir) + t = builderthread.BuilderThread( + self, i, mrproper, per_board_out_dir, + test_exception=test_thread_exceptions) t.setDaemon(True) t.start() self.threads.append(t) @@ -1676,6 +1684,7 @@ class Builder: Tuple containing: - number of boards that failed to build - number of boards that issued warnings + - list of thread exceptions raised """ self.commit_count = len(commits) if commits else 1 self.commits = commits @@ -1689,7 +1698,7 @@ class Builder: Print('\rStarting build...', newline=False) self.SetupBuild(board_selected, commits) self.ProcessResult(None) - + self.thread_exceptions = [] # Create jobs to build all commits for each board for brd in board_selected.values(): job = builderthread.BuilderJob() @@ -1728,5 +1737,8 @@ class Builder: rate = float(self.count) / duration.total_seconds() msg += ', duration %s, rate %1.2f' % (duration, rate) Print(msg) + if self.thread_exceptions: + Print('Failed: %d thread exceptions' % len(self.thread_exceptions), + colour=self.col.RED) - return (self.fail, self.warned) + return (self.fail, self.warned, self.thread_exceptions) diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index 76ffbb66be6..ddb3eab8c03 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -97,12 +97,15 @@ class BuilderThread(threading.Thread): test_exception: Used for testing; True to raise an exception instead of reporting the build result """ + def __init__(self, builder, thread_num, mrproper, per_board_out_dir, + test_exception=False): """Set up a new builder thread""" threading.Thread.__init__(self) self.builder = builder self.thread_num = thread_num self.mrproper = mrproper self.per_board_out_dir = per_board_out_dir + self.test_exception = test_exception def Make(self, commit, brd, stage, cwd, *args, **kwargs): """Run 'make' on a particular commit and board. @@ -449,7 +452,12 @@ class BuilderThread(threading.Thread): Args: result: CommandResult object containing the results of the build + + Raises: + ValueError if self.test_exception is true (for testing) """ + if self.test_exception: + raise ValueError('test exception') if self.thread_num != -1: self.builder.out_queue.put(result) else: @@ -547,5 +555,9 @@ class BuilderThread(threading.Thread): """ while True: job = self.builder.queue.get() - self.RunJob(job) + try: + self.RunJob(job) + except Exception as e: + print('Thread exception:', e) + self.builder.thread_exceptions.append(e) self.builder.queue.task_done() diff --git a/tools/buildman/control.py b/tools/buildman/control.py index 5fcfba7ca36..a98d1b4c06f 100644 --- a/tools/buildman/control.py +++ b/tools/buildman/control.py @@ -110,7 +110,7 @@ def ShowToolchainPrefix(boards, toolchains): return None def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, - clean_dir=False): + clean_dir=False, test_thread_exceptions=False): """The main control code for buildman Args: @@ -126,6 +126,9 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, boards. If this is None it will be created and scanned. clean_dir: Used for tests only, indicates that the existing output_dir should be removed before starting the build + test_thread_exceptions: Uses for tests only, True to make the threads + raise an exception instead of reporting their result. This simulates + a failure in the code somewhere """ global builder @@ -330,7 +333,8 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, config_only=options.config_only, squash_config_y=not options.preserve_config_y, warnings_as_errors=options.warnings_as_errors, - work_in_output=options.work_in_output) + work_in_output=options.work_in_output, + test_thread_exceptions=test_thread_exceptions) builder.force_config_on_failure = not options.quick if make_func: builder.do_make = make_func @@ -370,9 +374,11 @@ def DoBuildman(options, args, toolchains=None, make_func=None, boards=None, if options.summary: builder.ShowSummary(commits, board_selected) else: - fail, warned = builder.BuildBoards(commits, board_selected, - options.keep_outputs, options.verbose) - if fail: + fail, warned, excs = builder.BuildBoards( + commits, board_selected, options.keep_outputs, options.verbose) + if excs: + return 102 + elif fail: return 100 elif warned and not options.ignore_warnings: return 101 diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py index c6997d1ee25..61e3012d2b1 100644 --- a/tools/buildman/func_test.py +++ b/tools/buildman/func_test.py @@ -16,6 +16,7 @@ from buildman import toolchain from patman import command from patman import gitutil from patman import terminal +from patman import test_util from patman import tools settings_data = ''' @@ -219,6 +220,8 @@ class TestFunctional(unittest.TestCase): return command.RunPipe([[self._buildman_pathname] + list(args)], capture=True, capture_stderr=True) + def _RunControl(self, *args, boards=None, clean_dir=False, + test_thread_exceptions=False): """Run buildman Args: @@ -226,6 +229,9 @@ class TestFunctional(unittest.TestCase): boards: clean_dir: Used for tests only, indicates that the existing output_dir should be removed before starting the build + test_thread_exceptions: Uses for tests only, True to make the threads + raise an exception instead of reporting their result. This simulates + a failure in the code somewhere Returns: result code from buildman @@ -234,6 +240,8 @@ class TestFunctional(unittest.TestCase): options, args = cmdline.ParseArgs() result = control.DoBuildman(options, args, toolchains=self._toolchains, make_func=self._HandleMake, boards=boards or self._boards, + clean_dir=clean_dir, + test_thread_exceptions=test_thread_exceptions) self._builder = control.builder return result @@ -597,3 +605,10 @@ class TestFunctional(unittest.TestCase): with self.assertRaises(SystemExit) as e: self._RunControl('-w', clean_dir=False) self.assertIn("specify -o", str(e.exception)) + + def testThreadExceptions(self): + """Test that exceptions in threads are reported""" + with test_util.capture_sys_output() as (stdout, stderr): + self.assertEqual(102, self._RunControl('-o', self._output_dir, + test_thread_exceptions=True)) + self.assertIn('Thread exception: test exception', stdout.getvalue()) From patchwork Sun Apr 11 04:27:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1464745 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=Qf714ZZA; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FHzRT14Pcz9sVt for ; Sun, 11 Apr 2021 14:29:41 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id E6FCD80C69; Sun, 11 Apr 2021 06:28:52 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="Qf714ZZA"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 2862380C5B; Sun, 11 Apr 2021 06:28:42 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 9ED978051D for ; Sun, 11 Apr 2021 06:28:31 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-pf1-x42e.google.com with SMTP id d124so6930012pfa.13 for ; Sat, 10 Apr 2021 21:28:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yyrkOzoeWR24obpUOD7nH6fq87hBfapNQJ6UVDVr8ys=; b=Qf714ZZAOzpP36RU/dcpcTKqS5HefREMluo08sDSPaIMYzpu/N9AIK005fNcVWjrzT uZZdZX+3E5Q9zkP7cAxBiPOwBgk21JhoNot8WXDPeDeucBi3byHeWtrsW5+/kfeUrne1 rN2kIBxq+Aavur+1PvEPJBmx5/GwsI6z5FhJY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yyrkOzoeWR24obpUOD7nH6fq87hBfapNQJ6UVDVr8ys=; b=G0rOUQ8KerSh1wEawII2XFzSqXMNGRR0GJunp4SeMHBfLFM0lVDt2Ftbnl6Iq5aK2o RRz0cg0V/GdhBGdlERKmQGsUN3IAs3qylHSOW0XGBwYvmCzXFMNYmQeH4/cTm2PtZJ72 iC6uRvZN9qcoOKUtJ0VJRh9tEKf7yo0eBP3VkbZ0azn2SHq1YDFt7EgBb/yMr9kZyerM 8qvmvEVTZsBUm+x9NvGyXQjXf7u2T38DXTVYjFXRm5LZQ3itna7Qje3UHLB+vA1ijkqC BcOO+xpc8R2TGTWPRyc5bDXQyMIVMCY7JBq2R14b0H9sjpI3Y2aVEbxjUnOq0K5FqDtg 8yqg== X-Gm-Message-State: AOAM532N8pGriKBGwm/ndwZTOcJxa4EtCHOjWLB1MfC0cD9I6K2hE2fQ sbUtZIrHiYKjEUrtn+ZL34/gU1g5ofLV+rAD X-Google-Smtp-Source: ABdhPJzyr4HH8pxeYa1nYxy7VPPzTwTCqn/As31SG01WGniHoaM/jngIwe/NZcER1AnfaJyYuQLKpg== X-Received: by 2002:a63:f914:: with SMTP id h20mr20842216pgi.114.1618115309897; Sat, 10 Apr 2021 21:28:29 -0700 (PDT) Received: from localhost.localdomain (203-173-153-56.dsl.dyn.ihug.co.nz. [203.173.153.56]) by smtp.gmail.com with ESMTPSA id 186sm2856589pfe.212.2021.04.10.21.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 10 Apr 2021 21:28:29 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Marek Vasut , Simon Glass Subject: [PATCH 4/4] buildman: Use bytes for the environment Date: Sun, 11 Apr 2021 16:27:28 +1200 Message-Id: <20210411162720.4.I7de274bda337a2f2c3cb952c78aeda3f2155d521@changeid> X-Mailer: git-send-email 2.31.1.295.g9ea45b61b8-goog In-Reply-To: <20210411042728.3549261-1-sjg@chromium.org> References: <20210411042728.3549261-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean At present we sometimes see problems in gitlab where the environment has 0x80 characters or sequences which are not valid UTF-8. Avoid this by using bytes for the environment, both internal to buildman and when writing out the 'env' file. Add a test to make sure this works as expected. Reported-by: Marek Vasut Fixes: e5fc79ea718 ("buildman: Write the environment out to an 'env' file") Signed-off-by: Simon Glass Reported-by: Marek Vasut Signed-off-by: Simon Glass --- tools/buildman/builderthread.py | 5 ++--- tools/buildman/func_test.py | 12 ++++++++++++ tools/buildman/toolchain.py | 24 ++++++++++++++++-------- 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/tools/buildman/builderthread.py b/tools/buildman/builderthread.py index ddb3eab8c03..48128cf6732 100644 --- a/tools/buildman/builderthread.py +++ b/tools/buildman/builderthread.py @@ -351,10 +351,9 @@ class BuilderThread(threading.Thread): # Write out the image and function size information and an objdump env = result.toolchain.MakeEnvironment(self.builder.full_path) - with open(os.path.join(build_dir, 'out-env'), 'w', - encoding='utf-8') as fd: + with open(os.path.join(build_dir, 'out-env'), 'wb') as fd: for var in sorted(env.keys()): - print('%s="%s"' % (var, env[var]), file=fd) + fd.write(b'%s="%s"' % (var, env[var])) lines = [] for fname in BASE_ELF_FILENAMES: cmd = ['%snm' % self.toolchain.cross, '--size-sort', fname] diff --git a/tools/buildman/func_test.py b/tools/buildman/func_test.py index 61e3012d2b1..7edbee0652f 100644 --- a/tools/buildman/func_test.py +++ b/tools/buildman/func_test.py @@ -572,6 +572,18 @@ class TestFunctional(unittest.TestCase): self.assertTrue(os.path.exists(os.path.join(board0_dir, 'done'))) self.assertTrue(os.path.exists(os.path.join(board0_dir, 'out-env'))) + def testEnvironmentUnicode(self): + """Test there are no unicode errors when the env has non-ASCII chars""" + try: + varname = b'buildman_test_var' + os.environb[varname] = b'strange\x80chars' + self.assertEqual(0, self._RunControl('-o', self._output_dir)) + board0_dir = os.path.join(self._output_dir, 'current', 'board0') + self.assertTrue(os.path.exists(os.path.join(board0_dir, 'done'))) + self.assertTrue(os.path.exists(os.path.join(board0_dir, 'out-env'))) + finally: + del os.environb[varname] + def testWorkInOutput(self): """Test the -w option which should write directly to the output dir""" board_list = board.Boards() diff --git a/tools/buildman/toolchain.py b/tools/buildman/toolchain.py index acb5a29c8fb..fd137f7300e 100644 --- a/tools/buildman/toolchain.py +++ b/tools/buildman/toolchain.py @@ -179,27 +179,35 @@ class Toolchain: output and possibly unicode encoded output of all build tools by adding LC_ALL=C. + Note that os.environb is used to obtain the environment, since in some + cases the environment many contain non-ASCII characters and we see + errors like: + + UnicodeEncodeError: 'utf-8' codec can't encode characters in position + 569-570: surrogates not allowed + Args: full_path: Return the full path in CROSS_COMPILE and don't set PATH Returns: - Dict containing the environemnt to use. This is based on the current - environment, with changes as needed to CROSS_COMPILE, PATH and - LC_ALL. + Dict containing the (bytes) environment to use. This is based on the + current environment, with changes as needed to CROSS_COMPILE, PATH + and LC_ALL. """ - env = dict(os.environ) + env = dict(os.environb) wrapper = self.GetWrapper() if self.override_toolchain: # We'll use MakeArgs() to provide this pass elif full_path: - env['CROSS_COMPILE'] = wrapper + os.path.join(self.path, self.cross) + env[b'CROSS_COMPILE'] = tools.ToBytes( + wrapper + os.path.join(self.path, self.cross)) else: - env['CROSS_COMPILE'] = wrapper + self.cross - env['PATH'] = self.path + ':' + env['PATH'] + env[b'CROSS_COMPILE'] = tools.ToBytes(wrapper + self.cross) + env[b'PATH'] = tools.ToBytes(self.path) + b':' + env[b'PATH'] - env['LC_ALL'] = 'C' + env[b'LC_ALL'] = b'C' return env