From patchwork Mon Jul 8 19:18:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1129284 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.denx.de (client-ip=81.169.180.215; helo=lists.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="BURK4cXN"; dkim-atps=neutral Received: from lists.denx.de (dione.denx.de [81.169.180.215]) by ozlabs.org (Postfix) with ESMTP id 45jFvn5PSkz9sDB for ; Tue, 9 Jul 2019 05:32:21 +1000 (AEST) Received: by lists.denx.de (Postfix, from userid 105) id 661F6C21DDC; Mon, 8 Jul 2019 19:27:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on lists.denx.de X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.denx.de (localhost [IPv6:::1]) by lists.denx.de (Postfix) with ESMTP id 3A0B0C21C51; Mon, 8 Jul 2019 19:21:41 +0000 (UTC) Received: by lists.denx.de (Postfix, from userid 105) id 2D605C21E7F; Mon, 8 Jul 2019 19:21:26 +0000 (UTC) Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) by lists.denx.de (Postfix) with ESMTPS id D99C9C21DF8 for ; Mon, 8 Jul 2019 19:21:21 +0000 (UTC) Received: by mail-io1-f50.google.com with SMTP id q22so16029535iog.4 for ; Mon, 08 Jul 2019 12:21:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ycwjNOSW5hV5OgJnto3yKTM2jTr8VGafdLaoDtPQRgk=; b=BURK4cXNk0vrYJYQ/UCyL7JrbmHmdSaYp4k/SWOknDCmKc4neaHmBB8jlGwAAJLR2j 2rQzL5BfKrk0XdKQw0wapXE1c3YL3chsgIyg/oERCKHq/FCRLaMtAZwjT4TMGbdSvxiS dM36wMu5N1Ypqt+9bckHK7ktr34RIg3OTxSQg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ycwjNOSW5hV5OgJnto3yKTM2jTr8VGafdLaoDtPQRgk=; b=BTSI72PvqjH14aQnjCTknaKyzcaPcCe2tHdetNIQBSp8DKcs4ALxqDvfCnEXoqoVDi uDzuf1uplOo/gDzfDR+Dc/c8wFTt03zdmiM1yPDrCgkV2g2PZnSfSQeeA11m0woXyxg7 ZyZOmbyHfyD5Vx3QNwRvPG4DP16sZXYV2YA29fbw50x1wc93+v+mZsFLM+w4Bzeld0xN nB3SeYLF/SzQIojYyXJr1W0OKnp9fps+UwDLhK/roic4YLKyiLrVcHarx2+PIYTKJcG9 uvZ8lbnx1ppNVcyrI2caIXn5tAADBq9sEkiTPZy8OsaaT7/8rI7XKNRpDtrg3dHFEHkZ c40g== X-Gm-Message-State: APjAAAVnkjDKpi6vcRPWe+U8K0IjD2PwIQWSJydkO9MGcatbkh59S+gz XGq0vAiGvck12FSAvyUoEuAYtTENlVQ= X-Google-Smtp-Source: APXvYqxf9sC+Ym1BRnBgVdm6c1BW9VZBVPaMyKEvoy01Yf0/TNASLzNFpz0YimvNoJmncVgI2ufWUA== X-Received: by 2002:a02:c492:: with SMTP id t18mr23469610jam.67.1562613680658; Mon, 08 Jul 2019 12:21:20 -0700 (PDT) Received: from kiwi.bld.corp.google.com ([2620:15c:183:0:8223:87c:a681:66aa]) by smtp.gmail.com with ESMTPSA id j1sm15545094iop.14.2019.07.08.12.21.20 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Mon, 08 Jul 2019 12:21:20 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Date: Mon, 8 Jul 2019 13:18:49 -0600 Message-Id: <20190708191856.138863-31-sjg@chromium.org> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog In-Reply-To: <20190708191856.138863-1-sjg@chromium.org> References: <20190708191856.138863-1-sjg@chromium.org> MIME-Version: 1.0 Subject: [U-Boot] [PATCH v2 30/37] binman: Allow preserving test directories X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.18 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" Sometimes when debugging tests it is useful to keep the input and output directories so they can be examined later. Add an option for this and update the binman tests to support it. This affects both the test class and the tearDown() function called after each test. Signed-off-by: Simon Glass Signed-off-by: Simon Glass --- Changes in v2: None tools/binman/README | 8 ++++++++ tools/binman/binman.py | 16 +++++++++++++--- tools/binman/cmdline.py | 4 ++++ tools/binman/ftest.py | 28 +++++++++++++++++++++++++--- 4 files changed, 50 insertions(+), 6 deletions(-) diff --git a/tools/binman/README b/tools/binman/README index 7eda244bbe2..2f4c7fec21e 100644 --- a/tools/binman/README +++ b/tools/binman/README @@ -731,6 +731,14 @@ Use '-P 1' to disable this. It is automatically disabled when code coverage is being used (-T) since they are incompatible. +Debugging tests +--------------- + +Sometimes when debugging tests it is useful to keep the input and output +directories so they can be examined later. Use -X or --test-preserve-dirs for +this. + + Advanced Features / Technical docs ---------------------------------- diff --git a/tools/binman/binman.py b/tools/binman/binman.py index 7c1dcfb65fc..9878eb86d4f 100755 --- a/tools/binman/binman.py +++ b/tools/binman/binman.py @@ -46,15 +46,20 @@ except: import control import test_util -def RunTests(debug, verbosity, processes, args): +def RunTests(debug, verbosity, processes, test_preserve_dirs, args): """Run the functional tests and any embedded doctests Args: debug: True to enable debugging, which shows a full stack trace on error verbosity: Verbosity level to use + test_preserve_dirs: True to preserve the input directory used by tests + so that it can be examined afterwards (only useful for debugging + tests). If a single test is selected (in args[0]) it also preserves + the output directory for this test. Both directories are displayed + on the command line. + processes: Number of processes to use to run tests (None=same as #CPUs) args: List of positional args provided to binman. This can hold a test name to execute (as in 'binman -t testSections', for example) - processes: Number of processes to use to run tests (None=same as #CPUs) """ import elf_test import entry_test @@ -82,6 +87,11 @@ def RunTests(debug, verbosity, processes, args): loader = unittest.TestLoader() for module in (entry_test.TestEntry, ftest.TestFunctional, fdt_test.TestFdt, elf_test.TestElf, image_test.TestImage): + # Test the test module about our arguments, if it is interested + if hasattr(module, 'setup_test_args'): + setup_test_args = getattr(module, 'setup_test_args') + setup_test_args(preserve_indir=test_preserve_dirs, + preserve_outdirs=test_preserve_dirs and test_name is not None) if test_name: try: suite.addTests(loader.loadTestsFromName(test_name, module)) @@ -157,7 +167,7 @@ def RunBinman(options, args): if options.test: ret_code = RunTests(options.debug, options.verbosity, options.processes, - args[1:]) + options.test_preserve_dirs, args[1:]) elif options.test_coverage: RunTestCoverage() diff --git a/tools/binman/cmdline.py b/tools/binman/cmdline.py index 39b835666ea..91e007e4e03 100644 --- a/tools/binman/cmdline.py +++ b/tools/binman/cmdline.py @@ -59,6 +59,10 @@ def ParseArgs(argv): parser.add_option('-v', '--verbosity', default=1, type='int', help='Control verbosity: 0=silent, 1=progress, 3=full, ' '4=debug') + parser.add_option('-X', '--test-preserve-dirs', action='store_true', + help='Preserve and display test-created input directories; also ' + 'preserve the output directory if a single test is run (pass test ' + 'name at the end of the command line') parser.usage += """ diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index f5e0b9b9742..256d4a1c5d8 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -6,6 +6,8 @@ # # python -m unittest func_test.TestFunctional.testHelp +from __future__ import print_function + import hashlib from optparse import OptionParser import os @@ -134,10 +136,27 @@ class TestFunctional(unittest.TestCase): @classmethod def tearDownClass(self): """Remove the temporary input directory and its contents""" - if self._indir: - shutil.rmtree(self._indir) + if self.preserve_indir: + print('Preserving input dir: %s' % self._indir) + else: + if self._indir: + shutil.rmtree(self._indir) self._indir = None + @classmethod + def setup_test_args(cls, preserve_indir=False, preserve_outdirs=False): + """Accept arguments controlling test execution + + Args: + preserve_indir: Preserve the shared input directory used by all + tests in this class. + preserve_outdir: Preserve the output directories used by tests. Each + test has its own, so this is normally only useful when running a + single test. + """ + cls.preserve_indir = preserve_indir + cls.preserve_outdirs = preserve_outdirs + def setUp(self): # Enable this to turn on debugging output # tout.Init(tout.DEBUG) @@ -145,7 +164,10 @@ class TestFunctional(unittest.TestCase): def tearDown(self): """Remove the temporary output directory""" - tools._FinaliseForTest() + if self.preserve_outdirs: + print('Preserving output dir: %s' % tools.outdir) + else: + tools._FinaliseForTest() @classmethod def _ResetDtbs(self):