From patchwork Wed Feb 3 12:44:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1435259 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=FFIWvrqI; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DW1d64tBWz9sWR for ; Wed, 3 Feb 2021 23:46:02 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 54E08827E1; Wed, 3 Feb 2021 13:45:56 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="FFIWvrqI"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id DDD57827E1; Wed, 3 Feb 2021 13:45:53 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 1FB9F827C3 for ; Wed, 3 Feb 2021 13:45:50 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-qk1-x731.google.com with SMTP id n15so23156657qkh.8 for ; Wed, 03 Feb 2021 04:45:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=cspdXYRzDEPztTweYwQME6oL8w+mKZGVXbJWDkPw5Oc=; b=FFIWvrqIWP5RtQ7pT2wXA/43BbLk6sR9bzvWWllKKxrY/OUWYejX3QSHdQqrwlSDPG J+/gbxEyvhp7PEPWNd/TTFaJdtcTqPCURJAmQk1fO9Y9YVX1ii7VhYtd44B4lDK4JFZJ es+a0Jw6NFYWPFcT5d8O+CvaESrfKCcMcEehQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=cspdXYRzDEPztTweYwQME6oL8w+mKZGVXbJWDkPw5Oc=; b=Fri0SKL73en7m+mgcdOdHhFJDEZeQ7VSyiRTcV2iEQhO18EuIf7jJLGBnG2Z6WUMR2 trNX/qaaajlqCi+T+Us4S2lWMds80Pjn0kfvpIUNYVgYgQajekYCz2owzLG9En7/CLf8 ntD8rgGJYA4ZK4CXp9pVV0VNN5UTIAJvIodb0dFSlf9Cdrb6ZHzEIpcO+Aga7tzZC9La wT93xEoXaM0e0/61eS7JQGvFAITaDkjRjVkwVZYSxd6xITqxXSOXRUzm5oOReu+Nyvdl f4aFcHlyTQmSJoSqyOkkBOMVGZ5TqCPfVrRb1q+FXVXFMVOs2S5SJWB+bKogqbqlrxzC wCGw== X-Gm-Message-State: AOAM531fp2FfwINI02+UoA6sQFXLb0IXy+yqW8Y7+oGNHlp6+dIoWdyY UdLyHjENoab5WUhEsrqdTxSS6nKR5N9qiA== X-Google-Smtp-Source: ABdhPJzQ5POvAwEwQ4mki6D2Pf/m9S70JSv0Pl3+8EjDi6HyVluwyu24p2wYihHe5hVtElASgZhruQ== X-Received: by 2002:a05:620a:209a:: with SMTP id e26mr2188891qka.313.1612356348447; Wed, 03 Feb 2021 04:45:48 -0800 (PST) Received: from sjg1.roam.corp.google.com ([50.225.158.38]) by smtp.gmail.com with ESMTPSA id p26sm1560903qkk.6.2021.02.03.04.45.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 04:45:46 -0800 (PST) From: Simon Glass To: U-Boot Mailing List Cc: Stephen Warren , Stephen Warren , Heinrich Schuchardt , Tom Rini , Simon Glass , Alexandru Gagniuc , Pratyush Yadav Subject: [PATCH v3 00/41] test: Refactor tests to have a single test runner Date: Wed, 3 Feb 2021 05:44:06 -0700 Message-Id: <20210203124447.2458527-1-sjg@chromium.org> X-Mailer: git-send-email 2.30.0.365.g02bc693789-goog MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean At present U-Boot has two broad sets of tests in the C code: driver model tests which do a lot of pre-/post-init and command tests which do not. This separation makes it slightly harder to write a test, since there are two different test-state structures and different rules for running the two different test types. At present these rules are determined by where the test is (actually its prefix). All unit tests can be run from the command line with the 'ut' command. Since SPL does not have commands, it currently calls the test runner directly and offers no control of which tests are run. This seems like a good time to refactor the tests into a unified test runner, allowing U-Boot proper and SPL to use the same path, perhaps with some different conditions along the way. This series sets up a unified runner called ut_run_list(), which runs a set of tests from a linker_list. Driver model tests are distinguished by a new UT_TESTF_DM flag so that the necessary init and cleanup can still be done. The runner is modified to support running SPL tests that are not solely for driver model. An example test for FIT loading is added as a demonstration. In addition, some documentation is added to explain how to write tests. This series is available at u-boot-dm/test-working Changes in v3: - Add new patch to re-enable test_ofplatdata - Reword the SPL tests section for clarity - Use test_set_state() throughout test-main.c instead of direct assignment - Update the pytest collector as well Changes in v2: - Use correct rst format for 'Ad-hoc tests' section - Expand docs on how each type of test is marked - Put the docs in tests_sandbox since it is more related to sandbox - Put in a mention of tests_sandbox in the main testing docs - Add a note that SPL tests can in fact be run individualy - Document how to run all C tests with 'ut all' - Fix 'get list' typo - Fix conditions so non-DM SPL tests are actually run - Allow for prefix to be NULL, to match function comment - Add new patches to cover running an SPL test Simon Glass (41): doc: Tidy up testing section doc: Document make tcheck sandbox: Drop the 'starting...' message unless testing test: Re-enable test_ofplatdata doc: Explain how to run tests without pytest doc: Document how sandbox_spl_tests are run test: Correct setexpr test prefix test: Mark all driver model tests with a flag test: Rename test-main.c to test-dm.c test: Add an overall test runner test: Create pre/post-run functions test: Call test_pre/post_run() from driver model tests test: Move dm_extended_scan() to test_pre_run() test: Move do_autoprobe() to test_pre_run() test: Move dm_scan_plat() to test_pre_run() test: Drop mallinfo() work-around test: Move console silencing to test_pre_run() test: Move delay skipping to test_pre_run() test: Handle driver model reinit in test_pre_run() test: Drop struct dm_test_state test: Move dm_test_init() into test-main.c test: Move dm_test_destroy() into test-main.c test: Move test running into a separate function test: Use ut_run_test() to run driver model tests test: Drop dm_do_test() test: Add ut_run_test_live_flat() to run tests twice test: Use a local variable for test state test: Run driver-model tests using ut_run_list() test: Use return values in dm_test_run() test: Move the devicetree check into ut_run_list() test: Move restoring of driver model state to ut_run_list() test: log: Rename log main test file to log_ut.c test: Add a macros for finding tests in linker_lists test: Rename all linker lists to have a ut_ prefix test: Allow SPL to run any available test sandbox: Update os_find_u_boot() to find the .img file spl: Convert spl_fit to work with sandbox doc: Move coccinelle into its own section spl: test: Add a test for spl_load_simple_fit() test: sandbox: Move sandbox test docs into doc/develop doc: Explain briefly how to write new tests arch/sandbox/cpu/os.c | 8 +- arch/sandbox/cpu/spl.c | 9 +- arch/sandbox/cpu/start.c | 15 +- arch/sandbox/include/asm/state.h | 1 + common/spl/spl.c | 3 +- common/spl/spl_fit.c | 27 +- configs/sandbox_spl_defconfig | 2 +- doc/arch/sandbox.rst | 48 +--- doc/develop/index.rst | 11 +- doc/develop/py_testing.rst | 3 +- doc/develop/testing.rst | 46 +++- doc/develop/tests_sandbox.rst | 207 ++++++++++++++ doc/develop/tests_writing.rst | 337 +++++++++++++++++++++++ include/dm/test.h | 20 +- include/os.h | 3 +- include/spl.h | 9 + include/test/test.h | 38 ++- include/test/ut.h | 45 +++ test/Makefile | 3 + test/bloblist.c | 5 +- test/bootm.c | 4 +- test/cmd/mem.c | 4 +- test/cmd/setexpr.c | 28 +- test/cmd_ut.c | 38 +-- test/compression.c | 5 +- test/dm/Makefile | 2 +- test/dm/core.c | 47 ++-- test/dm/test-dm.c | 50 ++++ test/dm/test-driver.c | 12 +- test/dm/test-main.c | 229 ---------------- test/dm/test-uclass.c | 10 +- test/env/cmd_ut_env.c | 4 +- test/image/Makefile | 5 + test/image/spl_load.c | 91 ++++++ test/lib/cmd_ut_lib.c | 4 +- test/log/Makefile | 2 +- test/log/{test-main.c => log_ut.c} | 4 +- test/optee/cmd_ut_optee.c | 5 +- test/overlay/cmd_ut_overlay.c | 5 +- test/py/conftest.py | 2 +- test/py/tests/test_log.py | 2 +- test/py/tests/test_ofplatdata.py | 2 +- test/str_ut.c | 5 +- test/test-main.c | 426 +++++++++++++++++++++++++++++ test/unicode_ut.c | 4 +- test/ut.c | 7 + 46 files changed, 1400 insertions(+), 437 deletions(-) create mode 100644 doc/develop/tests_sandbox.rst create mode 100644 doc/develop/tests_writing.rst create mode 100644 test/dm/test-dm.c delete mode 100644 test/dm/test-main.c create mode 100644 test/image/Makefile create mode 100644 test/image/spl_load.c rename test/log/{test-main.c => log_ut.c} (75%) create mode 100644 test/test-main.c