From patchwork Mon Mar 8 00:35:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1448820 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=eHSmPcw0; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Dv03F1Bglz9sSC for ; Mon, 8 Mar 2021 11:44:25 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id D66E3828DF; Mon, 8 Mar 2021 01:41:42 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="eHSmPcw0"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 3554D82898; Mon, 8 Mar 2021 01:37:37 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-oi1-x234.google.com (mail-oi1-x234.google.com [IPv6:2607:f8b0:4864:20::234]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id B47D882893 for ; Mon, 8 Mar 2021 01:35:57 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-oi1-x234.google.com with SMTP id x135so4782907oia.9 for ; Sun, 07 Mar 2021 16:35:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KQpJ2JgqxfO5Uc43h5w/EW7ZN/VJLHFxPWdGxAEmEpQ=; b=eHSmPcw0hlQdNmAroTwoR+f+46+uvAe9RYUCicddiuatIybZVX1aq8iBTnOj3pPNz7 Ul6HV0KMMvfOVOpjwvdL3tnNc5IT8hdkDv15PNk7xGK7YDzRf8+0Dcie9L/AcN1NxGKK zhS+dcs8Bbe3WEV66BG+OKYr32uxFOfPZv0vg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KQpJ2JgqxfO5Uc43h5w/EW7ZN/VJLHFxPWdGxAEmEpQ=; b=H4JckYcDcxzyd/i+C7OrNwMe1Iqq63b1Re58xe6AMb6kULdUoBIhOgA+DJFPb+cx/b RBkfy3xfyBlBn4X87oc+uXfMDGl5kBPBPeyuEkfh0yQcjl5Qz/PaMM+Bm6YUr3U1PhlZ ugqSeBaP1xaWi1pqi9VzaBdfIAfIImHmYoEehcENa7acqal7rmxsun6crEoqiS2kUnBD xfXnkS9MCdu1S2alNc9NtRTJswQeVvbPm7N5qNcO9NYcOGXcrAtcw8R95GAENO1gTmK5 rfMTAq3rCCVa0aqTWK2otn0giCsTQ/uiN7yi+XuOG7P3yhtqtRaQAdDlZOaaO5Uq6csQ zKyQ== X-Gm-Message-State: AOAM531OxIB3OJ5+4lvo7PsVHsuD9n1VIx9LyQQWp9/rm/7uVV0fcHXT 39gAUbl0UGrnhDhyqCBKhzO1hNtFol78JqT1 X-Google-Smtp-Source: ABdhPJxY+w0r2gCDde7M/+ZhEANwZke5vYgJRlqaVU9It3C6Fnef4jR8GFBt4mHNnr2xUjglb7t6bw== X-Received: by 2002:aca:4d4e:: with SMTP id a75mr5880568oib.107.1615163756078; Sun, 07 Mar 2021 16:35:56 -0800 (PST) Received: from kiwi.bld.corp.google.com (c-67-190-101-114.hsd1.co.comcast.net. [67.190.101.114]) by smtp.gmail.com with ESMTPSA id a7sm2078112oon.8.2021.03.07.16.35.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 07 Mar 2021 16:35:55 -0800 (PST) From: Simon Glass To: U-Boot Mailing List Cc: Heinrich Schuchardt , Tom Rini , Stephen Warren , Stephen Warren , Simon Glass Subject: [PATCH v5 40/41] test: sandbox: Move sandbox test docs into doc/develop Date: Sun, 7 Mar 2021 17:35:16 -0700 Message-Id: <20210308003517.1574569-39-sjg@chromium.org> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog In-Reply-To: <20210308003517.1574569-1-sjg@chromium.org> References: <20210308003517.1574569-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean At present some of the documentation about running sandbox tests is in the sandbox docs. It makes more sense to put it in with the other testing docs, with a link there from sandbox. Update the documentation accordingly. Also add a paragraph explaining why sandbox exists and the test philosophy that it uses. Signed-off-by: Simon Glass --- (no changes since v1) doc/arch/sandbox.rst | 44 +++++++---------------------------- doc/develop/tests_sandbox.rst | 44 +++++++++++++++++++++++++++++++++-- 2 files changed, 51 insertions(+), 37 deletions(-) diff --git a/doc/arch/sandbox.rst b/doc/arch/sandbox.rst index 1ccb5344ac6..1638b050002 100644 --- a/doc/arch/sandbox.rst +++ b/doc/arch/sandbox.rst @@ -17,6 +17,12 @@ of the sandbox U-Boot. The purpose of running U-Boot under Linux is to test all the generic code, not specific to any one architecture. The idea is to create unit tests which we can run to test this upper level code. +Sandbox allows development of many types of new features in a traditional way, +rather than needing to test each iteration on real hardware. Many U-Boot +features were developed on sandbox, including the core driver model, most +uclasses, verified boot, bloblist, logging and dozens of others. Sandbox has +enabled many large-scale code refactors as well. + CONFIG_SANDBOX is defined when building a native board. The board name is 'sandbox' but the vendor name is unset, so there is a @@ -486,42 +492,10 @@ Testing ------- U-Boot sandbox can be used to run various tests, mostly in the test/ -directory. These include: - -command_ut: - Unit tests for command parsing and handling -compression: - Unit tests for U-Boot's compression algorithms, useful for - security checking. It supports gzip, bzip2, lzma and lzo. -driver model: - Run this pytest:: - - ./test/py/test.py --bd sandbox --build -k ut_dm -v - -image: - Unit tests for images: - test/image/test-imagetools.sh - multi-file images - test/image/test-fit.py - FIT images -tracing: - test/trace/test-trace.sh tests the tracing system (see README.trace) -verified boot: - See test/vboot/vboot_test.sh for this - -If you change or enhance any of the above subsystems, you shold write or -expand a test and include it with your patch series submission. Test -coverage in U-Boot is limited, as we need to work to improve it. - -Note that many of these tests are implemented as commands which you can -run natively on your board if desired (and enabled). - -To run all tests use "make check". - -To run a single test in an existing sandbox build, you can use -T to use the -test device tree, and -c to select the test: - - /tmp/b/sandbox/u-boot -T -c "ut dm pci_busdev" +directory. -This runs dm_test_pci_busdev() which is in test/dm/pci.c +See :doc:`../develop/tests_sandbox` for more information and +:doc:`../develop/testing` for information about testing generally. Memory Map diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index 2b2c4be2dd1..dd15692f656 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -1,7 +1,47 @@ .. SPDX-License-Identifier: GPL-2.0+ -Tests Under the Hood -==================== +Sandbox tests +============= + +Test Design +----------- + +Most uclasses and many functions of U-Boot have sandbox tests. This allows much +of the code to be checked in an developer-friendly environment. + +Sandbox provides a way to write and run unit tests. The traditional approach to +unit tests is to build lots of little executables, one for each test or +category of tests. With sandbox, so far as possible, all the tests share a +small number of executables (e.g. 'u-boot' for sandbox, 'u-boot-spl' and +'u-boot' for sandbox_spl) and can be run very quickly. The vast majority of +tests can run on the 'sandbox' build, + +Available tests +--------------- + +Some of the available tests are: + + - command_ut: Unit tests for command parsing and handling + - compression: Unit tests for U-Boot's compression algorithms, useful for + security checking. It supports gzip, bzip2, lzma and lzo. + - image: Unit tests for images: + + - test/image/test-imagetools.sh - multi-file images + - test/py/tests/test-fit.py - FIT images + - tracing: test/trace/test-trace.sh tests the tracing system (see + README.trace) + - verified boot: test/py/tests/test_vboot.py + +If you change or enhance any U-Boot subsystem, you should write or expand a +test and include it with your patch series submission. Test coverage in some +older areas of U-Boot is still somewhat limited and we need to work to improve +it. + +Note that many of these tests are implemented as commands which you can +run natively on your board if desired (and enabled). + +To run all tests, use 'make check'. + Running sandbox tests directly ------------------------------