From patchwork Thu Mar 4 13:51:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1447358 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=fCVY9Zys; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DrstZ18F7z9sRf for ; Fri, 5 Mar 2021 00:59:34 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 4663B8292E; Thu, 4 Mar 2021 14:54:05 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="fCVY9Zys"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 4F6B88287F; Thu, 4 Mar 2021 14:52:46 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-ot1-x331.google.com (mail-ot1-x331.google.com [IPv6:2607:f8b0:4864:20::331]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 9A06C8282E for ; Thu, 4 Mar 2021 14:52:23 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-ot1-x331.google.com with SMTP id r19so27308328otk.2 for ; Thu, 04 Mar 2021 05:52:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4uQ8eXfW8WWgby0JzrJtcjVof3dtHyIPEAiOoeUhx+I=; b=fCVY9ZysLiYtZeq429WGj19PwNiTSjpoEXHSpfZUVlwU/7afN7CAFYNVVR2oFdhiCe 7qhQWpS01R3A/Qh2kTi/VDrNYHYnfK3udVil3belcNHOD/wgrtGNwuBNcOe+KA0PLqyF t45UUZlnqL4lFt8na6sFLQk9HuD7aoK3TLFm0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4uQ8eXfW8WWgby0JzrJtcjVof3dtHyIPEAiOoeUhx+I=; b=Z6bhzOui83oXtuTcQyhvTcjU4S1aMaNWPvX10bDjbBTyu3QnzGoeNxPcSH64VGf7jW lWEylIYJWDxB7upxgOjf01pkwIVyOS5N+mM8U7xHMq3uL0DdcnnVNPpgaNSWnibp/dqO x7a3RGGXzsEC4t2Mb/QrhTmGkPnzq2KT5o8HWza1jQIdqrAVqKO9GFcvm/Q7tYTtBw2H SYkXpudKVZmyVh6kg2N6lK2AK6+VMiY5OWnEKDeL0jItXwd3OyNKfWkJb/SJyc4cHxlQ 4HezlAQAfDnJLWi93fLX2sFXAJaOIgOfa+N68rMXJ8OPwX32gZE9a85qgOlUIxRnFfMd g7rA== X-Gm-Message-State: AOAM531pl7Er55UbhZabAEyWlrLZMCM1MSSxwnRys5uAMPs9ESg7bX7S uXUsDa32uqX+8PGd3xQrwpUP4KZABRvhdSwp X-Google-Smtp-Source: ABdhPJz1914msnwNfHa/ykuCu5INut652xEwgxtmdPg7k3p4ZMOGEe5sQDCoi4QzG6b3qwhfVI954Q== X-Received: by 2002:a9d:3bc4:: with SMTP id k62mr3444503otc.318.1614865941909; Thu, 04 Mar 2021 05:52:21 -0800 (PST) Received: from kiwi.bld.corp.google.com (c-67-190-101-114.hsd1.co.comcast.net. [67.190.101.114]) by smtp.gmail.com with ESMTPSA id x10sm4029465otp.19.2021.03.04.05.52.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Mar 2021 05:52:21 -0800 (PST) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Heinrich Schuchardt , Stephen Warren , Stephen Warren , Simon Glass Subject: [PATCH v4 41/42] test: sandbox: Move sandbox test docs into doc/develop Date: Thu, 4 Mar 2021 06:51:17 -0700 Message-Id: <20210304135118.643277-40-sjg@chromium.org> X-Mailer: git-send-email 2.30.1.766.gb4fecdf3b7-goog In-Reply-To: <20210304135118.643277-1-sjg@chromium.org> References: <20210304135118.643277-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.4 at phobos.denx.de X-Virus-Status: Clean At present some of the documentation about running sandbox tests is in the sandbox docs. It makes more sense to put it in with the other testing docs, with a link there from sandbox. Update the documentation accordingly. Also add a paragraph explaining why sandbox exists and the test philosophy that it uses. Signed-off-by: Simon Glass --- (no changes since v1) doc/arch/sandbox.rst | 44 +++++++---------------------------- doc/develop/tests_sandbox.rst | 44 +++++++++++++++++++++++++++++++++-- 2 files changed, 51 insertions(+), 37 deletions(-) diff --git a/doc/arch/sandbox.rst b/doc/arch/sandbox.rst index 1ccb5344ac6..1638b050002 100644 --- a/doc/arch/sandbox.rst +++ b/doc/arch/sandbox.rst @@ -17,6 +17,12 @@ of the sandbox U-Boot. The purpose of running U-Boot under Linux is to test all the generic code, not specific to any one architecture. The idea is to create unit tests which we can run to test this upper level code. +Sandbox allows development of many types of new features in a traditional way, +rather than needing to test each iteration on real hardware. Many U-Boot +features were developed on sandbox, including the core driver model, most +uclasses, verified boot, bloblist, logging and dozens of others. Sandbox has +enabled many large-scale code refactors as well. + CONFIG_SANDBOX is defined when building a native board. The board name is 'sandbox' but the vendor name is unset, so there is a @@ -486,42 +492,10 @@ Testing ------- U-Boot sandbox can be used to run various tests, mostly in the test/ -directory. These include: - -command_ut: - Unit tests for command parsing and handling -compression: - Unit tests for U-Boot's compression algorithms, useful for - security checking. It supports gzip, bzip2, lzma and lzo. -driver model: - Run this pytest:: - - ./test/py/test.py --bd sandbox --build -k ut_dm -v - -image: - Unit tests for images: - test/image/test-imagetools.sh - multi-file images - test/image/test-fit.py - FIT images -tracing: - test/trace/test-trace.sh tests the tracing system (see README.trace) -verified boot: - See test/vboot/vboot_test.sh for this - -If you change or enhance any of the above subsystems, you shold write or -expand a test and include it with your patch series submission. Test -coverage in U-Boot is limited, as we need to work to improve it. - -Note that many of these tests are implemented as commands which you can -run natively on your board if desired (and enabled). - -To run all tests use "make check". - -To run a single test in an existing sandbox build, you can use -T to use the -test device tree, and -c to select the test: - - /tmp/b/sandbox/u-boot -T -c "ut dm pci_busdev" +directory. -This runs dm_test_pci_busdev() which is in test/dm/pci.c +See :doc:`../develop/tests_sandbox` for more information and +:doc:`../develop/testing` for information about testing generally. Memory Map diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst index daf87e53a99..8c2eac4c5f4 100644 --- a/doc/develop/tests_sandbox.rst +++ b/doc/develop/tests_sandbox.rst @@ -1,7 +1,47 @@ .. SPDX-License-Identifier: GPL-2.0+ -Tests Under the Hood -==================== +Sandbox tests +============= + +Test Design +----------- + +Most uclasses and many functions of U-Boot have sandbox tests. This allows much +of the code to be checked in an developer-friendly environment. + +Sandbox provides a way to write and run unit tests. The traditional approach to +unit tests is to build lots of little executables, one for each test or +category of tests. With sandbox, so far as possible, all the tests share a +small number of executables (e.g. 'u-boot' for sandbox, 'u-boot-spl' and +'u-boot' for sandbox_spl) and can be run very quickly. The vast majority of +tests can run on the 'sandbox' build, + +Available tests +--------------- + +Some of the available tests are: + + - command_ut: Unit tests for command parsing and handling + - compression: Unit tests for U-Boot's compression algorithms, useful for + security checking. It supports gzip, bzip2, lzma and lzo. + - image: Unit tests for images: + + - test/image/test-imagetools.sh - multi-file images + - test/py/tests/test-fit.py - FIT images + - tracing: test/trace/test-trace.sh tests the tracing system (see + README.trace) + - verified boot: test/py/tests/test_vboot.py + +If you change or enhance any U-Boot subsystem, you should write or expand a +test and include it with your patch series submission. Test coverage in some +older areas of U-Boot is still somewhat limited and we need to work to improve +it. + +Note that many of these tests are implemented as commands which you can +run natively on your board if desired (and enabled). + +To run all tests, use 'make check'. + Running sandbox tests directly ------------------------------