From patchwork Tue Nov 5 19:29:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 1189908 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47709R5zb0z9sNx for ; Wed, 6 Nov 2019 06:29:47 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C89DF845E0; Tue, 5 Nov 2019 19:29:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ce3EJa7F22gk; Tue, 5 Nov 2019 19:29:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8593D844A7; Tue, 5 Nov 2019 19:29:44 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 92C091BF40D for ; Tue, 5 Nov 2019 19:29:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 8AE5589F6D for ; Tue, 5 Nov 2019 19:29:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RgIQIrs6Z5UT for ; Tue, 5 Nov 2019 19:29:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs02.rockwellcollins.com (ch3vs02.rockwellcollins.com [205.175.226.29]) by whitealder.osuosl.org (Postfix) with ESMTPS id 29F3D89F6F for ; Tue, 5 Nov 2019 19:29:40 +0000 (UTC) IronPort-SDR: jqVfEPouE/SVySAlRmvdy24+OtLTvjm2ix2tkbEDPvG1XQ2R5B6hhCBDOM5XLtBEjWA2xelwO+ la5agVud9f52Nsn/rj0zdzsuU4CvsPMM2EhdDKMyMoJ3YHFKzAxFAWbuDWt+sXyHWwV6Po811F YjFTS3QIAYOIrf1rPReZg+gPrU0hmdOVyyTmUdfD4dUDN2OW2AMgA1kgztJnwsyPN3AvefK/s5 wwFzG5nqNcyo8l+Yiyna59t3L+WgmxxfWTmONywUOBO6TRUB1RxkVlUu+8pj7zRkKyohAjGg2+ ucE= Received: from ofwch3n02.rockwellcollins.com (HELO ciulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs02.rockwellcollins.com with ESMTP; 05 Nov 2019 13:29:39 -0600 X-Received: from biscuits.rockwellcollins.lab (biscuits.rockwellcollins.lab [10.148.119.137]) by ciulimr02.rockwellcollins.com (Postfix) with ESMTP id EE10C2037C; Tue, 5 Nov 2019 13:29:38 -0600 (CST) From: Matt Weber To: buildroot@buildroot.org Date: Tue, 5 Nov 2019 13:29:38 -0600 Message-Id: <20191105192938.60771-1-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 2.17.1 Subject: [Buildroot] [PATCH v2] docs/manual: run-tests run-time test framework X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Patrick Havelange , Sam Voss , Ricardo Martincoski , Jared Bents MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This patch adds a new manual section that captures an overview of the run-tests tool, how to manually run a test and where to find the test case script. A brief set of steps is included to go through how to add a new test case and suggestions on how to test/debug. Cc: Ricardo Martincoski Cc: Patrick Havelange Cc: Sam Voss Cc: Jared Bents Signed-off-by: Matthew Weber --- Changes v1 -> v2 [Patrick - Fixed tests folder reference to be plural - Added notes about rebuilding and modifying source in the output_folder --- docs/manual/contribute.txt | 125 +++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/docs/manual/contribute.txt b/docs/manual/contribute.txt index f339ca50b8..91ebd01b47 100644 --- a/docs/manual/contribute.txt +++ b/docs/manual/contribute.txt @@ -487,3 +487,128 @@ preserve Unix-style line terminators when downloading raw pastes. Following pastebin services are known to work correctly: - https://gist.github.com/ - http://code.bulix.org/ + +=== Contributing run-time tests + +Buildroot includes a run-time testing framework called run-tests built +upon python scripting and QEMU runtime execution. + +* Builds a well defined configuration +* Boots it under QEMU +* Runs some test to verify that a given feature is working + +These tests are hooked into the Gitlab CI's build and testing +infrastructure. To see the current job status, visit +https://gitlab.com/buildroot.org/buildroot/-/jobs. + +Within the Buildroot repository, the testing framework is organized at the +top level in +support/testing/+ by folders of +conf+, +infra+ and +tests+. +All the test cases live under the +tests+ folder and are organized by +boot+, ++core+, +download+, +fs+, +init+, +package+, +toolchain+, and +utils+. + +The Gitlab CI job's execute the +support/testing/run-tests+ tool. For a +current set of tool options see the help description by executing the tool +with '-h'. Some common options include setting the download folder, the +output folder, keeping build output, and for multiple test cases, you +can set the JLEVEL for each. + +Here is an example walk through of running a test case. + +* For a first step, lets see what all the test case options are. The test +cases can be listed by executing +support/testing/run-tests -l+. These tests +can all be ran individually during test development from the console. Both +one at a time and selectively as a group of a subset of tests. + +--------------------- +$ support/testing/run-tests -l +List of tests +test_run (tests.utils.test_check_package.TestCheckPackage) +Test the various ways the script can be called in a simple top to ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootMusl) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainCCache) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainCtngMusl) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainLinaroArm) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv4) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv5) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv7) ... ok +[snip] +test_run (tests.init.test_systemd.TestInitSystemSystemdRoFull) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRoIfupdown) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRoNetworkd) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwFull) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwIfupdown) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwNetworkd) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRo) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRoNet) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRw) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRwNet) ... ok + +Ran 157 tests in 0.021s + +OK +--------------------- + +* Next let's use the Busybox Init system test case with a read/write rootfs ++tests.init.test_busybox.TestInitSystemBusyboxRw+ as our example test case. +* A minimal set of command line arguments when debugging a test case would +include '-d' which points to your dl folder, '-o' to an output folder, and +'-k' to keep any output on both pass/fail. With those options, the test will +retain logging and build artifacts providing status of the build and +execution of the test case. + +--------------------- +$ support/testing/run-tests -d dl -o output_folder -k tests.init.test_busybox.TestInitSystemBusyboxRw +15:03:26 TestInitSystemBusyboxRw Starting +15:03:28 TestInitSystemBusyboxRw Building +15:08:18 TestInitSystemBusyboxRw Building done +15:08:27 TestInitSystemBusyboxRw Cleaning up +. +Ran 1 test in 301.140s + +OK +--------------------- + +* For the case of a successful build, the +output_folder+ would contain a + folder with the Buildroot build, build log and run-time log. If +the build failed, the console output would show the stage at which it failed +(setup / build / run). Depending on the failure stage, the build/run logs +and/or Buildroot build artifacts can be inspected and instrumented. If the +QEMU instance needs to be launched for additional testing, the first few +lines of the run-time log capture it and it would allow some incremental +testing without re-running +support/testing/run-tests+. + +You can also make modifications to the current sources inside the ++output_folder+ (e.g. for debug purposes) and rerun the standard +buildroot make targets (in order to regenerate the complete image with +the new modifications) and then rerun the test. Modifying the sources +directly can speed up debugging compared to adding patches file, wiping +the output directory, and starting the test again. + +--------------------- +$ ls output_folder/ +TestInitSystemBusyboxRw/ +TestInitSystemBusyboxRw-build.log +TestInitSystemBusyboxRw-run.log +--------------------- + +* The source file used to implement this example test is found under ++support/testing/tests/init/test_busybox.py+. This file outlines the +minimal defconfig that creates the build, QEMU configuration to launch +the built images and the test case assertions. + +The best way to get familiar with how to create a test case is to look at a +few of the basic file system +support/testing/tests/fs/+ and init ++support/testing/tests/init/+ test scripts. Those tests give good examples +of a basic build and build with run type of tests. There are other more +advanced cases that use things like nested +br2-external+ folders to provide +skeletons and additional packages. Beyond creating the test script, there +are a couple additional steps that should be taken once you have your initial +test case script. The first is to add yourself in the +DEVELOPERS+ file to +be the maintainer of that test case. The second is to update the Gitlab CI +yml by executing +make .gitlab-ci.yml+. + +To test an existing or new test case within Gitlab CI, there is a method of +invoking a specific test by creating a Buildroot fork in Gitlab under your +account and then follow the instructions outlined in +https://git.busybox.net/buildroot/commit/?id=12904c03a7ccbf0c04c5b1f7e3302916c6f37f50.