[v4,06/42] doc: Explain how to run tests without pytest

Add details about how to run a sandbox test directly, without using
pytest. This is more convenient for rapid development, since it is faster
and allows easier use of a debugger. Also mention sandbox_flattree as an
example of the different sandbox builds available.

Signed-off-by: Simon Glass <sjg@chromium.org>
---

(no changes since v2)

Changes in v2:
- Put the docs in tests_sandbox since it is more related to sandbox
- Put in a mention of tests_sandbox in the main testing docs

 doc/develop/index.rst         |  1 +
 doc/develop/testing.rst       |  9 ++++
 doc/develop/tests_sandbox.rst | 79 +++++++++++++++++++++++++++++++++++
 3 files changed, 89 insertions(+)
 create mode 100644 doc/develop/tests_sandbox.rst

On 3/4/21 2:50 PM, Simon Glass wrote:
> Add details about how to run a sandbox test directly, without using
> pytest. This is more convenient for rapid development, since it is faster
> and allows easier use of a debugger. Also mention sandbox_flattree as an
> example of the different sandbox builds available.
>
> Signed-off-by: Simon Glass <sjg@chromium.org>
> ---
>
> (no changes since v2)
>
> Changes in v2:
> - Put the docs in tests_sandbox since it is more related to sandbox
> - Put in a mention of tests_sandbox in the main testing docs
>
>   doc/develop/index.rst         |  1 +
>   doc/develop/testing.rst       |  9 ++++
>   doc/develop/tests_sandbox.rst | 79 +++++++++++++++++++++++++++++++++++

Thank you for adding this documentation.

tests_sandbox.rst seems to be the wrong file name:

We have three types of tests:

* Python tests
* library tests started using the 'ut' shell commad
* UEFI related tests started using the 'bootefi selftest' command

All three tpyes of tests are run both on real hardware and on the sandbox.

Unfortunately a few Python tests require the sandbox.

Some tests are not run on the sandbox by default though they could, e.g.
the UEFI tests run only on QEMU because you decided not to activate
CONFIG_EFI_SELFTEST on sandbox_defconfig.

Some tests can only be used interactively, e.g.

     setenv efi_selftest block image transfer
     bootefi selftest

Some tests cannot be run on the sandbox at all, e.g.
test/py/tests/test_fpga.py

>   3 files changed, 89 insertions(+)
>   create mode 100644 doc/develop/tests_sandbox.rst
>
> diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> index ac57fdb8f30..50b1de3bdff 100644
> --- a/doc/develop/index.rst
> +++ b/doc/develop/index.rst
> @@ -33,3 +33,4 @@ Testing
>      coccinelle
>      testing
>      py_testing
> +   tests_sandbox
> diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
> index f01ca4dc408..87c90eee271 100644
> --- a/doc/develop/testing.rst
> +++ b/doc/develop/testing.rst
> @@ -36,6 +36,7 @@ U-Boot can be built as a user-space application (e.g. for Linux). This
>   allows test to be executed without needing target hardware. The 'sandbox'
>   target provides this feature and it is widely used in tests.
>
> +See :doc:`tests_sandbox` for more information.
>
>   Pytest Suite
>   ------------
> @@ -51,8 +52,16 @@ You can run the tests on sandbox with::
>
>   This will produce HTML output in build-sandbox/test-log.html
>
> +Some tests run with other versions of sandbox. For example sandbox_flattree
> +runs the tests with livetree (the hierachical devicetree) disabled. You can
> +also select particular tests with -k::
> +
> +   ./test/py/test.py --bd sandbox_flattree --build -k hello
> +
>   See test/py/README.md for more information about the pytest suite.
>
> +See :doc:`tests_sandbox` for how to run tests directly (not through pytest).
> +
>
>   tbot
>   ----
> diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst
> new file mode 100644
> index 00000000000..85bbd4f6734
> --- /dev/null
> +++ b/doc/develop/tests_sandbox.rst
> @@ -0,0 +1,79 @@
> +.. SPDX-License-Identifier: GPL-2.0+
> +
> +Tests Under the Hood
> +====================
> +
> +Running sandbox tests directly
> +------------------------------
> +
> +Typically tests are run using the pytest suite. This is easy and always gets
> +things right.

No, in the contrary:

It is difficult to set up Python testing for real hardware. Running
non-Python tests from the console is trivial both on the sandbox and on
real hardware.

> +
> +But it is also possible to run some sandbox tests directly. For example, this
> +runs the dm_test_gpio() test which you can find in test/dm/gpio.c::
> +
> +   $ ./u-boot -T -c "ut dm gpio"
This works only on the sandbox. To create examples assume that the shell
is already started:

     => ut dm gpio

and

     => bootefi selftest

We should add a man-page for the ut command in /doc/usage/.

> +
> +
> +   U-Boot 2021.01
> +
> +   Model: sandbox
> +   DRAM:  128 MiB
> +   WDT:   Started with servicing (60s timeout)
> +   MMC:   mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD)
> +   In:    serial
> +   Out:   vidconsole
> +   Err:   vidconsole
> +   Model: sandbox
> +   SCSI:
> +   Net:   eth0: eth@10002000, eth5: eth@10003000, eth3: sbe5, eth6: eth@10004000
> +   Test: dm_test_gpio: gpio.c
> +   Test: dm_test_gpio: gpio.c (flat tree)
> +   Failures: 0
> +

Only the text below is sandbox specific.

> +The -T option tells U-Boot to run with the 'test' devicetree (test.dts) instead

%s/U-Boot/the U-Boot sandbox/g

> +of -D which selects the normal sandbox.dts - this is necessary because many
> +tests rely on things in the test devicetree. If you try running tests without

%s/things/nodes/ ?

> +-T then you may see failures, like::
> +
> +   $ ./u-boot -c "ut dm gpio"
> +
> +
> +   U-Boot 2021.01
> +
> +   DRAM:  128 MiB
> +   WDT:   Not found!
> +   MMC:
> +   In:    serial
> +   Out:   serial
> +   Err:   serial
> +   SCSI:
> +   Net:   No ethernet found.
> +   Please run with test device tree:
> +       ./u-boot -d arch/sandbox/dts/test.dtb
> +   Test: dm_test_gpio: gpio.c
> +   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
> +   Test: dm_test_gpio: gpio.c (flat tree)
> +   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
> +   Failures: 2
> +
> +The message above should provide a hint if you forget. Even running with -D

If I forget what? Please, complete the sentence.

Best regards

Heinrich

> +will produce different results.
> +
> +You can easily use gdb on these tests, without needing --gdbserver::
> +
> +   $ gdb u-boot --args -T -c "ut dm gpio"
> +   ...
> +   (gdb) break dm_test_gpio
> +   Breakpoint 1 at 0x1415bd: file test/dm/gpio.c, line 37.
> +   (gdb) run -T -c "ut dm gpio"
> +   Starting program: u-boot -T -c "ut dm gpio"
> +   Test: dm_test_gpio: gpio.c
> +
> +   Breakpoint 1, dm_test_gpio (uts=0x5555558029a0 <global_dm_test_state>)
> +       at files/test/dm/gpio.c:37
> +   37		ut_assertok(gpio_lookup_name("b4", &dev, &offset, &gpio));
> +   (gdb)
> +
> +You can then single-step and look at variables as needed.
> +
>

Hi Heinrich,

On Thu, 4 Mar 2021 at 11:13, Heinrich Schuchardt <xypron.glpk@gmx.de> wrote:
>
> On 3/4/21 2:50 PM, Simon Glass wrote:
> > Add details about how to run a sandbox test directly, without using
> > pytest. This is more convenient for rapid development, since it is faster
> > and allows easier use of a debugger. Also mention sandbox_flattree as an
> > example of the different sandbox builds available.
> >
> > Signed-off-by: Simon Glass <sjg@chromium.org>
> > ---
> >
> > (no changes since v2)
> >
> > Changes in v2:
> > - Put the docs in tests_sandbox since it is more related to sandbox
> > - Put in a mention of tests_sandbox in the main testing docs
> >
> >   doc/develop/index.rst         |  1 +
> >   doc/develop/testing.rst       |  9 ++++
> >   doc/develop/tests_sandbox.rst | 79 +++++++++++++++++++++++++++++++++++
>
> Thank you for adding this documentation.
>
> tests_sandbox.rst seems to be the wrong file name:

But if you look at it, it is all about running tests using sandbox.

>
> We have three types of tests:
>
> * Python tests
> * library tests started using the 'ut' shell commad
> * UEFI related tests started using the 'bootefi selftest' command
>
> All three tpyes of tests are run both on real hardware and on the sandbox.
>
> Unfortunately a few Python tests require the sandbox.
>
> Some tests are not run on the sandbox by default though they could, e.g.
> the UEFI tests run only on QEMU because you decided not to activate
> CONFIG_EFI_SELFTEST on sandbox_defconfig.

It was a long time ago, but I recall spending a year trying to
upstream a bit of refactoring to allow it and then giving up. If you
have a way to do it, please go ahead.

>
> Some tests can only be used interactively, e.g.
>
>      setenv efi_selftest block image transfer
>      bootefi selftest
>
> Some tests cannot be run on the sandbox at all, e.g.
> test/py/tests/test_fpga.py

That is a bug :-)

>
> >   3 files changed, 89 insertions(+)
> >   create mode 100644 doc/develop/tests_sandbox.rst
> >
> > diff --git a/doc/develop/index.rst b/doc/develop/index.rst
> > index ac57fdb8f30..50b1de3bdff 100644
> > --- a/doc/develop/index.rst
> > +++ b/doc/develop/index.rst
> > @@ -33,3 +33,4 @@ Testing
> >      coccinelle
> >      testing
> >      py_testing
> > +   tests_sandbox
> > diff --git a/doc/develop/testing.rst b/doc/develop/testing.rst
> > index f01ca4dc408..87c90eee271 100644
> > --- a/doc/develop/testing.rst
> > +++ b/doc/develop/testing.rst
> > @@ -36,6 +36,7 @@ U-Boot can be built as a user-space application (e.g. for Linux). This
> >   allows test to be executed without needing target hardware. The 'sandbox'
> >   target provides this feature and it is widely used in tests.
> >
> > +See :doc:`tests_sandbox` for more information.
> >
> >   Pytest Suite
> >   ------------
> > @@ -51,8 +52,16 @@ You can run the tests on sandbox with::
> >
> >   This will produce HTML output in build-sandbox/test-log.html
> >
> > +Some tests run with other versions of sandbox. For example sandbox_flattree
> > +runs the tests with livetree (the hierachical devicetree) disabled. You can
> > +also select particular tests with -k::
> > +
> > +   ./test/py/test.py --bd sandbox_flattree --build -k hello
> > +
> >   See test/py/README.md for more information about the pytest suite.
> >
> > +See :doc:`tests_sandbox` for how to run tests directly (not through pytest).
> > +
> >
> >   tbot
> >   ----
> > diff --git a/doc/develop/tests_sandbox.rst b/doc/develop/tests_sandbox.rst
> > new file mode 100644
> > index 00000000000..85bbd4f6734
> > --- /dev/null
> > +++ b/doc/develop/tests_sandbox.rst
> > @@ -0,0 +1,79 @@
> > +.. SPDX-License-Identifier: GPL-2.0+
> > +
> > +Tests Under the Hood
> > +====================
> > +
> > +Running sandbox tests directly
> > +------------------------------
> > +
> > +Typically tests are run using the pytest suite. This is easy and always gets
> > +things right.
>
> No, in the contrary:
>
> It is difficult to set up Python testing for real hardware. Running
> non-Python tests from the console is trivial both on the sandbox and on
> real hardware.

OK I will reword it.

>
> > +
> > +But it is also possible to run some sandbox tests directly. For example, this
> > +runs the dm_test_gpio() test which you can find in test/dm/gpio.c::
> > +
> > +   $ ./u-boot -T -c "ut dm gpio"
> This works only on the sandbox. To create examples assume that the shell
> is already started:

The whole point of this file is to describe using sandbox!

>
>      => ut dm gpio
>
> and
>
>      => bootefi selftest
>
> We should add a man-page for the ut command in /doc/usage/.

OK, please go ahead.

>
> > +
> > +
> > +   U-Boot 2021.01
> > +
> > +   Model: sandbox
> > +   DRAM:  128 MiB
> > +   WDT:   Started with servicing (60s timeout)
> > +   MMC:   mmc2: 2 (SD), mmc1: 1 (SD), mmc0: 0 (SD)
> > +   In:    serial
> > +   Out:   vidconsole
> > +   Err:   vidconsole
> > +   Model: sandbox
> > +   SCSI:
> > +   Net:   eth0: eth@10002000, eth5: eth@10003000, eth3: sbe5, eth6: eth@10004000
> > +   Test: dm_test_gpio: gpio.c
> > +   Test: dm_test_gpio: gpio.c (flat tree)
> > +   Failures: 0
> > +
>
> Only the text below is sandbox specific.
>
> > +The -T option tells U-Boot to run with the 'test' devicetree (test.dts) instead
>
> %s/U-Boot/the U-Boot sandbox/g
>
> > +of -D which selects the normal sandbox.dts - this is necessary because many
> > +tests rely on things in the test devicetree. If you try running tests without
>
> %s/things/nodes/ ?

or properties

>
> > +-T then you may see failures, like::
> > +
> > +   $ ./u-boot -c "ut dm gpio"
> > +
> > +
> > +   U-Boot 2021.01
> > +
> > +   DRAM:  128 MiB
> > +   WDT:   Not found!
> > +   MMC:
> > +   In:    serial
> > +   Out:   serial
> > +   Err:   serial
> > +   SCSI:
> > +   Net:   No ethernet found.
> > +   Please run with test device tree:
> > +       ./u-boot -d arch/sandbox/dts/test.dtb
> > +   Test: dm_test_gpio: gpio.c
> > +   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
> > +   Test: dm_test_gpio: gpio.c (flat tree)
> > +   test/dm/gpio.c:37, dm_test_gpio(): 0 == gpio_lookup_name("b4", &dev, &offset, &gpio): Expected 0x0 (0), got 0xffffffea (-22)
> > +   Failures: 2
> > +
> > +The message above should provide a hint if you forget. Even running with -D
>
> If I forget what? Please, complete the sentence.

OK.

Regards,
Simon