[U-Boot,9/9] test/py: run C-based unit tests

From: Stephen Warren <swarren@nvidia.com>

Add tests that execute the existing C-based unit test commands. These
simply run the command and validate the overall result. For now,
fine-grained details are not mapped into separate pytest test results in
the current implementation. However, the detail is available in the log
file for inspection, if attention is needed.

Now that the DM unit test runs under test/py, remove the manual shell
script that invokes it.

Signed-off-by: Stephen Warren <swarren@nvidia.com>
---
 test/dm/test-dm.sh       | 16 -------------
 test/py/tests/test_ut.py | 59 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+), 16 deletions(-)
 delete mode 100755 test/dm/test-dm.sh
 create mode 100644 test/py/tests/test_ut.py

Hi Stephen,

On 27 January 2016 at 23:57, Stephen Warren <swarren@wwwdotorg.org> wrote:
> From: Stephen Warren <swarren@nvidia.com>
>
> Add tests that execute the existing C-based unit test commands. These
> simply run the command and validate the overall result. For now,
> fine-grained details are not mapped into separate pytest test results in
> the current implementation. However, the detail is available in the log
> file for inspection, if attention is needed.
>
> Now that the DM unit test runs under test/py, remove the manual shell
> script that invokes it.
>
> Signed-off-by: Stephen Warren <swarren@nvidia.com>
> ---
>  test/dm/test-dm.sh       | 16 -------------
>  test/py/tests/test_ut.py | 59 ++++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 59 insertions(+), 16 deletions(-)
>  delete mode 100755 test/dm/test-dm.sh
>  create mode 100644 test/py/tests/test_ut.py
>
> diff --git a/test/dm/test-dm.sh b/test/dm/test-dm.sh
> deleted file mode 100755
> index 1a0f1509b415..000000000000
> --- a/test/dm/test-dm.sh
> +++ /dev/null
> @@ -1,16 +0,0 @@
> -#!/bin/sh
> -
> -die() {
> -       echo $1
> -       exit 1
> -}
> -
> -NUM_CPUS=$(cat /proc/cpuinfo |grep -c processor)
> -make O=sandbox sandbox_config || die "Cannot configure U-Boot"
> -make O=sandbox -s -j${NUM_CPUS} || die "Cannot build U-Boot"
> -dd if=/dev/zero of=spi.bin bs=1M count=2
> -echo -n "this is a test" > testflash.bin
> -dd if=/dev/zero bs=1M count=4 >>testflash.bin
> -./sandbox/u-boot -d ./sandbox/arch/sandbox/dts/test.dtb -c "ut dm"
> -rm spi.bin
> -rm testflash.bin
> diff --git a/test/py/tests/test_ut.py b/test/py/tests/test_ut.py
> new file mode 100644
> index 000000000000..b033ca54d756
> --- /dev/null
> +++ b/test/py/tests/test_ut.py
> @@ -0,0 +1,59 @@
> +# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
> +#
> +# SPDX-License-Identifier: GPL-2.0
> +
> +import os.path
> +import pytest
> +
> +@pytest.mark.buildconfigspec('ut_dm')
> +def test_ut_dm(u_boot_console):
> +    """Execute the "ut dm" command."""
> +
> +    fn = u_boot_console.config.source_dir + '/testflash.bin'
> +    if not os.path.exists(fn):
> +        data = 'this is a test'
> +        data += '\x00' * ((4 * 1024 * 1024) - len(data))
> +        with open(fn, 'wb') as fh:
> +            fh.write(data)
> +

The SPI tests are currently disabled, but they will need the spi.bin file.

> +    output = u_boot_console.run_command('ut dm')
> +    assert output.endswith('Failures: 0')
> +
> +@pytest.mark.buildconfigspec('ut_env')
> +def test_ut_env(u_boot_console):
> +    """Execute the "ut env" command."""
> +
> +    output = u_boot_console.run_command('ut env')
> +    assert output.endswith('Failures: 0')
> +
> +@pytest.mark.buildconfigspec('ut_time')
> +def test_ut_time(u_boot_console):
> +    """Execute the "ut time" command."""
> +
> +    output = u_boot_console.run_command('ut time')
> +    assert output.endswith('Test passed')
> +
> +@pytest.mark.buildconfigspec('sandbox')
> +def test_ut_cmd(u_boot_console):
> +    """Execute the "ut_cmd" command."""
> +
> +    output = u_boot_console.run_command('ut_cmd')
> +    assert output.endswith('do_ut_cmd: Everything went swimmingly')
> +
> +@pytest.mark.buildconfigspec('sandbox')
> +def test_ut_compression(u_boot_console):
> +    """Execute the "ut_compression" command."""
> +
> +    output = u_boot_console.run_command('ut_compression')
> +    assert output.endswith('ut_compression ok')
> +
> +# Even when this passes, it prints lots of scary messages such as:
> +#     Must RESET board to recover
> +# Equally, it fails if "ut dm" has been run first in the U-Boot session.
> +# Don't enable this test until those issues have been researched/solved.

The messages are correct because the test is deliberating making the
output buffer to small to check that decompression doesn't overrun the
end.

One way around this would be to silence the console. Sett
dm_test_main() for how it silences the console, and records the
output. Alternatively you could just look for the 'ok' message at the
end.

> +#@pytest.mark.buildconfigspec('sandbox')
> +#def test_ut_compression(u_boot_console):
> +#    """Execute the "ut_image_decomp" command."""
> +#
> +#    output = u_boot_console.run_command('ut_image_decomp')
> +#    assert output.endswith('ut_image_decomp ok')
> --
> 2.7.0
>

Regards,
Simon