[v7,4/8] Acceptance tests: use relative location for tests
diff mbox series

Message ID 20191104151323.9883-5-crosa@redhat.com
State New
Headers show
Series
  • Acceptance test: Add "boot_linux" acceptance test
Related show

Commit Message

Cleber Rosa Nov. 4, 2019, 3:13 p.m. UTC
An Avocado Test ID[1] is composed by a number of components, but it
starts with the Test Name, usually a file system location that was
given to the loader.

Because the source directory is being given as a prefix to the
"tests/acceptance" directory containing the acceptance tests, the test
names will needlessly include the directory the user is using to host
the QEMU sources (and/or build tree).

Let's remove the source dir (or a build dir) from the path given to
the test loader.  This should give more constant names, and when using
result servers and databases, it should give the same test names
across executions from different people or from different directories.

[1] - https://avocado-framework.readthedocs.io/en/69.0/ReferenceGuide.html#test-id

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Comments

Philippe Mathieu-Daudé Nov. 4, 2019, 6:26 p.m. UTC | #1
On 11/4/19 4:13 PM, Cleber Rosa wrote:
> An Avocado Test ID[1] is composed by a number of components, but it
> starts with the Test Name, usually a file system location that was
> given to the loader.
> 
> Because the source directory is being given as a prefix to the
> "tests/acceptance" directory containing the acceptance tests, the test
> names will needlessly include the directory the user is using to host
> the QEMU sources (and/or build tree).
> 
> Let's remove the source dir (or a build dir) from the path given to
> the test loader.  This should give more constant names, and when using
> result servers and databases, it should give the same test names
> across executions from different people or from different directories.

Can we strip the full path to directory and only keep the filename in 
the database? (Thinking about out-of-tree tests).

> 
> [1] - https://avocado-framework.readthedocs.io/en/69.0/ReferenceGuide.html#test-id
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>   tests/Makefile.include | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 56f73b46e2..65e85f5275 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -1180,7 +1180,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>               --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>               --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
>               $(AVOCADO_TAGS) \
> -            --failfast=on $(SRC_PATH)/tests/acceptance, \
> +            --failfast=on tests/acceptance, \
>               "AVOCADO", "tests/acceptance")
>   
>   # Consolidated targets
>
Wainer dos Santos Moschetta Nov. 7, 2019, 6:52 p.m. UTC | #2
On 11/4/19 1:13 PM, Cleber Rosa wrote:
> An Avocado Test ID[1] is composed by a number of components, but it
> starts with the Test Name, usually a file system location that was
> given to the loader.
>
> Because the source directory is being given as a prefix to the
> "tests/acceptance" directory containing the acceptance tests, the test
> names will needlessly include the directory the user is using to host
> the QEMU sources (and/or build tree).
>
> Let's remove the source dir (or a build dir) from the path given to
> the test loader.  This should give more constant names, and when using
> result servers and databases, it should give the same test names
> across executions from different people or from different directories.

Completely agree.

>
> [1] - https://avocado-framework.readthedocs.io/en/69.0/ReferenceGuide.html#test-id
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   tests/Makefile.include | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)

Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>

- Wainer

>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 56f73b46e2..65e85f5275 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -1180,7 +1180,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR)
>               --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>               --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
>               $(AVOCADO_TAGS) \
> -            --failfast=on $(SRC_PATH)/tests/acceptance, \
> +            --failfast=on tests/acceptance, \
>               "AVOCADO", "tests/acceptance")
>   
>   # Consolidated targets
Cleber Rosa Nov. 11, 2019, 10:11 p.m. UTC | #3
On Mon, Nov 04, 2019 at 07:26:23PM +0100, Philippe Mathieu-Daudé wrote:
> On 11/4/19 4:13 PM, Cleber Rosa wrote:
> > An Avocado Test ID[1] is composed by a number of components, but it
> > starts with the Test Name, usually a file system location that was
> > given to the loader.
> > 
> > Because the source directory is being given as a prefix to the
> > "tests/acceptance" directory containing the acceptance tests, the test
> > names will needlessly include the directory the user is using to host
> > the QEMU sources (and/or build tree).
> > 
> > Let's remove the source dir (or a build dir) from the path given to
> > the test loader.  This should give more constant names, and when using
> > result servers and databases, it should give the same test names
> > across executions from different people or from different directories.
> 
> Can we strip the full path to directory and only keep the filename in the
> database? (Thinking about out-of-tree tests).
>

Yes, absolutely, but this needs to be done one the Avocado side.  TBH,
I have ideas to make this go even further, such as:

 1) the stripping of the "test_" prefix of the test method

 2) replace the full path to a directory with tests for a "test suite"
    alias (default to the directory name itself)

 3) test suite alias will be persisted on test result such as reports
    or machine, but ommited from the human UI

 4) full path to directory, exact version of test files (git hash) will
    all be metadata and not part of the test ID

Roughly speaking, something which is listed like this currently:

  $ avocado list tests/acceptance/
  INSTRUMENTED tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc
  ...

When executed, would be shown as:

  JOB ID     : fb885e9c3e7dc50534ec380a7e988cbf94233f07
  JOB LOG    : /home/cleber/avocado/job-results/job-2019-11-11T17.07-fb885e9/job.log
   (1/1) acceptance/boot_linux_console.py:BootLinuxConsole.x86_64_pc: PASS (2.17 s)
  RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 2.35 s

How does that sound?

- Cleber.
Philippe Mathieu-Daudé Nov. 12, 2019, 6:17 p.m. UTC | #4
On 11/11/19 11:11 PM, Cleber Rosa wrote:
> On Mon, Nov 04, 2019 at 07:26:23PM +0100, Philippe Mathieu-Daudé wrote:
>> On 11/4/19 4:13 PM, Cleber Rosa wrote:
>>> An Avocado Test ID[1] is composed by a number of components, but it
>>> starts with the Test Name, usually a file system location that was
>>> given to the loader.
>>>
>>> Because the source directory is being given as a prefix to the
>>> "tests/acceptance" directory containing the acceptance tests, the test
>>> names will needlessly include the directory the user is using to host
>>> the QEMU sources (and/or build tree).
>>>
>>> Let's remove the source dir (or a build dir) from the path given to
>>> the test loader.  This should give more constant names, and when using
>>> result servers and databases, it should give the same test names
>>> across executions from different people or from different directories.
>>
>> Can we strip the full path to directory and only keep the filename in the
>> database? (Thinking about out-of-tree tests).
>>
> 
> Yes, absolutely, but this needs to be done one the Avocado side.  TBH,
> I have ideas to make this go even further, such as:
> 
>   1) the stripping of the "test_" prefix of the test method
> 
>   2) replace the full path to a directory with tests for a "test suite"
>      alias (default to the directory name itself)
> 
>   3) test suite alias will be persisted on test result such as reports
>      or machine, but ommited from the human UI
> 
>   4) full path to directory, exact version of test files (git hash) will
>      all be metadata and not part of the test ID
> 
> Roughly speaking, something which is listed like this currently:
> 
>    $ avocado list tests/acceptance/
>    INSTRUMENTED tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc
>    ...
> 
> When executed, would be shown as:
> 
>    JOB ID     : fb885e9c3e7dc50534ec380a7e988cbf94233f07
>    JOB LOG    : /home/cleber/avocado/job-results/job-2019-11-11T17.07-fb885e9/job.log
>     (1/1) acceptance/boot_linux_console.py:BootLinuxConsole.x86_64_pc: PASS (2.17 s)

For the particular use case of QEMU, we can also strip the "acceptance/" 
part (and eventually ".py").

>    RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>    JOB TIME   : 2.35 s
> 
> How does that sound?
> 
> - Cleber.
>

Patch
diff mbox series

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 56f73b46e2..65e85f5275 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1180,7 +1180,7 @@  check-acceptance: check-venv $(TESTS_RESULTS_DIR)
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             --filter-by-tags-include-empty --filter-by-tags-include-empty-key \
             $(AVOCADO_TAGS) \
-            --failfast=on $(SRC_PATH)/tests/acceptance, \
+            --failfast=on tests/acceptance, \
             "AVOCADO", "tests/acceptance")
 
 # Consolidated targets