| Message ID | 20191104151323.9883-5-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | Acceptance test: Add "boot_linux" acceptance test | expand |
On 11/4/19 4:13 PM, Cleber Rosa wrote: > An Avocado Test ID[1] is composed by a number of components, but it > starts with the Test Name, usually a file system location that was > given to the loader. > > Because the source directory is being given as a prefix to the > "tests/acceptance" directory containing the acceptance tests, the test > names will needlessly include the directory the user is using to host > the QEMU sources (and/or build tree). > > Let's remove the source dir (or a build dir) from the path given to > the test loader. This should give more constant names, and when using > result servers and databases, it should give the same test names > across executions from different people or from different directories. Can we strip the full path to directory and only keep the filename in the database? (Thinking about out-of-tree tests). > > [1] - https://avocado-framework.readthedocs.io/en/69.0/ReferenceGuide.html#test-id > > Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > --- > tests/Makefile.include | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 56f73b46e2..65e85f5275 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -1180,7 +1180,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > $(AVOCADO_TAGS) \ > - --failfast=on $(SRC_PATH)/tests/acceptance, \ > + --failfast=on tests/acceptance, \ > "AVOCADO", "tests/acceptance") > > # Consolidated targets >
On 11/4/19 1:13 PM, Cleber Rosa wrote: > An Avocado Test ID[1] is composed by a number of components, but it > starts with the Test Name, usually a file system location that was > given to the loader. > > Because the source directory is being given as a prefix to the > "tests/acceptance" directory containing the acceptance tests, the test > names will needlessly include the directory the user is using to host > the QEMU sources (and/or build tree). > > Let's remove the source dir (or a build dir) from the path given to > the test loader. This should give more constant names, and when using > result servers and databases, it should give the same test names > across executions from different people or from different directories. Completely agree. > > [1] - https://avocado-framework.readthedocs.io/en/69.0/ReferenceGuide.html#test-id > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/Makefile.include | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> - Wainer > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 56f73b46e2..65e85f5275 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -1180,7 +1180,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > $(AVOCADO_TAGS) \ > - --failfast=on $(SRC_PATH)/tests/acceptance, \ > + --failfast=on tests/acceptance, \ > "AVOCADO", "tests/acceptance") > > # Consolidated targets
On Mon, Nov 04, 2019 at 07:26:23PM +0100, Philippe Mathieu-Daudé wrote: > On 11/4/19 4:13 PM, Cleber Rosa wrote: > > An Avocado Test ID[1] is composed by a number of components, but it > > starts with the Test Name, usually a file system location that was > > given to the loader. > > > > Because the source directory is being given as a prefix to the > > "tests/acceptance" directory containing the acceptance tests, the test > > names will needlessly include the directory the user is using to host > > the QEMU sources (and/or build tree). > > > > Let's remove the source dir (or a build dir) from the path given to > > the test loader. This should give more constant names, and when using > > result servers and databases, it should give the same test names > > across executions from different people or from different directories. > > Can we strip the full path to directory and only keep the filename in the > database? (Thinking about out-of-tree tests). > Yes, absolutely, but this needs to be done one the Avocado side. TBH, I have ideas to make this go even further, such as: 1) the stripping of the "test_" prefix of the test method 2) replace the full path to a directory with tests for a "test suite" alias (default to the directory name itself) 3) test suite alias will be persisted on test result such as reports or machine, but ommited from the human UI 4) full path to directory, exact version of test files (git hash) will all be metadata and not part of the test ID Roughly speaking, something which is listed like this currently: $ avocado list tests/acceptance/ INSTRUMENTED tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc ... When executed, would be shown as: JOB ID : fb885e9c3e7dc50534ec380a7e988cbf94233f07 JOB LOG : /home/cleber/avocado/job-results/job-2019-11-11T17.07-fb885e9/job.log (1/1) acceptance/boot_linux_console.py:BootLinuxConsole.x86_64_pc: PASS (2.17 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 2.35 s How does that sound? - Cleber.
On 11/11/19 11:11 PM, Cleber Rosa wrote: > On Mon, Nov 04, 2019 at 07:26:23PM +0100, Philippe Mathieu-Daudé wrote: >> On 11/4/19 4:13 PM, Cleber Rosa wrote: >>> An Avocado Test ID[1] is composed by a number of components, but it >>> starts with the Test Name, usually a file system location that was >>> given to the loader. >>> >>> Because the source directory is being given as a prefix to the >>> "tests/acceptance" directory containing the acceptance tests, the test >>> names will needlessly include the directory the user is using to host >>> the QEMU sources (and/or build tree). >>> >>> Let's remove the source dir (or a build dir) from the path given to >>> the test loader. This should give more constant names, and when using >>> result servers and databases, it should give the same test names >>> across executions from different people or from different directories. >> >> Can we strip the full path to directory and only keep the filename in the >> database? (Thinking about out-of-tree tests). >> > > Yes, absolutely, but this needs to be done one the Avocado side. TBH, > I have ideas to make this go even further, such as: > > 1) the stripping of the "test_" prefix of the test method > > 2) replace the full path to a directory with tests for a "test suite" > alias (default to the directory name itself) > > 3) test suite alias will be persisted on test result such as reports > or machine, but ommited from the human UI > > 4) full path to directory, exact version of test files (git hash) will > all be metadata and not part of the test ID > > Roughly speaking, something which is listed like this currently: > > $ avocado list tests/acceptance/ > INSTRUMENTED tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc > ... > > When executed, would be shown as: > > JOB ID : fb885e9c3e7dc50534ec380a7e988cbf94233f07 > JOB LOG : /home/cleber/avocado/job-results/job-2019-11-11T17.07-fb885e9/job.log > (1/1) acceptance/boot_linux_console.py:BootLinuxConsole.x86_64_pc: PASS (2.17 s) For the particular use case of QEMU, we can also strip the "acceptance/" part (and eventually ".py"). > RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 2.35 s > > How does that sound? > > - Cleber. >
diff --git a/tests/Makefile.include b/tests/Makefile.include index 56f73b46e2..65e85f5275 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -1180,7 +1180,7 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ $(AVOCADO_TAGS) \ - --failfast=on $(SRC_PATH)/tests/acceptance, \ + --failfast=on tests/acceptance, \ "AVOCADO", "tests/acceptance") # Consolidated targets
An Avocado Test ID[1] is composed by a number of components, but it starts with the Test Name, usually a file system location that was given to the loader. Because the source directory is being given as a prefix to the "tests/acceptance" directory containing the acceptance tests, the test names will needlessly include the directory the user is using to host the QEMU sources (and/or build tree). Let's remove the source dir (or a build dir) from the path given to the test loader. This should give more constant names, and when using result servers and databases, it should give the same test names across executions from different people or from different directories. [1] - https://avocado-framework.readthedocs.io/en/69.0/ReferenceGuide.html#test-id Signed-off-by: Cleber Rosa <crosa@redhat.com> --- tests/Makefile.include | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)