Message ID | 20210707152454.246519-1-willianr@redhat.com |
---|---|
State | New |
Headers | show |
Series | [v2] tests/acceptance: allow control over tags during check-acceptance | expand |
Hi, On 7/7/21 12:24 PM, Willian Rampazzo wrote: > Although it is possible to run a specific test using the avocado > command-line, a user may want to use a specific tag while running the > ``make check-acceptance`` during the development or debugging. I always wanted that feature, thanks for implementing it! > > This allows using the AVOCADO_TAGS environment variable where the user > takes total control of which tests should run based on the tags defined. > > This also makes the check-acceptance command flexible to restrict tests > based on tags while running on CI. > > e.g.: > > AVOCADO_TAGS="foo bar baz" make check-acceptance If I run `AVOCADO_TAGS="machine:malta arch:mips" make check-acceptance` it is going to get the tests with 'machine:malta' *or* 'arch:mips' tags, whereas with AVOCADO_TAGS="machine:malta,arch:mips" it gets those with 'machine:malta' *and* 'arch:mips', i.e., the semantic is different. I think it deserves a mention in the docs/devel/testing.rst, maybe linking to Avocado's doc [1] as well. [1] https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > --- > docs/devel/testing.rst | 8 ++++++++ > tests/Makefile.include | 12 +++++++++--- > 2 files changed, 17 insertions(+), 3 deletions(-) Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> > > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > index 4e42392810..f6e4fc08a7 100644 > --- a/docs/devel/testing.rst > +++ b/docs/devel/testing.rst > @@ -760,6 +760,14 @@ in the current directory, tagged as "quick", run: > > avocado run -t quick . > > +It is also possible to run tests based on tags using the > +``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment > +variable: > + > +.. code:: > + > + AVOCADO_TAGS=quick make check-acceptance > + > The ``avocado_qemu.Test`` base test class > ----------------------------------------- > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 8f220e15d1..d464f06e4a 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -92,7 +92,12 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > # Any number of command separated loggers are accepted. For more > # information please refer to "avocado --help". > AVOCADO_SHOW=app > -AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS))) > +ifndef AVOCADO_TAGS > + AVOCADO_CMDLINE_TAGS=$(patsubst %-softmmu,-t arch:%, \ > + $(filter %-softmmu,$(TARGETS))) > +else > + AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS)) > +endif > > $(TESTS_VENV_DIR): $(TESTS_VENV_REQ) > $(call quiet-command, \ > @@ -128,8 +133,9 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images > $(call quiet-command, \ > $(TESTS_VENV_DIR)/bin/python -m avocado \ > --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ > - --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ > - $(AVOCADO_TAGS) \ > + $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ > + --filter-by-tags-include-empty-key) \ > + $(AVOCADO_CMDLINE_TAGS) \ > $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ > "AVOCADO", "tests/acceptance") >
Hi, Wainer, On Fri, Jul 9, 2021 at 4:32 PM Wainer dos Santos Moschetta <wainersm@redhat.com> wrote: > > Hi, > > On 7/7/21 12:24 PM, Willian Rampazzo wrote: > > Although it is possible to run a specific test using the avocado > > command-line, a user may want to use a specific tag while running the > > ``make check-acceptance`` during the development or debugging. > I always wanted that feature, thanks for implementing it! I'm glad it is helpful. > > > > This allows using the AVOCADO_TAGS environment variable where the user > > takes total control of which tests should run based on the tags defined. > > > > This also makes the check-acceptance command flexible to restrict tests > > based on tags while running on CI. > > > > e.g.: > > > > AVOCADO_TAGS="foo bar baz" make check-acceptance > > If I run `AVOCADO_TAGS="machine:malta arch:mips" make check-acceptance` > it is going to get the tests with 'machine:malta' *or* 'arch:mips' tags, > whereas with AVOCADO_TAGS="machine:malta,arch:mips" it gets those with > 'machine:malta' *and* 'arch:mips', i.e., the semantic is different. I > think it deserves a mention in the docs/devel/testing.rst, maybe linking > to Avocado's doc [1] as well. Okay, that is fair. I'll summarize something in the testing documentation and point to the Avocado doc to avoid duplication. > > [1] > https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html > > > > > Signed-off-by: Willian Rampazzo <willianr@redhat.com> > > --- > > docs/devel/testing.rst | 8 ++++++++ > > tests/Makefile.include | 12 +++++++++--- > > 2 files changed, 17 insertions(+), 3 deletions(-) > > Tested-by: Wainer dos Santos Moschetta <wainersm@redhat.com> > > Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> > Thanks for testing and reviewing.
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst index 4e42392810..f6e4fc08a7 100644 --- a/docs/devel/testing.rst +++ b/docs/devel/testing.rst @@ -760,6 +760,14 @@ in the current directory, tagged as "quick", run: avocado run -t quick . +It is also possible to run tests based on tags using the +``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment +variable: + +.. code:: + + AVOCADO_TAGS=quick make check-acceptance + The ``avocado_qemu.Test`` base test class ----------------------------------------- diff --git a/tests/Makefile.include b/tests/Makefile.include index 8f220e15d1..d464f06e4a 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -92,7 +92,12 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help". AVOCADO_SHOW=app -AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS))) +ifndef AVOCADO_TAGS + AVOCADO_CMDLINE_TAGS=$(patsubst %-softmmu,-t arch:%, \ + $(filter %-softmmu,$(TARGETS))) +else + AVOCADO_CMDLINE_TAGS=$(addprefix -t , $(AVOCADO_TAGS)) +endif $(TESTS_VENV_DIR): $(TESTS_VENV_REQ) $(call quiet-command, \ @@ -128,8 +133,9 @@ check-acceptance: check-venv $(TESTS_RESULTS_DIR) get-vm-images $(call quiet-command, \ $(TESTS_VENV_DIR)/bin/python -m avocado \ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ - --filter-by-tags-include-empty --filter-by-tags-include-empty-key \ - $(AVOCADO_TAGS) \ + $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ + --filter-by-tags-include-empty-key) \ + $(AVOCADO_CMDLINE_TAGS) \ $(if $(GITLAB_CI),,--failfast) tests/acceptance, \ "AVOCADO", "tests/acceptance")
Although it is possible to run a specific test using the avocado command-line, a user may want to use a specific tag while running the ``make check-acceptance`` during the development or debugging. This allows using the AVOCADO_TAGS environment variable where the user takes total control of which tests should run based on the tags defined. This also makes the check-acceptance command flexible to restrict tests based on tags while running on CI. e.g.: AVOCADO_TAGS="foo bar baz" make check-acceptance Signed-off-by: Willian Rampazzo <willianr@redhat.com> --- docs/devel/testing.rst | 8 ++++++++ tests/Makefile.include | 12 +++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-)