| Message ID | 20190202005610.24048-3-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | Acceptance Tests: target architecture support | expand |
On Fri, 1 Feb 2019 19:55:52 -0500 Cleber Rosa <crosa@redhat.com> wrote: > The current version of the "check-acceptance" target will only show > one line for execution of all tests. That's probably OK if the tests > to be run are quick enough and they're always the same. > > But, there's already one test alone that takes on average ~5 seconds > to run, we intend to adapt the list of tests to match the user's build > environment (among other choices). Btw: What are our expectations regarding execution time for tests? Especially if we continue adding tests, and architecture-specific tests are bound to be slower if run on a foreign architecture via tcg. Would a make check-acceptance-quick command make sense? ("I only want to verify quickly that I didn't break too much, so run the quicker tests only, probably only for my host architecture") > > Because of that, let's present the default Avocado UI by default. > Users can always choose a different output by setting the AVOCADO_SHOW > variable. > > Reviewed-by: Caio Carrara <ccarrara@redhat.com> > Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > Reviewed-by: Alex Bennée <alex.bennee@linaro.org> > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > .travis.yml | 2 +- > tests/Makefile.include | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-)
On 2/6/19 9:36 AM, Cornelia Huck wrote: > On Fri, 1 Feb 2019 19:55:52 -0500 > Cleber Rosa <crosa@redhat.com> wrote: > >> The current version of the "check-acceptance" target will only show >> one line for execution of all tests. That's probably OK if the tests >> to be run are quick enough and they're always the same. >> >> But, there's already one test alone that takes on average ~5 seconds >> to run, we intend to adapt the list of tests to match the user's build >> environment (among other choices). > > Btw: What are our expectations regarding execution time for tests? > Especially if we continue adding tests, and architecture-specific tests > are bound to be slower if run on a foreign architecture via tcg. > > Would a make check-acceptance-quick command make sense? ("I only want > to verify quickly that I didn't break too much, so run the quicker > tests only, probably only for my host architecture") > Yes, it definitely makes sense. Now, let me know if the following also makes sense to you: 1) Because these tests focus on functional testing, the default target/shortcut ("make check-acceptance") should run the complete set of test cases (including the slow ones). 2) Requirements vary greatly from user to user, to while adding a "check-acceptance-quick" is fine, you just mentioned one extra test execution variation ("for my host architecture"). For those, the idea is that: a) "make check-acceptance[-quick]" will adapt to the build environment (if you only built s390x targets, that's all it's going to use) b) "avocado" command line interface *should* be easy enough to fulfill other requirements, and not necessarily require a "make" target. For instance, if you're only interested in your host arch and one specific machine type, a command line such as the following should do the trick: $ make check-venv $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t machine:WHAT_I_CARE_ABOUT tests/acceptance/ How does that sound? Regards, - Cleber. >> >> Because of that, let's present the default Avocado UI by default. >> Users can always choose a different output by setting the AVOCADO_SHOW >> variable. >> >> Reviewed-by: Caio Carrara <ccarrara@redhat.com> >> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> >> Reviewed-by: Alex Bennée <alex.bennee@linaro.org> >> Signed-off-by: Cleber Rosa <crosa@redhat.com> >> --- >> .travis.yml | 2 +- >> tests/Makefile.include | 2 +- >> 2 files changed, 2 insertions(+), 2 deletions(-)
On Wed, 6 Feb 2019 12:02:55 -0500 Cleber Rosa <crosa@redhat.com> wrote: > On 2/6/19 9:36 AM, Cornelia Huck wrote: > > On Fri, 1 Feb 2019 19:55:52 -0500 > > Cleber Rosa <crosa@redhat.com> wrote: > > > >> The current version of the "check-acceptance" target will only show > >> one line for execution of all tests. That's probably OK if the tests > >> to be run are quick enough and they're always the same. > >> > >> But, there's already one test alone that takes on average ~5 seconds > >> to run, we intend to adapt the list of tests to match the user's build > >> environment (among other choices). > > > > Btw: What are our expectations regarding execution time for tests? > > Especially if we continue adding tests, and architecture-specific tests > > are bound to be slower if run on a foreign architecture via tcg. > > > > Would a make check-acceptance-quick command make sense? ("I only want > > to verify quickly that I didn't break too much, so run the quicker > > tests only, probably only for my host architecture") > > > > Yes, it definitely makes sense. Now, let me know if the following also > makes sense to you: > > 1) Because these tests focus on functional testing, the default > target/shortcut ("make check-acceptance") should run the complete set of > test cases (including the slow ones). Nod. > > 2) Requirements vary greatly from user to user, to while adding a > "check-acceptance-quick" is fine, you just mentioned one extra test > execution variation ("for my host architecture"). For those, the idea > is that: > > a) "make check-acceptance[-quick]" will adapt to the build environment > (if you only built s390x targets, that's all it's going to use) Yes. However, I usually build some extra targets (just to verify that files that e.g. include headers I modify don't break for other environments), but don't necessarily want to run all acceptance tests for them (as I don't expect functional changes for them). That's probably something I only want to do when I run the full set anyway. > > b) "avocado" command line interface *should* be easy enough to fulfill > other requirements, and not necessarily require a "make" target. For > instance, if you're only interested in your host arch and one specific > machine type, a command line such as the following should do the trick: > > $ make check-venv > $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t > machine:WHAT_I_CARE_ABOUT tests/acceptance/ > > How does that sound? Yes, that looks quite usable (I can easily define aliases locally for shortcuts) and solves the case I mentioned above. So, to summarize: - add 'make check-acceptance-quick' that excludes tests marked as 'slow' - use the avocado command line interface to further narrow down architectures and machines, if wanted ?
On 2/6/19 12:20 PM, Cornelia Huck wrote: > On Wed, 6 Feb 2019 12:02:55 -0500 > Cleber Rosa <crosa@redhat.com> wrote: > >> On 2/6/19 9:36 AM, Cornelia Huck wrote: >>> On Fri, 1 Feb 2019 19:55:52 -0500 >>> Cleber Rosa <crosa@redhat.com> wrote: >>> >>>> The current version of the "check-acceptance" target will only show >>>> one line for execution of all tests. That's probably OK if the tests >>>> to be run are quick enough and they're always the same. >>>> >>>> But, there's already one test alone that takes on average ~5 seconds >>>> to run, we intend to adapt the list of tests to match the user's build >>>> environment (among other choices). >>> >>> Btw: What are our expectations regarding execution time for tests? >>> Especially if we continue adding tests, and architecture-specific tests >>> are bound to be slower if run on a foreign architecture via tcg. >>> >>> Would a make check-acceptance-quick command make sense? ("I only want >>> to verify quickly that I didn't break too much, so run the quicker >>> tests only, probably only for my host architecture") >>> >> >> Yes, it definitely makes sense. Now, let me know if the following also >> makes sense to you: >> >> 1) Because these tests focus on functional testing, the default >> target/shortcut ("make check-acceptance") should run the complete set of >> test cases (including the slow ones). > > Nod. > >> >> 2) Requirements vary greatly from user to user, to while adding a >> "check-acceptance-quick" is fine, you just mentioned one extra test >> execution variation ("for my host architecture"). For those, the idea >> is that: >> >> a) "make check-acceptance[-quick]" will adapt to the build environment >> (if you only built s390x targets, that's all it's going to use) > > Yes. However, I usually build some extra targets (just to verify that > files that e.g. include headers I modify don't break for other > environments), but don't necessarily want to run all acceptance tests > for them (as I don't expect functional changes for them). That's > probably something I only want to do when I run the full set anyway. > Right, that's an interesting point I haven't thought of. >> >> b) "avocado" command line interface *should* be easy enough to fulfill >> other requirements, and not necessarily require a "make" target. For >> instance, if you're only interested in your host arch and one specific >> machine type, a command line such as the following should do the trick: >> >> $ make check-venv >> $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t >> machine:WHAT_I_CARE_ABOUT tests/acceptance/ >> >> How does that sound? > > Yes, that looks quite usable (I can easily define aliases locally for > shortcuts) and solves the case I mentioned above. > > So, to summarize: > - add 'make check-acceptance-quick' that excludes tests marked as 'slow' Yes... with one caveat: since the default is to have functional tests that are not necessarily quick, the other approach has been used. Tests that are known to be quick are currently marked as such. Anyway, this task is being tracked here: https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target > - use the avocado command line interface to further narrow down > architectures and machines, if wanted > ? > Yes. Do you think this deserves a section in the docs? Thanks! - Cleber.
On Wed, 6 Feb 2019 12:36:23 -0500 Cleber Rosa <crosa@redhat.com> wrote: > On 2/6/19 12:20 PM, Cornelia Huck wrote: > > So, to summarize: > > - add 'make check-acceptance-quick' that excludes tests marked as 'slow' > > Yes... with one caveat: since the default is to have functional tests > that are not necessarily quick, the other approach has been used. Tests > that are known to be quick are currently marked as such. > > Anyway, this task is being tracked here: > > https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target Yes, that sounds good as well. > > > - use the avocado command line interface to further narrow down > > architectures and machines, if wanted > > ? > > > > Yes. Do you think this deserves a section in the docs? I know that I'll find myself searching the docs for that info at some point in the future, so yes :)
On 2/7/19 5:25 AM, Cornelia Huck wrote: > On Wed, 6 Feb 2019 12:36:23 -0500 > Cleber Rosa <crosa@redhat.com> wrote: > >> On 2/6/19 12:20 PM, Cornelia Huck wrote: >>> So, to summarize: >>> - add 'make check-acceptance-quick' that excludes tests marked as 'slow' >> >> Yes... with one caveat: since the default is to have functional tests >> that are not necessarily quick, the other approach has been used. Tests >> that are known to be quick are currently marked as such. >> >> Anyway, this task is being tracked here: >> >> https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target > > Yes, that sounds good as well. > >> >>> - use the avocado command line interface to further narrow down >>> architectures and machines, if wanted >>> ? >>> >> >> Yes. Do you think this deserves a section in the docs? > > I know that I'll find myself searching the docs for that info at some > point in the future, so yes :) > Sure thing, tracking it here: https://trello.com/c/7Fxbtx6F/80-document-how-to-use-the-avocado-command-line-and-tags Regards, - Cleber.
On Thu, 7 Feb 2019 13:32:54 -0500 Cleber Rosa <crosa@redhat.com> wrote: > On 2/7/19 5:25 AM, Cornelia Huck wrote: > > On Wed, 6 Feb 2019 12:36:23 -0500 > > Cleber Rosa <crosa@redhat.com> wrote: > > > >> On 2/6/19 12:20 PM, Cornelia Huck wrote: > >>> So, to summarize: > >>> - add 'make check-acceptance-quick' that excludes tests marked as 'slow' > >> > >> Yes... with one caveat: since the default is to have functional tests > >> that are not necessarily quick, the other approach has been used. Tests > >> that are known to be quick are currently marked as such. > >> > >> Anyway, this task is being tracked here: > >> > >> https://trello.com/c/9HZWfKeE/79-add-a-make-check-acceptance-quick-target > > > > Yes, that sounds good as well. > > > >> > >>> - use the avocado command line interface to further narrow down > >>> architectures and machines, if wanted > >>> ? > >>> > >> > >> Yes. Do you think this deserves a section in the docs? > > > > I know that I'll find myself searching the docs for that info at some > > point in the future, so yes :) > > > > Sure thing, tracking it here: > > https://trello.com/c/7Fxbtx6F/80-document-how-to-use-the-avocado-command-line-and-tags Great, thanks!
diff --git a/.travis.yml b/.travis.yml index 87d9fa971c..1b43ed4df6 100644 --- a/.travis.yml +++ b/.travis.yml @@ -190,7 +190,7 @@ matrix: # Acceptance (Functional) tests - env: - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu" - - TEST_CMD="make AVOCADO_SHOW=app check-acceptance" + - TEST_CMD="make check-acceptance" addons: apt: packages: diff --git a/tests/Makefile.include b/tests/Makefile.include index 75ad9c0dd3..67e222eeff 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -1093,7 +1093,7 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results # Controls the output generated by Avocado when running tests. # Any number of command separated loggers are accepted. For more # information please refer to "avocado --help". -AVOCADO_SHOW=none +AVOCADO_SHOW=app ifneq ($(findstring v2,"v$(PYTHON_VERSION)"),v2) $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)