| Message ID | 20230330101141.30199-10-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | more misc fixes for 8.0 (tests, gdbstub, meta, docs) | expand |
On 30/03/2023 12.11, Alex Bennée wrote: > From: Kautuk Consul <kconsul@linux.vnet.ibm.com> > > Avocado version 101.0 has a fix to re-compute the checksum > of an asset file if the algorithm used in the *-CHECKSUM > file isn't the same as the one being passed to it by the > avocado user (i.e. the avocado_qemu python module). > In the earlier avocado versions this fix wasn't there due > to which if the checksum wouldn't match the earlier > checksum (calculated by a different algorithm), the avocado > code would start downloading a fresh image from the internet > URL thus making the test-cases take longer to execute. > > Bump up the avocado-framework version to 101.0. > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> > Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> > Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> > --- > tests/requirements.txt | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/tests/requirements.txt b/tests/requirements.txt > index 0ba561b6bd..a6f73da681 100644 > --- a/tests/requirements.txt > +++ b/tests/requirements.txt > @@ -2,5 +2,5 @@ > # in the tests/venv Python virtual environment. For more info, > # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 > # Note that qemu.git/python/ is always implicitly installed. > -avocado-framework==88.1 > +avocado-framework==101.0 > pycdlib==1.11.0 Did you check whether the same amount of avocado tests still works as before? ... last time I tried to bump the version, a lot of things were failing, and I think Cleber was recently working on fixing things, but I haven't heart anything back from him yet that it would be OK to bump to a newer version now ... So upgrading to a new version of Avocado during the softfreeze sounds somewhat risky to me right now - I'd appreciate if we could do that after the release instead. Thomas
Thomas Huth <thuth@redhat.com> writes: > On 30/03/2023 12.11, Alex Bennée wrote: >> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> Avocado version 101.0 has a fix to re-compute the checksum >> of an asset file if the algorithm used in the *-CHECKSUM >> file isn't the same as the one being passed to it by the >> avocado user (i.e. the avocado_qemu python module). >> In the earlier avocado versions this fix wasn't there due >> to which if the checksum wouldn't match the earlier >> checksum (calculated by a different algorithm), the avocado >> code would start downloading a fresh image from the internet >> URL thus making the test-cases take longer to execute. >> Bump up the avocado-framework version to 101.0. >> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >> --- >> tests/requirements.txt | 2 +- >> 1 file changed, 1 insertion(+), 1 deletion(-) >> diff --git a/tests/requirements.txt b/tests/requirements.txt >> index 0ba561b6bd..a6f73da681 100644 >> --- a/tests/requirements.txt >> +++ b/tests/requirements.txt >> @@ -2,5 +2,5 @@ >> # in the tests/venv Python virtual environment. For more info, >> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >> # Note that qemu.git/python/ is always implicitly installed. >> -avocado-framework==88.1 >> +avocado-framework==101.0 >> pycdlib==1.11.0 > > Did you check whether the same amount of avocado tests still works as > before? ... last time I tried to bump the version, a lot of things > were failing, and I think Cleber was recently working on fixing > things, but I haven't heart anything back from him yet that it would > be OK to bump to a newer version now ... I ran it on my default build and the only failure was: (008/222) tests/avocado/boot_linux.py:BootLinuxS390X.test_s390_ccw_virtio_tcg: INTERRUPTED: timeout (240.01 s) which passed on a retry. But now I realise with failfast it skipped a bunch: RESULTS : PASS 46 | ERROR 0 | FAIL 0 | SKIP 174 | WARN 0 | INTERRUPT 1 | CANCEL 1 JOB TIME : 290.26 s > So upgrading to a new version of Avocado during the softfreeze sounds > somewhat risky to me right now - I'd appreciate if we could do that > after the release instead. Sure. I was hoping we would speed up avocado a little by avoiding double downloads. > > Thomas
On 30/03/2023 14.12, Alex Bennée wrote: > > Thomas Huth <thuth@redhat.com> writes: > >> On 30/03/2023 12.11, Alex Bennée wrote: >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>> Avocado version 101.0 has a fix to re-compute the checksum >>> of an asset file if the algorithm used in the *-CHECKSUM >>> file isn't the same as the one being passed to it by the >>> avocado user (i.e. the avocado_qemu python module). >>> In the earlier avocado versions this fix wasn't there due >>> to which if the checksum wouldn't match the earlier >>> checksum (calculated by a different algorithm), the avocado >>> code would start downloading a fresh image from the internet >>> URL thus making the test-cases take longer to execute. >>> Bump up the avocado-framework version to 101.0. >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >>> --- >>> tests/requirements.txt | 2 +- >>> 1 file changed, 1 insertion(+), 1 deletion(-) >>> diff --git a/tests/requirements.txt b/tests/requirements.txt >>> index 0ba561b6bd..a6f73da681 100644 >>> --- a/tests/requirements.txt >>> +++ b/tests/requirements.txt >>> @@ -2,5 +2,5 @@ >>> # in the tests/venv Python virtual environment. For more info, >>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >>> # Note that qemu.git/python/ is always implicitly installed. >>> -avocado-framework==88.1 >>> +avocado-framework==101.0 >>> pycdlib==1.11.0 >> >> Did you check whether the same amount of avocado tests still works as >> before? ... last time I tried to bump the version, a lot of things >> were failing, and I think Cleber was recently working on fixing >> things, but I haven't heart anything back from him yet that it would >> be OK to bump to a newer version now ... > > I ran it on my default build and the only failure was: > > (008/222) tests/avocado/boot_linux.py:BootLinuxS390X.test_s390_ccw_virtio_tcg: INTERRUPTED: timeout (240.01 s) > > which passed on a retry. But now I realise with failfast it skipped a bunch: That one is also failing for me here when I apply the patch. Without the patch, the test is working fine. I think this needs more careful testing first - e.g. the tests are run in parallel now by default, which breaks a lot of our timeout settings. Thomas
On 30/03/2023 14.21, Thomas Huth wrote: > On 30/03/2023 14.12, Alex Bennée wrote: >> >> Thomas Huth <thuth@redhat.com> writes: >> >>> On 30/03/2023 12.11, Alex Bennée wrote: >>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>>> Avocado version 101.0 has a fix to re-compute the checksum >>>> of an asset file if the algorithm used in the *-CHECKSUM >>>> file isn't the same as the one being passed to it by the >>>> avocado user (i.e. the avocado_qemu python module). >>>> In the earlier avocado versions this fix wasn't there due >>>> to which if the checksum wouldn't match the earlier >>>> checksum (calculated by a different algorithm), the avocado >>>> code would start downloading a fresh image from the internet >>>> URL thus making the test-cases take longer to execute. >>>> Bump up the avocado-framework version to 101.0. >>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com> >>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com> >>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com> >>>> --- >>>> tests/requirements.txt | 2 +- >>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>> diff --git a/tests/requirements.txt b/tests/requirements.txt >>>> index 0ba561b6bd..a6f73da681 100644 >>>> --- a/tests/requirements.txt >>>> +++ b/tests/requirements.txt >>>> @@ -2,5 +2,5 @@ >>>> # in the tests/venv Python virtual environment. For more info, >>>> # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 >>>> # Note that qemu.git/python/ is always implicitly installed. >>>> -avocado-framework==88.1 >>>> +avocado-framework==101.0 >>>> pycdlib==1.11.0 >>> >>> Did you check whether the same amount of avocado tests still works as >>> before? ... last time I tried to bump the version, a lot of things >>> were failing, and I think Cleber was recently working on fixing >>> things, but I haven't heart anything back from him yet that it would >>> be OK to bump to a newer version now ... >> >> I ran it on my default build and the only failure was: >> >> (008/222) >> tests/avocado/boot_linux.py:BootLinuxS390X.test_s390_ccw_virtio_tcg: >> INTERRUPTED: timeout (240.01 s) >> >> which passed on a retry. But now I realise with failfast it skipped a bunch: > > That one is also failing for me here when I apply the patch. Without the > patch, the test is working fine. I think this needs more careful testing > first - e.g. the tests are run in parallel now by default, which breaks a > lot of our timeout settings. FWIW, I think we likely want something like this added to this patch, so we avoid to run those tests in parallel (unless requested with -jX): diff a/tests/Makefile.include b/tests/Makefile.include --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -138,12 +138,15 @@ get-vm-image-fedora-31-%: check-venv # download all vm images, according to defined targets get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD)) +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%, $(MAKEFLAGS)))) + check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images $(call quiet-command, \ $(TESTS_PYTHON) -m avocado \ --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \ $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \ --filter-by-tags-include-empty-key) \ + --max-parallel-tasks $(JOBS_OPTION:-j%=%) \ $(AVOCADO_CMDLINE_TAGS) \ $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \ "AVOCADO", "tests/avocado") That way we can avoid the timeout problems unless we found a proper solution for those. Thomas
diff --git a/tests/requirements.txt b/tests/requirements.txt index 0ba561b6bd..a6f73da681 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -2,5 +2,5 @@ # in the tests/venv Python virtual environment. For more info, # refer to: https://pip.pypa.io/en/stable/user_guide/#id1 # Note that qemu.git/python/ is always implicitly installed. -avocado-framework==88.1 +avocado-framework==101.0 pycdlib==1.11.0