diff mbox series

[v4,1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

Message ID 20230417134321.3627231-2-alex.bennee@linaro.org
State New
Headers show
Series testing/next: avocado, docs, gitlab | expand

Commit Message

Alex Bennée April 17, 2023, 1:43 p.m. UTC 
From: Kautuk Consul <kconsul@linux.vnet.ibm.com>

Avocado version 101.0 has a fix to re-compute the checksum
of an asset file if the algorithm used in the *-CHECKSUM
file isn't the same as the one being passed to it by the
avocado user (i.e. the avocado_qemu python module).
In the earlier avocado versions this fix wasn't there due
to which if the checksum wouldn't match the earlier
checksum (calculated by a different algorithm), the avocado
code would start downloading a fresh image from the internet
URL thus making the test-cases take longer to execute.

Bump up the avocado-framework version to 101.0.

Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>

---
v2
  - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
---
 tests/Makefile.include | 18 +++++++++++-------
 tests/requirements.txt |  2 +-
 2 files changed, 12 insertions(+), 8 deletions(-)

Comments

John Snow April 17, 2023, 4:50 p.m. UTC | #1 
On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
>   - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>  tests/Makefile.include | 18 +++++++++++-------
>  tests/requirements.txt |  2 +-
>  2 files changed, 12 insertions(+), 8 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>  # download all vm images, according to defined targets
>  get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
>  check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> -       $(call quiet-command, \
> -            $(TESTS_PYTHON) -m avocado \
> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> -                       --filter-by-tags-include-empty-key) \
> -            $(AVOCADO_CMDLINE_TAGS) \
> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> +       $(call quiet-command,                                                   \
> +            $(TESTS_PYTHON) -m avocado                                                 \
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
> +            $(if $(AVOCADO_TAGS),,                                             \
> +                       --filter-by-tags-include-empty                          \
> +                       --filter-by-tags-include-empty-key)                     \
> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
> +            $(AVOCADO_CMDLINE_TAGS)                                            \
> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>              "AVOCADO", "tests/avocado")
>
>  check-acceptance-deprecated-warning:
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 0ba561b6bd..a6f73da681 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,5 +2,5 @@
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>  # Note that qemu.git/python/ is always implicitly installed.
> -avocado-framework==88.1
> +avocado-framework==101.0
>  pycdlib==1.11.0
> --
> 2.39.2
>

I thought there were test failures that prohibited us from bumping the
Avocado-Framework version. Did those get rectified recently?
Alex Bennée April 17, 2023, 5:44 p.m. UTC | #2 
John Snow <jsnow@redhat.com> writes:

> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>>   - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>>  tests/Makefile.include | 18 +++++++++++-------
>>  tests/requirements.txt |  2 +-
>>  2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>  # download all vm images, according to defined targets
>>  get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>
>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>> +
>>  check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> -       $(call quiet-command, \
>> -            $(TESTS_PYTHON) -m avocado \
>> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> -                       --filter-by-tags-include-empty-key) \
>> -            $(AVOCADO_CMDLINE_TAGS) \
>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> +       $(call quiet-command,                                                   \
>> +            $(TESTS_PYTHON) -m avocado                                                 \
>> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
>> +            $(if $(AVOCADO_TAGS),,                                             \
>> +                       --filter-by-tags-include-empty                          \
>> +                       --filter-by-tags-include-empty-key)                     \
>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
>> +            $(AVOCADO_CMDLINE_TAGS)                                            \
>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>>              "AVOCADO", "tests/avocado")
>>
>>  check-acceptance-deprecated-warning:
>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>> index 0ba561b6bd..a6f73da681 100644
>> --- a/tests/requirements.txt
>> +++ b/tests/requirements.txt
>> @@ -2,5 +2,5 @@
>>  # in the tests/venv Python virtual environment. For more info,
>>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>  # Note that qemu.git/python/ is always implicitly installed.
>> -avocado-framework==88.1
>> +avocado-framework==101.0
>>  pycdlib==1.11.0
>> --
>> 2.39.2
>>
>
> I thought there were test failures that prohibited us from bumping the
> Avocado-Framework version. Did those get rectified recently?

I'm not seeing any with your patch applied.
Thomas Huth April 18, 2023, 7:10 a.m. UTC | #3 
On 17/04/2023 15.43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> 
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
> 
> Bump up the avocado-framework version to 101.0.
> 
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> 
> ---
> v2
>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>   tests/Makefile.include | 18 +++++++++++-------
>   tests/requirements.txt |  2 +-
>   2 files changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>   # download all vm images, according to defined targets
>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>   
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> -	$(call quiet-command, \
> -            $(TESTS_PYTHON) -m avocado \
> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> -			--filter-by-tags-include-empty-key) \
> -            $(AVOCADO_CMDLINE_TAGS) \
> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> +	$(call quiet-command, 							\
> +            $(TESTS_PYTHON) -m avocado 						\
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
> +            $(if $(AVOCADO_TAGS),, 						\
> +			--filter-by-tags-include-empty 				\
> +			--filter-by-tags-include-empty-key) 			\
> +		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
> +            $(AVOCADO_CMDLINE_TAGS) 						\
> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\

We might need to revisit this --failfast logic, too. If I've got that right, 
failfast is now the default with the new system? So we might want to disable 
it by default again if GITLAB_CI is not set?

  Thomas
Philippe Mathieu-Daudé April 20, 2023, 9:18 a.m. UTC | #4 
On 17/4/23 19:44, Alex Bennée wrote:
> 
> John Snow <jsnow@redhat.com> writes:
> 
>> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the
>>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due
>>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado
>>> code would start downloading a fresh image from the internet
>>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>> ---
>>>   tests/Makefile.include | 18 +++++++++++-------
>>>   tests/requirements.txt |  2 +-
>>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>   # download all vm images, according to defined targets
>>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>>
>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>>> +
>>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> -       $(call quiet-command, \
>>> -            $(TESTS_PYTHON) -m avocado \
>>> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>> -                       --filter-by-tags-include-empty-key) \
>>> -            $(AVOCADO_CMDLINE_TAGS) \
>>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> +       $(call quiet-command,                                                   \
>>> +            $(TESTS_PYTHON) -m avocado                                                 \
>>> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
>>> +            $(if $(AVOCADO_TAGS),,                                             \
>>> +                       --filter-by-tags-include-empty                          \
>>> +                       --filter-by-tags-include-empty-key)                     \
>>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
>>> +            $(AVOCADO_CMDLINE_TAGS)                                            \
>>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>>>               "AVOCADO", "tests/avocado")
>>>
>>>   check-acceptance-deprecated-warning:
>>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>> index 0ba561b6bd..a6f73da681 100644
>>> --- a/tests/requirements.txt
>>> +++ b/tests/requirements.txt
>>> @@ -2,5 +2,5 @@
>>>   # in the tests/venv Python virtual environment. For more info,
>>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>   # Note that qemu.git/python/ is always implicitly installed.
>>> -avocado-framework==88.1
>>> +avocado-framework==101.0
>>>   pycdlib==1.11.0
>>> --
>>> 2.39.2
>>>
>>
>> I thought there were test failures that prohibited us from bumping the
>> Avocado-Framework version. Did those get rectified recently?

No, still not working on Darwin. At this point I'm thinking at sending
a patch clarifying testing QEMU with Avocado is not supported on Darwin.

> I'm not seeing any with your patch applied.

Are you using running the Cirrus-CI jobs?
John Snow April 20, 2023, 4:54 p.m. UTC | #5 
On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé <philmd@linaro.org>
wrote:

> On 17/4/23 19:44, Alex Bennée wrote:
> >
> > John Snow <jsnow@redhat.com> writes:
> >
> >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org>
> wrote:
> >>>
> >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>>
> >>> Avocado version 101.0 has a fix to re-compute the checksum
> >>> of an asset file if the algorithm used in the *-CHECKSUM
> >>> file isn't the same as the one being passed to it by the
> >>> avocado user (i.e. the avocado_qemu python module).
> >>> In the earlier avocado versions this fix wasn't there due
> >>> to which if the checksum wouldn't match the earlier
> >>> checksum (calculated by a different algorithm), the avocado
> >>> code would start downloading a fresh image from the internet
> >>> URL thus making the test-cases take longer to execute.
> >>>
> >>> Bump up the avocado-framework version to 101.0.
> >>>
> >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> >>>
> >>> ---
> >>> v2
> >>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> >>> ---
> >>>   tests/Makefile.include | 18 +++++++++++-------
> >>>   tests/requirements.txt |  2 +-
> >>>   2 files changed, 12 insertions(+), 8 deletions(-)
> >>>
> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>> index 9422ddaece..a4de0ad5a2 100644
> >>> --- a/tests/Makefile.include
> >>> +++ b/tests/Makefile.include
> >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
> >>>   # download all vm images, according to defined targets
> >>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%,
> $(FEDORA_31_DOWNLOAD))
> >>>
> >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
> -j%,$(MAKEFLAGS))))
> >>> +
> >>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> >>> -       $(call quiet-command, \
> >>> -            $(TESTS_PYTHON) -m avocado \
> >>> -            --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR) \
> >>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> >>> -                       --filter-by-tags-include-empty-key) \
> >>> -            $(AVOCADO_CMDLINE_TAGS) \
> >>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> >>> +       $(call quiet-command,
>          \
> >>> +            $(TESTS_PYTHON) -m avocado
>                  \
> >>> +            --show=$(AVOCADO_SHOW) run
> --job-results-dir=$(TESTS_RESULTS_DIR)  \
> >>> +            $(if $(AVOCADO_TAGS),,
>          \
> >>> +                       --filter-by-tags-include-empty
>           \
> >>> +                       --filter-by-tags-include-empty-key)
>          \
> >>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>          \
> >>> +            $(AVOCADO_CMDLINE_TAGS)
>           \
> >>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),
>          \
> >>>               "AVOCADO", "tests/avocado")
> >>>
> >>>   check-acceptance-deprecated-warning:
> >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
> >>> index 0ba561b6bd..a6f73da681 100644
> >>> --- a/tests/requirements.txt
> >>> +++ b/tests/requirements.txt
> >>> @@ -2,5 +2,5 @@
> >>>   # in the tests/venv Python virtual environment. For more info,
> >>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> >>>   # Note that qemu.git/python/ is always implicitly installed.
> >>> -avocado-framework==88.1
> >>> +avocado-framework==101.0
> >>>   pycdlib==1.11.0
> >>> --
> >>> 2.39.2
> >>>
> >>
> >> I thought there were test failures that prohibited us from bumping the
> >> Avocado-Framework version. Did those get rectified recently?
>
> No, still not working on Darwin. At this point I'm thinking at sending
> a patch clarifying testing QEMU with Avocado is not supported on Darwin.
>

What's broken on Darwin?

Is it broken with avocado-framework==88.1 too? (Is this a regression?)

We might need to consider entirely separate issues:

(1) We need to upgrade avocado-framework to a supported LTS version >= v90
for dependency harmony with qemu.git/python testing

(2) We need to upgrade avocado-framework to >= v101.0 to fix the stated
checksum issue in this patch

(3) We would like avocado tests to work on Darwin. (Have they ever worked?
When did they break? etc.)

My understanding is that this patch helps with 1 and 2 but not 3, but if
Darwin support isn't a regression, this is still a step forward.


> > I'm not seeing any with your patch applied.
>
> Are you using running the Cirrus-CI jobs?
>
>
Cleber Rosa April 21, 2023, 3:49 a.m. UTC | #6 
On 4/17/23 09:43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
>
> Bump up the avocado-framework version to 101.0.
>
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>
> ---
> v2
>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>   tests/Makefile.include | 18 +++++++++++-------
>   tests/requirements.txt |  2 +-
>   2 files changed, 12 insertions(+), 8 deletions(-)

Hi everyone,

Looks like I've mistakenly replied to the earlier thread, so here's 
pretty much the same message on a better location.

First of all, thanks to Kautuk for sending this and thanks to Hariharan 
for further testing it.

I'd like to give some context which not everyone may be aware of. 
Avocado 101.0 is a very different when compared with 88.1. Everything 
related to the execution of tests is brand new.  To be more precise, on 
version 91.0[1], this new runner[2] became the default. On version 97.0, 
the old runner implementation (currently in use in QEMU) was finally 
removed.

On most releases since then, I've been running the QEMU tests with the 
latest Avocado, and finding issues that are (as resources allow) 
addressed in later versions.   As you probably noticed, Avocado 101.0 
runs the QEMU tests without much (or any) visible issues for most 
people.  But, I'm aware of two pending issues that may or may not be a 
big deal to users:

I) The logging behavior is a bit different since Avocado 88.1. At a 
given point it was considered that Avocado should not mess around 
inadvertently with Python's root logger, and should be more picky about 
it includes in logs.  For most cases, a simple workaround[4] does the 
trick.  But, for some other use cases (say for 3rd party libraries' logs 
you want logged alongside Avocado's logs) there's a pending PR[5] that 
will take care of all known limitations.

II) The support for killing tests (internally in Avocado represented as 
more generic "tasks") and all its children is a bit lacking.  This is an 
issue I'm actively working on[6].  This may leave some processes (such 
as "qemu-system-*") running even after a test was interrupted.

Fixes for both of these issues are due to be in version 102.0. The ETA 
for version 102.0 is 1-2 weeks.

With that being said, I'm more than OK with this patch (alongside PATCH 
2, without which havoc ensues :) provided people understand the two 
pending issues above.  If this patch is taken before Avocado 102.0 is 
released, the delta from 101.0 would be much smaller, so it should be an 
easier change to test.

Cheers,

- Cleber.


[1] - https://avocado-framework.readthedocs.io/en/101.0/releases/91_0.html

[2] - The new runner is called "nrunner" and I am to be blamed for the 
naming lacking any originality

[3] - 
https://avocado-framework.readthedocs.io/en/101.0/releases/97_0.html#users-test-writers

[4] - 
https://gitlab.com/cleber.gnu/qemu/-/commit/a9f39c4f6671b756196a185c7275eb7ebd13e588

[5] - https://github.com/avocado-framework/avocado/pull/5645

[6] - https://github.com/avocado-framework/avocado/issues/4994
Cleber Rosa April 21, 2023, 3:56 a.m. UTC | #7 
On 4/17/23 12:50, John Snow wrote:
> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> wrote:
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>>   tests/Makefile.include | 18 +++++++++++-------
>>   tests/requirements.txt |  2 +-
>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>   # download all vm images, according to defined targets
>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>>
>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>> +
>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> -       $(call quiet-command, \
>> -            $(TESTS_PYTHON) -m avocado \
>> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> -                       --filter-by-tags-include-empty-key) \
>> -            $(AVOCADO_CMDLINE_TAGS) \
>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> +       $(call quiet-command,                                                   \
>> +            $(TESTS_PYTHON) -m avocado                                                 \
>> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR)  \
>> +            $(if $(AVOCADO_TAGS),,                                             \
>> +                       --filter-by-tags-include-empty                          \
>> +                       --filter-by-tags-include-empty-key)                     \
>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)                       \
>> +            $(AVOCADO_CMDLINE_TAGS)                                            \
>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),                   \
>>               "AVOCADO", "tests/avocado")
>>
>>   check-acceptance-deprecated-warning:
>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>> index 0ba561b6bd..a6f73da681 100644
>> --- a/tests/requirements.txt
>> +++ b/tests/requirements.txt
>> @@ -2,5 +2,5 @@
>>   # in the tests/venv Python virtual environment. For more info,
>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>   # Note that qemu.git/python/ is always implicitly installed.
>> -avocado-framework==88.1
>> +avocado-framework==101.0
>>   pycdlib==1.11.0
>> --
>> 2.39.2
>>
> I thought there were test failures that prohibited us from bumping the
> Avocado-Framework version. Did those get rectified recently?
>
Hi John,

While testing before and after behavior of pretty much every test in 
tests/avocado/,  I've found that some tests may get affected by the 
extra isolation (and overhead) of the new runner in recent Avocado and 
may get interrupted more often than on the previous runner.  But this is 
highly dependent on the amount of resources the machine you run the 
tests on has, and how tight the current timeout is.

Running all the tests under old and new Avocado also revealed some tests 
that are equally broken under both versions.  Thomas has bisected some, 
and Cedric has come up with some patches too.

To risk not being too verbose by default, let me know if you want the 
full details.  Also, see my previous reply about the two issues I'm 
aware that would make this bump as seamless as possible.

Thanks,

- Cleber.
Cleber Rosa April 21, 2023, 4:04 a.m. UTC | #8 
On 4/20/23 05:18, Philippe Mathieu-Daudé wrote:
> On 17/4/23 19:44, Alex Bennée wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>>> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée <alex.bennee@linaro.org> 
>>> wrote:
>>>>
>>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>>
>>>> Avocado version 101.0 has a fix to re-compute the checksum
>>>> of an asset file if the algorithm used in the *-CHECKSUM
>>>> file isn't the same as the one being passed to it by the
>>>> avocado user (i.e. the avocado_qemu python module).
>>>> In the earlier avocado versions this fix wasn't there due
>>>> to which if the checksum wouldn't match the earlier
>>>> checksum (calculated by a different algorithm), the avocado
>>>> code would start downloading a fresh image from the internet
>>>> URL thus making the test-cases take longer to execute.
>>>>
>>>> Bump up the avocado-framework version to 101.0.
>>>>
>>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>>
>>>> ---
>>>> v2
>>>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>>> ---
>>>>   tests/Makefile.include | 18 +++++++++++-------
>>>>   tests/requirements.txt |  2 +-
>>>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index 9422ddaece..a4de0ad5a2 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>>   # download all vm images, according to defined targets
>>>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
>>>> $(FEDORA_31_DOWNLOAD))
>>>>
>>>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter 
>>>> -j%,$(MAKEFLAGS))))
>>>> +
>>>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>>> -       $(call quiet-command, \
>>>> -            $(TESTS_PYTHON) -m avocado \
>>>> -            --show=$(AVOCADO_SHOW) run 
>>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>>> -                       --filter-by-tags-include-empty-key) \
>>>> -            $(AVOCADO_CMDLINE_TAGS) \
>>>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>>> +       $(call quiet-command, \
>>>> +            $(TESTS_PYTHON) -m 
>>>> avocado                                                 \
>>>> +            --show=$(AVOCADO_SHOW) run 
>>>> --job-results-dir=$(TESTS_RESULTS_DIR)  \
>>>> +            $(if $(AVOCADO_TAGS),, \
>>>> + --filter-by-tags-include-empty                          \
>>>> + --filter-by-tags-include-empty-key)                     \
>>>> +               --max-parallel-tasks 
>>>> $(JOBS_OPTION:-j%=%)                       \
>>>> + $(AVOCADO_CMDLINE_TAGS) \
>>>> +            $(if $(GITLAB_CI),,--failfast) 
>>>> $(AVOCADO_TESTS),                   \
>>>>               "AVOCADO", "tests/avocado")
>>>>
>>>>   check-acceptance-deprecated-warning:
>>>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>>> index 0ba561b6bd..a6f73da681 100644
>>>> --- a/tests/requirements.txt
>>>> +++ b/tests/requirements.txt
>>>> @@ -2,5 +2,5 @@
>>>>   # in the tests/venv Python virtual environment. For more info,
>>>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>>>   # Note that qemu.git/python/ is always implicitly installed.
>>>> -avocado-framework==88.1
>>>> +avocado-framework==101.0
>>>>   pycdlib==1.11.0
>>>> -- 
>>>> 2.39.2
>>>>
>>>
>>> I thought there were test failures that prohibited us from bumping the
>>> Avocado-Framework version. Did those get rectified recently?
>
> No, still not working on Darwin. At this point I'm thinking at sending
> a patch clarifying testing QEMU with Avocado is not supported on Darwin.
>
Hi Phil,

Do you mean with or without this version bump?

Even though it was somewhat recently[1] that another level of Darwin 
compatibility was added to Avocado, where a good part of Avocado's 
selftests[2] were enabled in OS X, Avocado 101.0 should not be that far 
behind.

I honestly believe people testing and reporting successful execution of 
the Avocado tests under Darwin with latest Avocado. Hopefully my memory 
is not playing tricks on me, but I'll attempt to verify that.

Cheers,

- Cleber.

[1] - https://github.com/avocado-framework/avocado/pull/5622

[2] - 
https://github.com/avocado-framework/avocado/actions/runs/4417285317/jobs/7742772932?pr=5622#step:10:11


>> I'm not seeing any with your patch applied.
>
> Are you using running the Cirrus-CI jobs?
>
Cleber Rosa April 21, 2023, 4:09 a.m. UTC | #9 
On 4/20/23 12:54, John Snow wrote:
>
>
> On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé 
> <philmd@linaro.org> wrote:
>
>     On 17/4/23 19:44, Alex Bennée wrote:
>     >
>     > John Snow <jsnow@redhat.com> writes:
>     >
>     >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée
>     <alex.bennee@linaro.org> wrote:
>     >>>
>     >>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>     >>>
>     >>> Avocado version 101.0 has a fix to re-compute the checksum
>     >>> of an asset file if the algorithm used in the *-CHECKSUM
>     >>> file isn't the same as the one being passed to it by the
>     >>> avocado user (i.e. the avocado_qemu python module).
>     >>> In the earlier avocado versions this fix wasn't there due
>     >>> to which if the checksum wouldn't match the earlier
>     >>> checksum (calculated by a different algorithm), the avocado
>     >>> code would start downloading a fresh image from the internet
>     >>> URL thus making the test-cases take longer to execute.
>     >>>
>     >>> Bump up the avocado-framework version to 101.0.
>     >>>
>     >>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>     >>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>     >>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>     >>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>     >>>
>     >>> ---
>     >>> v2
>     >>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>     >>> ---
>     >>>   tests/Makefile.include | 18 +++++++++++-------
>     >>>   tests/requirements.txt |  2 +-
>     >>>   2 files changed, 12 insertions(+), 8 deletions(-)
>     >>>
>     >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>     >>> index 9422ddaece..a4de0ad5a2 100644
>     >>> --- a/tests/Makefile.include
>     >>> +++ b/tests/Makefile.include
>     >>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>     >>>   # download all vm images, according to defined targets
>     >>>   get-vm-images: check-venv $(patsubst
>     %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>     >>>
>     >>> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter
>     -j%,$(MAKEFLAGS))))
>     >>> +
>     >>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>     >>> -       $(call quiet-command, \
>     >>> -            $(TESTS_PYTHON) -m avocado \
>     >>> -            --show=$(AVOCADO_SHOW) run
>     --job-results-dir=$(TESTS_RESULTS_DIR) \
>     >>> -            $(if $(AVOCADO_TAGS),,
>     --filter-by-tags-include-empty \
>     >>> -  --filter-by-tags-include-empty-key) \
>     >>> -            $(AVOCADO_CMDLINE_TAGS) \
>     >>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>     >>> +       $(call quiet-command,                                    \
>     >>> +            $(TESTS_PYTHON) -m avocado                      
>                          \
>     >>> +            --show=$(AVOCADO_SHOW) run
>     --job-results-dir=$(TESTS_RESULTS_DIR)  \
>     >>> +            $(if $(AVOCADO_TAGS),,                          
>              \
>     >>> +  --filter-by-tags-include-empty                          \
>     >>> +  --filter-by-tags-include-empty-key)                     \
>     >>> +               --max-parallel-tasks $(JOBS_OPTION:-j%=%)     
>                      \
>     >>> +            $(AVOCADO_CMDLINE_TAGS)                          
>             \
>     >>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 
>                      \
>     >>>               "AVOCADO", "tests/avocado")
>     >>>
>     >>>   check-acceptance-deprecated-warning:
>     >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>     >>> index 0ba561b6bd..a6f73da681 100644
>     >>> --- a/tests/requirements.txt
>     >>> +++ b/tests/requirements.txt
>     >>> @@ -2,5 +2,5 @@
>     >>>   # in the tests/venv Python virtual environment. For more info,
>     >>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>     >>>   # Note that qemu.git/python/ is always implicitly installed.
>     >>> -avocado-framework==88.1
>     >>> +avocado-framework==101.0
>     >>>   pycdlib==1.11.0
>     >>> --
>     >>> 2.39.2
>     >>>
>     >>
>     >> I thought there were test failures that prohibited us from
>     bumping the
>     >> Avocado-Framework version. Did those get rectified recently?
>
>     No, still not working on Darwin. At this point I'm thinking at sending
>     a patch clarifying testing QEMU with Avocado is not supported on
>     Darwin.
>
>
> What's broken on Darwin?
>
Hi John,

IIRC, the main issue is that the legacy runner (the one active in 88.1) 
pickles some stuff that do not play nice under Darwin.  It has never (to 
the best of my knowledge) worked under Darwin.

> Is it broken with avocado-framework==88.1 too? (Is this a regression?)
>
It was already broken with avocado-framework==88.1, but it should mostly 
work with 101.0 (see previous reply).

> We might need to consider entirely separate issues:
>
> (1) We need to upgrade avocado-framework to a supported LTS version >= 
> v90 for dependency harmony with qemu.git/python testing
>
> (2) We need to upgrade avocado-framework to >= v101.0 to fix the 
> stated checksum issue in this patch
>
> (3) We would like avocado tests to work on Darwin. (Have they ever 
> worked? When did they break? etc.)
>
> My understanding is that this patch helps with 1 and 2 but not 3, but 
> if Darwin support isn't a regression, this is still a step forward.
>
To the best of my knowledge, this patch would help with 1, 2 and even 3.

Cheers,

- Cleber.

>
>     > I'm not seeing any with your patch applied.
>
>     Are you using running the Cirrus-CI jobs?
>
Cleber Rosa April 21, 2023, 4:19 a.m. UTC | #10 
On 4/18/23 03:10, Thomas Huth wrote:
> On 17/04/2023 15.43, Alex Bennée wrote:
>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>
>> Avocado version 101.0 has a fix to re-compute the checksum
>> of an asset file if the algorithm used in the *-CHECKSUM
>> file isn't the same as the one being passed to it by the
>> avocado user (i.e. the avocado_qemu python module).
>> In the earlier avocado versions this fix wasn't there due
>> to which if the checksum wouldn't match the earlier
>> checksum (calculated by a different algorithm), the avocado
>> code would start downloading a fresh image from the internet
>> URL thus making the test-cases take longer to execute.
>>
>> Bump up the avocado-framework version to 101.0.
>>
>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>
>> ---
>> v2
>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>> ---
>>   tests/Makefile.include | 18 +++++++++++-------
>>   tests/requirements.txt |  2 +-
>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 9422ddaece..a4de0ad5a2 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>   # download all vm images, according to defined targets
>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
>> $(FEDORA_31_DOWNLOAD))
>>   +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter 
>> -j%,$(MAKEFLAGS))))
>> +
>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>> -    $(call quiet-command, \
>> -            $(TESTS_PYTHON) -m avocado \
>> -            --show=$(AVOCADO_SHOW) run 
>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>> -            --filter-by-tags-include-empty-key) \
>> -            $(AVOCADO_CMDLINE_TAGS) \
>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>> +    $(call quiet-command,                             \
>> +            $(TESTS_PYTHON) -m avocado \
>> +            --show=$(AVOCADO_SHOW) run 
>> --job-results-dir=$(TESTS_RESULTS_DIR)     \
>> +            $(if $(AVOCADO_TAGS),,                         \
>> +            --filter-by-tags-include-empty                 \
>> +            --filter-by-tags-include-empty-key)             \
>> +        --max-parallel-tasks $(JOBS_OPTION:-j%=%)             \
>> +            $(AVOCADO_CMDLINE_TAGS)                         \
>> +            $(if $(GITLAB_CI),,--failfast) 
>> $(AVOCADO_TESTS),             \
>
> We might need to revisit this --failfast logic, too. If I've got that 
> right, failfast is now the default with the new system? So we might 
> want to disable it by default again if GITLAB_CI is not set?
>
>  Thomas
>
Hi Thomas,

I must be missing something, because under Avocado 101.0, I'm getting 
the following behavior without the --failfast flag:

   $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true
   JOB ID     : 646f476f01b8d5599a57530606de543f2d9a5366
   JOB LOG    : 
/root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log
    (1/3) /bin/true: STARTED
    (1/3) /bin/true: PASS (0.01 s)
    (2/3) /bin/false: STARTED
    (2/3) /bin/false: FAIL (0.01 s)
    (3/3) /bin/true: STARTED
    (3/3) /bin/true: PASS (0.01 s)
   RESULTS    : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 
0 | CANCEL 0
   JOB TIME   : 1.01 s

   Test summary:
   /bin/false: FAIL

And this with --failfast:

   $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true 
/bin/false /bin/true
   JOB ID     : ae4894607a42194a7382efa545eccaccf7495fa3
   JOB LOG    : 
/root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log
    (1/3) /bin/true: STARTED
    (1/3) /bin/true: PASS (0.01 s)
    (2/3) /bin/false: STARTED
    (2/3) /bin/false: FAIL (0.01 s)
   Interrupting job (failfast).
   RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 
0 | CANCEL 0
   JOB TIME   : 0.75 s

   Test summary:
   /bin/false: FAIL

Maybe it's something in the Makefile I'm missing, or something specific 
to these tests, but that would be a long shot.

Thanks,

- Cleber.
Thomas Huth April 21, 2023, 6:21 a.m. UTC | #11 
On 21/04/2023 06.19, Cleber Rosa wrote:
> 
> On 4/18/23 03:10, Thomas Huth wrote:
>> On 17/04/2023 15.43, Alex Bennée wrote:
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the
>>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due
>>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado
>>> code would start downloading a fresh image from the internet
>>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>> ---
>>>   tests/Makefile.include | 18 +++++++++++-------
>>>   tests/requirements.txt |  2 +-
>>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>   # download all vm images, according to defined targets
>>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
>>> $(FEDORA_31_DOWNLOAD))
>>>   +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>>> +
>>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> -    $(call quiet-command, \
>>> -            $(TESTS_PYTHON) -m avocado \
>>> -            --show=$(AVOCADO_SHOW) run 
>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>> -            --filter-by-tags-include-empty-key) \
>>> -            $(AVOCADO_CMDLINE_TAGS) \
>>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> +    $(call quiet-command,                             \
>>> +            $(TESTS_PYTHON) -m avocado \
>>> +            --show=$(AVOCADO_SHOW) run 
>>> --job-results-dir=$(TESTS_RESULTS_DIR)     \
>>> +            $(if $(AVOCADO_TAGS),,                         \
>>> +            --filter-by-tags-include-empty                 \
>>> +            --filter-by-tags-include-empty-key)             \
>>> +        --max-parallel-tasks $(JOBS_OPTION:-j%=%)             \
>>> +            $(AVOCADO_CMDLINE_TAGS)                         \
>>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),             \
>>
>> We might need to revisit this --failfast logic, too. If I've got that 
>> right, failfast is now the default with the new system? So we might want 
>> to disable it by default again if GITLAB_CI is not set?
>>
>>  Thomas
>>
> Hi Thomas,
> 
> I must be missing something, because under Avocado 101.0, I'm getting the 
> following behavior without the --failfast flag:
> 
>    $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true
>    JOB ID     : 646f476f01b8d5599a57530606de543f2d9a5366
>    JOB LOG    : /root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log
>     (1/3) /bin/true: STARTED
>     (1/3) /bin/true: PASS (0.01 s)
>     (2/3) /bin/false: STARTED
>     (2/3) /bin/false: FAIL (0.01 s)
>     (3/3) /bin/true: STARTED
>     (3/3) /bin/true: PASS (0.01 s)
>    RESULTS    : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>    JOB TIME   : 1.01 s
> 
>    Test summary:
>    /bin/false: FAIL
> 
> And this with --failfast:
> 
>    $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true /bin/false 
> /bin/true
>    JOB ID     : ae4894607a42194a7382efa545eccaccf7495fa3
>    JOB LOG    : /root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log
>     (1/3) /bin/true: STARTED
>     (1/3) /bin/true: PASS (0.01 s)
>     (2/3) /bin/false: STARTED
>     (2/3) /bin/false: FAIL (0.01 s)
>    Interrupting job (failfast).
>    RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>    JOB TIME   : 0.75 s
> 
>    Test summary:
>    /bin/false: FAIL
> 
> Maybe it's something in the Makefile I'm missing, or something specific to 
> these tests, but that would be a long shot.

Ah, thanks, I think I simply got it mixed up while trying to read the above 
Makefile logic - never mind my question, it should be fine.

  Thomas
Thomas Huth April 21, 2023, 6:24 a.m. UTC | #12 
On 17/04/2023 15.43, Alex Bennée wrote:
> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> 
> Avocado version 101.0 has a fix to re-compute the checksum
> of an asset file if the algorithm used in the *-CHECKSUM
> file isn't the same as the one being passed to it by the
> avocado user (i.e. the avocado_qemu python module).
> In the earlier avocado versions this fix wasn't there due
> to which if the checksum wouldn't match the earlier
> checksum (calculated by a different algorithm), the avocado
> code would start downloading a fresh image from the internet
> URL thus making the test-cases take longer to execute.
> 
> Bump up the avocado-framework version to 101.0.
> 
> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
> 
> ---
> v2
>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
> ---
>   tests/Makefile.include | 18 +++++++++++-------
>   tests/requirements.txt |  2 +-
>   2 files changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 9422ddaece..a4de0ad5a2 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>   # download all vm images, according to defined targets
>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
>   
> +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
> +
>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
> -	$(call quiet-command, \
> -            $(TESTS_PYTHON) -m avocado \
> -            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
> -			--filter-by-tags-include-empty-key) \
> -            $(AVOCADO_CMDLINE_TAGS) \
> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
> +	$(call quiet-command, 							\
> +            $(TESTS_PYTHON) -m avocado 						\
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
> +            $(if $(AVOCADO_TAGS),, 						\
> +			--filter-by-tags-include-empty 				\
> +			--filter-by-tags-include-empty-key) 			\
> +		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
> +            $(AVOCADO_CMDLINE_TAGS) 						\
> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\
>               "AVOCADO", "tests/avocado")
>   
>   check-acceptance-deprecated-warning:
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 0ba561b6bd..a6f73da681 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -2,5 +2,5 @@
>   # in the tests/venv Python virtual environment. For more info,
>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>   # Note that qemu.git/python/ is always implicitly installed.
> -avocado-framework==88.1
> +avocado-framework==101.0
>   pycdlib==1.11.0

Reviewed-by: Thomas Huth <thuth@redhat.com>
Philippe Mathieu-Daudé April 21, 2023, 10:07 a.m. UTC | #13 
On 21/4/23 06:09, Cleber Rosa wrote:
> 
> On 4/20/23 12:54, John Snow wrote:
>>
>>
>> On Thu, Apr 20, 2023, 5:19 AM Philippe Mathieu-Daudé 
>> <philmd@linaro.org> wrote:
>>
>>     On 17/4/23 19:44, Alex Bennée wrote:
>>     >
>>     > John Snow <jsnow@redhat.com> writes:
>>     >
>>     >> On Mon, Apr 17, 2023 at 9:43 AM Alex Bennée
>>     <alex.bennee@linaro.org> wrote:

>>     >>> diff --git a/tests/requirements.txt b/tests/requirements.txt
>>     >>> index 0ba561b6bd..a6f73da681 100644
>>     >>> --- a/tests/requirements.txt
>>     >>> +++ b/tests/requirements.txt
>>     >>> @@ -2,5 +2,5 @@
>>     >>>   # in the tests/venv Python virtual environment. For more info,
>>     >>>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
>>     >>>   # Note that qemu.git/python/ is always implicitly installed.
>>     >>> -avocado-framework==88.1
>>     >>> +avocado-framework==101.0
>>     >>>   pycdlib==1.11.0
>>     >>> --
>>     >>> 2.39.2
>>     >>>
>>     >>
>>     >> I thought there were test failures that prohibited us from
>>     bumping the
>>     >> Avocado-Framework version. Did those get rectified recently?
>>
>>     No, still not working on Darwin. At this point I'm thinking at 
>> sending
>>     a patch clarifying testing QEMU with Avocado is not supported on
>>     Darwin.
>>
>>
>> What's broken on Darwin?
>>
> Hi John,
> 
> IIRC, the main issue is that the legacy runner (the one active in 88.1) 
> pickles some stuff that do not play nice under Darwin.  It has never (to 
> the best of my knowledge) worked under Darwin.

Correct.

>> Is it broken with avocado-framework==88.1 too? (Is this a regression?)
>>
> It was already broken with avocado-framework==88.1,

Correct.

> but it should mostly 
> work with 101.0 (see previous reply).
> 
>> We might need to consider entirely separate issues:
>>
>> (1) We need to upgrade avocado-framework to a supported LTS version >= 
>> v90 for dependency harmony with qemu.git/python testing
>>
>> (2) We need to upgrade avocado-framework to >= v101.0 to fix the 
>> stated checksum issue in this patch
>>
>> (3) We would like avocado tests to work on Darwin. (Have they ever 
>> worked? When did they break? etc.)
>>
>> My understanding is that this patch helps with 1 and 2 but not 3, but 
>> if Darwin support isn't a regression, this is still a step forward.
>>
> To the best of my knowledge, this patch would help with 1, 2 and even 3.

As of today using v8.0.0:

$ make check-avocado
   GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 
tests/fp/berkeley-softfloat-3 dtc
   MKDIR   /Users/philmd/source/qemu/build/tests/results
   AVOCADO Downloading avocado tests VM image for aarch64
The image was downloaded:
Provider Version Architecture File
fedora   31      aarch64 
/Users/philmd/avocado/data/cache/by_location/4f156e531446a679cbfe13caef8b7c9f9f79aafa/Fedora-Cloud-Base-31-1.9.aarch64.qcow2
   AVOCADO tests/avocado
Fetching asset from 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf
Fetching asset from 
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_411_and_dom0
Fetching asset from 
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_414_and_dom0
Fetching asset from 
tests/avocado/boot_xen.py:BootXen.test_arm64_xen_415_and_dom0
Fetching asset from 
tests/avocado/machine_aarch64_virt.py:Aarch64VirtMachine.test_alpine_virt_tcg_gic_max
Fetching asset from 
tests/avocado/replay_kernel.py:ReplayKernelNormal.test_aarch64_virt
Fetching asset from 
tests/avocado/reverse_debugging.py:ReverseDebugging_AArch64.test_aarch64_virt
JOB ID     : 142a0b2d3960c7daccd657781d52f814e1617392
JOB LOG    : 
/Users/philmd/source/qemu/build/tests/results/job-2023-04-21T11.35-142a0b2/job.log

Avocado crashed: TypeError: cannot pickle '_thread.RLock' object
Traceback (most recent call last):

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/job.py", 
line 632, in run_tests
     summary |= suite.run(self)

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/core/suite.py", 
line 311, in run
     return self.runner.run_suite(job, self)

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py", 
line 390, in run_suite
     if not self.run_test(job, test_factory, queue, summary,

   File 
"/Users/philmd/source/qemu/build/tests/venv/lib/python3.10/site-packages/avocado/plugins/runner.py", 
line 170, in run_test
     proc.start()

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", 
line 121, in start
     self._popen = self._Popen(self)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", 
line 224, in _Popen
     return _default_context.get_context().Process._Popen(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", 
line 288, in _Popen
     return Popen(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", 
line 32, in __init__
     super().__init__(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", 
line 19, in __init__
     self._launch(process_obj)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", 
line 47, in _launch
     reduction.dump(process_obj, fp)

   File 
"/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/reduction.py", 
line 60, in dump
     ForkingPickler(file, protocol).dump(obj)

TypeError: cannot pickle '_thread.RLock' object

Please include the traceback info and command line used on your bug report
Report bugs visiting https://github.com/avocado-framework/avocado/issues/new
make: *** [check-avocado] Error 4

This is https://github.com/avocado-framework/avocado/issues/4888
which is still opened.

Manually using "avocado run --test-runner=nrunner" helps running the
tests but the console output is never displayed (see [*]) and once
finished Avocado lets various QEMU processes running in detached
mode, wasting resources.

[*] 
https://lore.kernel.org/qemu-devel/1d0baf8b-c757-265c-b206-07ca3f218b2a@linaro.org/
Philippe Mathieu-Daudé April 21, 2023, 10:27 a.m. UTC | #14 
On 21/4/23 12:07, Philippe Mathieu-Daudé wrote:
> On 21/4/23 06:09, Cleber Rosa wrote:

> Manually using "avocado run --test-runner=nrunner" helps running the
> tests but the console output is never displayed (see [*]) and once
> finished Avocado lets various QEMU processes running in detached
> mode, wasting resources.=

Using this patch:

$ ./tests/venv/bin/avocado --show=console -V run -t arch:aarch64 
tests/avocado/boot_*
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
ERROR
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_xlnx_versal_virt: 
ERROR

Not asking for the console output at least show the test being
processed (similar to old runner):

$ ./tests/venv/bin/avocado run -t machine:raspi3b tests/avocado/boot_*
Fetching asset from 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf
JOB ID     : 1c963c12f7f07e109ef8480ddf204f02e61f8b8d
JOB LOG    : 
/Users/philmd/avocado/job-results/job-2023-04-21T12.24-1c963c1/job.log
  (1/1) 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
STARTED
  (1/1) 
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
ERROR: ConnectError: Failed to establish session: EOFError\n	Exit code: 
1\n	Command: ./qemu-system-aarch64 -display none -vga none -chardev 
socket,id=mon,fd=11 -mon chardev=mon,mode=control -machine raspi3b 
-serial null -chardev socket,id=console,path=/var/folders... (0.31 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
CANCEL 0
JOB TIME   : 2.20 s

Test summary:
tests/avocado/boot_linux_console.py:BootLinuxConsole.test_aarch64_raspi3_atf: 
ERROR
diff mbox series

Patch

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 9422ddaece..a4de0ad5a2 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -138,14 +138,18 @@  get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
+JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
+
 check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
-	$(call quiet-command, \
-            $(TESTS_PYTHON) -m avocado \
-            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
-            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
-			--filter-by-tags-include-empty-key) \
-            $(AVOCADO_CMDLINE_TAGS) \
-            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
+	$(call quiet-command, 							\
+            $(TESTS_PYTHON) -m avocado 						\
+            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
+            $(if $(AVOCADO_TAGS),, 						\
+			--filter-by-tags-include-empty 				\
+			--filter-by-tags-include-empty-key) 			\
+		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
+            $(AVOCADO_CMDLINE_TAGS) 						\
+            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\
             "AVOCADO", "tests/avocado")
 
 check-acceptance-deprecated-warning:
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 0ba561b6bd..a6f73da681 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -2,5 +2,5 @@ 
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
 # Note that qemu.git/python/ is always implicitly installed.
-avocado-framework==88.1
+avocado-framework==101.0
 pycdlib==1.11.0