[v2,7/8] iotests: register each I/O test separately with meson

Currently meson registers a single test that invokes an entire group of
I/O tests, hiding the test granularity from meson. There are various
downsides of doing this

 * You cannot ask 'meson test' to invoke a single I/O test
 * The meson test timeout can't be applied to the individual
   tests
 * Meson only gets a pass/fail for the overall I/O test group
   not individual tests
 * If a CI job gets killed by the GitLab timeout, we don't
   get visibility into how far through the I/O tests
   execution got.

This switches meson to perform test discovery by invoking 'check' in
dry-run mode. It then registers one meson test case for each I/O
test. Parallel execution remains disabled since the I/O tests do not
use self contained execution environments and thus conflict with
each other.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------
 1 file changed, 29 insertions(+), 6 deletions(-)

On 03/03/2023 17.07, Daniel P. Berrangé wrote:
> Currently meson registers a single test that invokes an entire group of
> I/O tests, hiding the test granularity from meson. There are various
> downsides of doing this
> 
>   * You cannot ask 'meson test' to invoke a single I/O test
>   * The meson test timeout can't be applied to the individual
>     tests
>   * Meson only gets a pass/fail for the overall I/O test group
>     not individual tests
>   * If a CI job gets killed by the GitLab timeout, we don't
>     get visibility into how far through the I/O tests
>     execution got.
> 
> This switches meson to perform test discovery by invoking 'check' in
> dry-run mode. It then registers one meson test case for each I/O
> test. Parallel execution remains disabled since the I/O tests do not
> use self contained execution environments and thus conflict with
> each other.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------
>   1 file changed, 29 insertions(+), 6 deletions(-)

Reviewed-by: Thomas Huth <thuth@redhat.com>

On 03/03/2023 17.07, Daniel P. Berrangé wrote:
> Currently meson registers a single test that invokes an entire group of
> I/O tests, hiding the test granularity from meson. There are various
> downsides of doing this
> 
>   * You cannot ask 'meson test' to invoke a single I/O test
>   * The meson test timeout can't be applied to the individual
>     tests
>   * Meson only gets a pass/fail for the overall I/O test group
>     not individual tests
>   * If a CI job gets killed by the GitLab timeout, we don't
>     get visibility into how far through the I/O tests
>     execution got.
> 
> This switches meson to perform test discovery by invoking 'check' in
> dry-run mode. It then registers one meson test case for each I/O
> test. Parallel execution remains disabled since the I/O tests do not
> use self contained execution environments and thus conflict with
> each other.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>   tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------
>   1 file changed, 29 insertions(+), 6 deletions(-)
> 
> diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
> index 323a4acb6a..a162f683ef 100644
> --- a/tests/qemu-iotests/meson.build
> +++ b/tests/qemu-iotests/meson.build
> @@ -32,16 +32,39 @@ foreach k, v : emulators
>     endif
>   endforeach
>   
> +qemu_iotests_check_cmd = files('check')
> +
>   foreach format, speed: qemu_iotests_formats
>     if speed == 'quick'
>       suites = 'block'
>     else
>       suites = ['block-' + speed, speed]
>     endif
> -  test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format],
> -       depends: qemu_iotests_binaries, env: qemu_iotests_env,
> -       protocol: 'tap',
> -       suite: suites,
> -       timeout: 0,
> -       is_parallel: false)
> +
> +  args = ['-tap', '-' + format]
> +  if speed == 'quick'
> +      args += ['-g', 'auto']
> +  endif
> +
> +  rc = run_command(
> +      [qemu_iotests_check_cmd] + args + ['-n'],
> +      check: true,
> +  )
> +
> +  foreach item: rc.stdout().strip().split()
> +      args = ['-tap', '-' + format, item,
> +              '--source-dir', meson.current_source_dir(),
> +              '--build-dir', meson.current_build_dir()]
> +      # Some individual tests take as long as 45 seconds
> +      # Bump the timeout to 3 minutes for some headroom
> +      # on slow machines to minimize spurious failures
> +      test('io-' + format + '-' + item,
> +           qemu_iotests_check_cmd,
> +           args: args,
> +           depends: qemu_iotests_binaries,
> +           env: qemu_iotests_env,
> +           protocol: 'tap',
> +           timeout: 180,
> +           suite: suites)
> +  endforeach
>   endforeach

Seems like this somehow broke compilation on NetBSD:

https://gitlab.com/thuth/qemu/-/jobs/4021584713#L2980

?

  Thomas

On Wed, Mar 29, 2023 at 12:47:51PM +0200, Thomas Huth wrote:
> On 03/03/2023 17.07, Daniel P. Berrangé wrote:
> > Currently meson registers a single test that invokes an entire group of
> > I/O tests, hiding the test granularity from meson. There are various
> > downsides of doing this
> > 
> >   * You cannot ask 'meson test' to invoke a single I/O test
> >   * The meson test timeout can't be applied to the individual
> >     tests
> >   * Meson only gets a pass/fail for the overall I/O test group
> >     not individual tests
> >   * If a CI job gets killed by the GitLab timeout, we don't
> >     get visibility into how far through the I/O tests
> >     execution got.
> > 
> > This switches meson to perform test discovery by invoking 'check' in
> > dry-run mode. It then registers one meson test case for each I/O
> > test. Parallel execution remains disabled since the I/O tests do not
> > use self contained execution environments and thus conflict with
> > each other.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >   tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------
> >   1 file changed, 29 insertions(+), 6 deletions(-)
> > 
> > diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
> > index 323a4acb6a..a162f683ef 100644
> > --- a/tests/qemu-iotests/meson.build
> > +++ b/tests/qemu-iotests/meson.build
> > @@ -32,16 +32,39 @@ foreach k, v : emulators
> >     endif
> >   endforeach
> > +qemu_iotests_check_cmd = files('check')
> > +
> >   foreach format, speed: qemu_iotests_formats
> >     if speed == 'quick'
> >       suites = 'block'
> >     else
> >       suites = ['block-' + speed, speed]
> >     endif
> > -  test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format],
> > -       depends: qemu_iotests_binaries, env: qemu_iotests_env,
> > -       protocol: 'tap',
> > -       suite: suites,
> > -       timeout: 0,
> > -       is_parallel: false)
> > +
> > +  args = ['-tap', '-' + format]
> > +  if speed == 'quick'
> > +      args += ['-g', 'auto']
> > +  endif
> > +
> > +  rc = run_command(
> > +      [qemu_iotests_check_cmd] + args + ['-n'],
> > +      check: true,
> > +  )
> > +
> > +  foreach item: rc.stdout().strip().split()
> > +      args = ['-tap', '-' + format, item,
> > +              '--source-dir', meson.current_source_dir(),
> > +              '--build-dir', meson.current_build_dir()]
> > +      # Some individual tests take as long as 45 seconds
> > +      # Bump the timeout to 3 minutes for some headroom
> > +      # on slow machines to minimize spurious failures
> > +      test('io-' + format + '-' + item,
> > +           qemu_iotests_check_cmd,
> > +           args: args,
> > +           depends: qemu_iotests_binaries,
> > +           env: qemu_iotests_env,
> > +           protocol: 'tap',
> > +           timeout: 180,
> > +           suite: suites)
> > +  endforeach
> >   endforeach
> 
> Seems like this somehow broke compilation on NetBSD:
> 
> https://gitlab.com/thuth/qemu/-/jobs/4021584713#L2980

I ran it locally and got the meson-log.txt file which reports

  env: python3: No such file or directory

and indeed there is no python3 binary present in our netbsd
VM.

our tests/vm/netbsd script works around this by passing an
explicit --python=python3.7 arg to configure, but the way
we invoke the 'check' script means it is just using the
"#!/usr/bin/env python3"  logic instead.

With regards,
Daniel

On 29/03/2023 13.18, Daniel P. Berrangé wrote:
> On Wed, Mar 29, 2023 at 12:47:51PM +0200, Thomas Huth wrote:
>> On 03/03/2023 17.07, Daniel P. Berrangé wrote:
>>> Currently meson registers a single test that invokes an entire group of
>>> I/O tests, hiding the test granularity from meson. There are various
>>> downsides of doing this
>>>
>>>    * You cannot ask 'meson test' to invoke a single I/O test
>>>    * The meson test timeout can't be applied to the individual
>>>      tests
>>>    * Meson only gets a pass/fail for the overall I/O test group
>>>      not individual tests
>>>    * If a CI job gets killed by the GitLab timeout, we don't
>>>      get visibility into how far through the I/O tests
>>>      execution got.
>>>
>>> This switches meson to perform test discovery by invoking 'check' in
>>> dry-run mode. It then registers one meson test case for each I/O
>>> test. Parallel execution remains disabled since the I/O tests do not
>>> use self contained execution environments and thus conflict with
>>> each other.
>>>
>>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>>> ---
>>>    tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------
>>>    1 file changed, 29 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build
>>> index 323a4acb6a..a162f683ef 100644
>>> --- a/tests/qemu-iotests/meson.build
>>> +++ b/tests/qemu-iotests/meson.build
>>> @@ -32,16 +32,39 @@ foreach k, v : emulators
>>>      endif
>>>    endforeach
>>> +qemu_iotests_check_cmd = files('check')
>>> +
>>>    foreach format, speed: qemu_iotests_formats
>>>      if speed == 'quick'
>>>        suites = 'block'
>>>      else
>>>        suites = ['block-' + speed, speed]
>>>      endif
>>> -  test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format],
>>> -       depends: qemu_iotests_binaries, env: qemu_iotests_env,
>>> -       protocol: 'tap',
>>> -       suite: suites,
>>> -       timeout: 0,
>>> -       is_parallel: false)
>>> +
>>> +  args = ['-tap', '-' + format]
>>> +  if speed == 'quick'
>>> +      args += ['-g', 'auto']
>>> +  endif
>>> +
>>> +  rc = run_command(
>>> +      [qemu_iotests_check_cmd] + args + ['-n'],
>>> +      check: true,
>>> +  )
>>> +
>>> +  foreach item: rc.stdout().strip().split()
>>> +      args = ['-tap', '-' + format, item,
>>> +              '--source-dir', meson.current_source_dir(),
>>> +              '--build-dir', meson.current_build_dir()]
>>> +      # Some individual tests take as long as 45 seconds
>>> +      # Bump the timeout to 3 minutes for some headroom
>>> +      # on slow machines to minimize spurious failures
>>> +      test('io-' + format + '-' + item,
>>> +           qemu_iotests_check_cmd,
>>> +           args: args,
>>> +           depends: qemu_iotests_binaries,
>>> +           env: qemu_iotests_env,
>>> +           protocol: 'tap',
>>> +           timeout: 180,
>>> +           suite: suites)
>>> +  endforeach
>>>    endforeach
>>
>> Seems like this somehow broke compilation on NetBSD:
>>
>> https://gitlab.com/thuth/qemu/-/jobs/4021584713#L2980
> 
> I ran it locally and got the meson-log.txt file which reports
> 
>    env: python3: No such file or directory
> 
> and indeed there is no python3 binary present in our netbsd
> VM.
> 
> our tests/vm/netbsd script works around this by passing an
> explicit --python=python3.7 arg to configure, but the way
> we invoke the 'check' script means it is just using the
> "#!/usr/bin/env python3"  logic instead.

Ah, that rings a bell - python scripts must not have the executable flags, 
otherwise meson tries to run them directly instead of using the interpreter 
that has been specified with the --python option.

  Thomas