diff mbox series

[v2,6/7] Acceptance Tests: add variants definition for architectures

Message ID 20181009232607.15521-7-crosa@redhat.com
State New
Headers show
Series Acceptance Tests: basic architecture support | expand

Commit Message

Cleber Rosa Oct. 9, 2018, 11:26 p.m. UTC
One of the Avocado features relevant to virtualization testing is the
ability to reuse tests in different scenarios, known as variants.
This adds a JSON based variants file, that can be used to run most
tests in a number of different architectures.  It can be run with:

   $ avocado run \
     --json-variants-load=tests/acceptance/variants/arch.json \
     --filter-by-tags='-x86_64' -- tests/acceptance/

Currently this covers 5 architectures, resulting in the execution
of 25 different combinations.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100644 tests/acceptance/variants/arch.json

Comments

Philippe Mathieu-Daudé Oct. 10, 2018, 10:51 a.m. UTC | #1
Hi Cleber,

On 10/10/2018 01:26, Cleber Rosa wrote:
> One of the Avocado features relevant to virtualization testing is the
> ability to reuse tests in different scenarios, known as variants.
> This adds a JSON based variants file, that can be used to run most
> tests in a number of different architectures.  It can be run with:
> 
>    $ avocado run \
>      --json-variants-load=tests/acceptance/variants/arch.json \
>      --filter-by-tags='-x86_64' -- tests/acceptance/
> 
> Currently this covers 5 architectures, resulting in the execution
> of 25 different combinations.

Can you provide here the command you used to generate this JSON?

Thanks,

Phil.

> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
>  1 file changed, 92 insertions(+)
>  create mode 100644 tests/acceptance/variants/arch.json
> 
> diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json
> new file mode 100644
> index 0000000000..64ff6ad278
> --- /dev/null
> +++ b/tests/acceptance/variants/arch.json
> @@ -0,0 +1,92 @@
> +[
> +    {
> +        "paths": [
> +            "/run/*"
> +        ],
> +        "variant": [
> +            [
> +                "/run/aarch64",
> +                [
> +                    [
> +                        "/run/aarch64",
> +                        "arch",
> +                        "aarch64"
> +                    ]
> +                ]
> +            ]
> +        ],
> +        "variant_id": "aarch64"
> +    },
> +    {
> +        "paths": [
> +            "/run/*"
> +        ],
> +        "variant": [
> +            [
> +                "/run/ppc",
> +                [
> +                    [
> +                        "/run/ppc",
> +                        "arch",
> +                        "ppc"
> +                    ]
> +                ]
> +            ]
> +        ],
> +        "variant_id": "ppc"
> +    },
> +    {
> +        "paths": [
> +            "/run/*"
> +        ],
> +        "variant": [
> +            [
> +                "/run/ppc64",
> +                [
> +                    [
> +                        "/run/ppc64",
> +                        "arch",
> +                        "ppc64"
> +                    ]
> +                ]
> +            ]
> +        ],
> +        "variant_id": "ppc64"
> +    },
> +    {
> +        "paths": [
> +            "/run/*"
> +        ],
> +        "variant": [
> +            [
> +                "/run/s390x",
> +                [
> +                    [
> +                        "/run/s390x",
> +                        "arch",
> +                        "s390x"
> +                    ]
> +                ]
> +            ]
> +        ],
> +        "variant_id": "s390x"
> +    },
> +    {
> +        "paths": [
> +            "/run/*"
> +        ],
> +        "variant": [
> +            [
> +                "/run/x86_64",
> +                [
> +                    [
> +                        "/run/x86_64",
> +                        "arch",
> +                        "x86_64"
> +                    ]
> +                ]
> +            ]
> +        ],
> +        "variant_id": "x86_64"
> +    }
> +]
>
Philippe Mathieu-Daudé Oct. 10, 2018, 10:59 a.m. UTC | #2
On 10/10/2018 01:26, Cleber Rosa wrote:
> One of the Avocado features relevant to virtualization testing is the
> ability to reuse tests in different scenarios, known as variants.
> This adds a JSON based variants file, that can be used to run most
> tests in a number of different architectures.  It can be run with:
> 
>    $ avocado run \
>      --json-variants-load=tests/acceptance/variants/arch.json \
>      --filter-by-tags='-x86_64' -- tests/acceptance/
> 
> Currently this covers 5 architectures, resulting in the execution
> of 25 different combinations.

IMHO we should add this file with all supported archs, and CANCEL a job
if there is no preferred machine in Qemu::MACHINE_TYPES.

> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
>  1 file changed, 92 insertions(+)
>  create mode 100644 tests/acceptance/variants/arch.json
Cleber Rosa Oct. 10, 2018, 12:48 p.m. UTC | #3
On 10/10/18 6:59 AM, Philippe Mathieu-Daudé wrote:
> On 10/10/2018 01:26, Cleber Rosa wrote:
>> One of the Avocado features relevant to virtualization testing is the
>> ability to reuse tests in different scenarios, known as variants.
>> This adds a JSON based variants file, that can be used to run most
>> tests in a number of different architectures.  It can be run with:
>>
>>    $ avocado run \
>>      --json-variants-load=tests/acceptance/variants/arch.json \
>>      --filter-by-tags='-x86_64' -- tests/acceptance/
>>
>> Currently this covers 5 architectures, resulting in the execution
>> of 25 different combinations.
> 
> IMHO we should add this file with all supported archs, and CANCEL a job
> if there is no preferred machine in Qemu::MACHINE_TYPES.
> 

While your proposal works, I see two issues:

 1) It defines a policy, which will affect all tests, existing and new -
even for tests that don't launch a VM and don't care about the arch.

 2) CANCEL is mostly silent, and the amount of noise in test results
would be quite large.

So, the real problem is that there are other issues when making basic
use of QEMUMachine with other archs.  The idea is to incrementally test
other archs, make the modifications necessary to scripts/qemu.py, and
end up with all architectures covered.

What do you think?

- Cleber.

>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
>>  1 file changed, 92 insertions(+)
>>  create mode 100644 tests/acceptance/variants/arch.json
Cleber Rosa Oct. 10, 2018, 12:59 p.m. UTC | #4
On 10/10/18 6:51 AM, Philippe Mathieu-Daudé wrote:
> Hi Cleber,
> 
> On 10/10/2018 01:26, Cleber Rosa wrote:
>> One of the Avocado features relevant to virtualization testing is the
>> ability to reuse tests in different scenarios, known as variants.
>> This adds a JSON based variants file, that can be used to run most
>> tests in a number of different architectures.  It can be run with:
>>
>>    $ avocado run \
>>      --json-variants-load=tests/acceptance/variants/arch.json \
>>      --filter-by-tags='-x86_64' -- tests/acceptance/
>>
>> Currently this covers 5 architectures, resulting in the execution
>> of 25 different combinations.
> 
> Can you provide here the command you used to generate this JSON?
> 

Yep, there are many ways to do it, one is to have the
varianter-yaml-to-mux plugin installed, and create a YAML file such as:

---- archs.yaml START -----
!mux
x86_64:
  arch: x86_64
aarch64:
  arch: aarch64
ppc64:
  arch: ppc64
ppc:
  arch: ppc
s390x:
  arch: s390x
---- archs.yaml END -----

And run:

 $ avocado run -m archs.yaml -- /bin/true
 $ python -m json.tool ~/avocado/job-results/latest/jobdata/variants.json

I did tweak the "variant_id" value of the resulting JSON file, because
the hashes are somewhat specific to yaml-to-mux, and don't add anything
to the meaning of the variants.

- Cleber.

> Thanks,
> 
> Phil.
> 
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> ---
>>  tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++
>>  1 file changed, 92 insertions(+)
>>  create mode 100644 tests/acceptance/variants/arch.json
>>
>> diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json
>> new file mode 100644
>> index 0000000000..64ff6ad278
>> --- /dev/null
>> +++ b/tests/acceptance/variants/arch.json
>> @@ -0,0 +1,92 @@
>> +[
>> +    {
>> +        "paths": [
>> +            "/run/*"
>> +        ],
>> +        "variant": [
>> +            [
>> +                "/run/aarch64",
>> +                [
>> +                    [
>> +                        "/run/aarch64",
>> +                        "arch",
>> +                        "aarch64"
>> +                    ]
>> +                ]
>> +            ]
>> +        ],
>> +        "variant_id": "aarch64"
>> +    },
>> +    {
>> +        "paths": [
>> +            "/run/*"
>> +        ],
>> +        "variant": [
>> +            [
>> +                "/run/ppc",
>> +                [
>> +                    [
>> +                        "/run/ppc",
>> +                        "arch",
>> +                        "ppc"
>> +                    ]
>> +                ]
>> +            ]
>> +        ],
>> +        "variant_id": "ppc"
>> +    },
>> +    {
>> +        "paths": [
>> +            "/run/*"
>> +        ],
>> +        "variant": [
>> +            [
>> +                "/run/ppc64",
>> +                [
>> +                    [
>> +                        "/run/ppc64",
>> +                        "arch",
>> +                        "ppc64"
>> +                    ]
>> +                ]
>> +            ]
>> +        ],
>> +        "variant_id": "ppc64"
>> +    },
>> +    {
>> +        "paths": [
>> +            "/run/*"
>> +        ],
>> +        "variant": [
>> +            [
>> +                "/run/s390x",
>> +                [
>> +                    [
>> +                        "/run/s390x",
>> +                        "arch",
>> +                        "s390x"
>> +                    ]
>> +                ]
>> +            ]
>> +        ],
>> +        "variant_id": "s390x"
>> +    },
>> +    {
>> +        "paths": [
>> +            "/run/*"
>> +        ],
>> +        "variant": [
>> +            [
>> +                "/run/x86_64",
>> +                [
>> +                    [
>> +                        "/run/x86_64",
>> +                        "arch",
>> +                        "x86_64"
>> +                    ]
>> +                ]
>> +            ]
>> +        ],
>> +        "variant_id": "x86_64"
>> +    }
>> +]
>>
diff mbox series

Patch

diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json
new file mode 100644
index 0000000000..64ff6ad278
--- /dev/null
+++ b/tests/acceptance/variants/arch.json
@@ -0,0 +1,92 @@ 
+[
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/aarch64",
+                [
+                    [
+                        "/run/aarch64",
+                        "arch",
+                        "aarch64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "aarch64"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/ppc",
+                [
+                    [
+                        "/run/ppc",
+                        "arch",
+                        "ppc"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "ppc"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/ppc64",
+                [
+                    [
+                        "/run/ppc64",
+                        "arch",
+                        "ppc64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "ppc64"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/s390x",
+                [
+                    [
+                        "/run/s390x",
+                        "arch",
+                        "s390x"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "s390x"
+    },
+    {
+        "paths": [
+            "/run/*"
+        ],
+        "variant": [
+            [
+                "/run/x86_64",
+                [
+                    [
+                        "/run/x86_64",
+                        "arch",
+                        "x86_64"
+                    ]
+                ]
+            ]
+        ],
+        "variant_id": "x86_64"
+    }
+]