| Message ID | 20190312171824.5134-1-crosa@redhat.com |
|---|---|
| Headers | show |
| Series | Acceptance Tests: target architecture support | expand |
I understand these tests could serve as "meaningful boot tests" in the sense I used in "Maintainers, please tell us how to boot your machines!"[*] Which machine types do they cover? [*] Message-ID: <87d0mwatbu.fsf@dusky.pond.sub.org> https://lists.nongnu.org/archive/html/qemu-devel/2019-03/msg04177.html
On Mon, Mar 18, 2019 at 09:30:13AM +0100, Markus Armbruster wrote: > I understand these tests could serve as "meaningful boot tests" in the > sense I used in "Maintainers, please tell us how to boot your > machines!"[*] Which machine types do they cover? These: * pc (x86_64) * malta (mips) * malta (mips64el) * pseries (ppc64) * virt (aarch64) * virt (arm) * s390-ccw-virtio (s390x) * clipper (alpha)
On Tue, Mar 12, 2019 at 01:18:04PM -0400, Cleber Rosa wrote: [...] Thanks, I'm queueing the whole series, except for two patches: > Cleber Rosa (19): > Acceptance tests: show avocado test execution by default > Acceptance tests: improve docstring on pick_default_qemu_bin() > Acceptance tests: fix doc reference to avocado_qemu directory > Acceptance tests: introduce arch parameter and attribute > Acceptance tests: use "arch:" tag to filter target specific tests > Acceptance tests: look for target architecture in test tags first > Boot Linux Console Test: rename the x86_64 after the arch and machine > Boot Linux Console Test: update the x86_64 kernel > Boot Linux Console Test: add common kernel command line options > Boot Linux Console Test: increase timeout > Boot Linux Console Test: refactor the console watcher into utility > method > scripts/qemu.py: support adding a console with the default serial > device > Boot Linux Console Test: add a test for mips64el + malta > Boot Linux Console Test: add a test for ppc64 + pseries This one, because it times out unless we apply the following patch. > Boot Linux Console Test for ppc64 + pseries: use power8 CPU model This one, because it was nacked by David Gibson. > Boot Linux Console Test: add a test for aarch64 + virt > Boot Linux Console Test: add a test for arm + virt > Boot Linux Console Test: add a test for s390x + s390-ccw-virtio > Boot Linux Console Test: add a test for alpha + clipper > > Philippe Mathieu-Daudé (1): > Boot Linux Console Test: add a test for mips + malta >
The current acceptance tests don't provide any type of architecture information that can be used to influence the selection of the QEMU binary used on them[1]. If one is running tests on a x86_64 host, the default QEMU binary will be "x86_64-softmmu/qemu-system-x86_64". Given the nature of QEMU, some tests will be architecture agnostic, while others will be architecture dependent. The "check-qtest" and "check-qtest-TARGET" make targets exemplify that pattern. For the acceptance tests, the same requirement exists. Tests should be allowed to influence the binary used, and when they don't, a default selection mechanism should kick in[2]. The proposed solution here requires only that an Avocado tag is set, such as: class My(Test): def test_nx_cpu_flag(self): """ :avocado: tags=arch:x86_64 """ test_code() The value of the "arch" key, in this case, "x86_64" will be used when selecting the QEMU binary to use in the test. At the same time, if "x86_64-softmmu" is not a built target, the test will be filtered out by "make check-acceptance"[3]. Besides the convention explained above, where the binary will be selected from the "arch" tag, it's also possible to set an "arch" *parameter* that will also influence the QEMU binary selection: $ avocado run -p arch=aarch64 works-on-many-arches.py Finally, it's also posible to set the "qemu_bin" parameter, which will define (instead of just influencing) the QEMU binary to be used: $ avocado run -p qemu_bin=qemu-bin-aarch64 test.py As examples for the idea proposed here, a number of "boot linux console" tests have been added, for a number of different target architectures. When the build environment includes them (as it has been added to Travis CI jobs) the architecture specific tests will be automatically executed. As mentioned previously, this patch series include ideas present in other patch series, and from different authors. I tried by best to include the information about authorship, but if I missed any, please accept my apologies and let me know. --- [1] - The "boot_linux_console.py" contains a "x86_64" test tag, but that is informational only, in the sense that it's not consumed by the test itself, or used by "make check-acceptance" to filter out tests. [2] - This patch series doesn't attempt to change the default selection mechanism. Possible changes in this area may include looking for any one built binary first, no matter the host architecture. [3] - On a previous proposed version, the test class would look at the "arch" parameter given, and would cancel the test if there wasn't a match. --- Changes from v4: ================ * On "scripts/qemu.py: support adding a console with the default serial device" - Improved commit message (s/the "-serial"/the "-serial" option/) and ( (s/as it is/as is/) (Cornelia) * On "Boot Linux Console Test: update the x86_64 kernel": - Use the standard Fedora download URI for the kernel image (Daniel) * On "Boot Linux Console Test: add a test for ppc64 + pseries" - Use the standard Fedora download URI for the kernel image (Daniel) * On "Boot Linux Console Test: add a test for aarch64 + virt" - Use the standard Fedora download URI for the kernel image (Daniel) * on "Boot Linux Console Test: add a test for arm + virt" - Use the standard Fedora download URI for the kernel image (Daniel) * New commit "Boot Linux Console Test for ppc64 + pseries: use power8 CPU model" - To cope with changes introduced by da874d90a Changes from v3: ================ * On "Acceptance tests: introduce arch parameter and attribute": - Fixed a typo (s/will will/will/) (Cornelia) - Added a comma to the pick_default_qemu_bin() docstring for easier reading (Cornelia) * On "Acceptance tests: use "arch:" tag to filter target specific tests": - Adjust the commit message to the fact that more tests use the "x86_64" tag (Cornelia) - Fixed a typo (s/class, class/class/) (Cornelia) - Dropped the bump in avocado-framework version, given that commit 9531d26c106 did that for other reasons (Cornelia) * On "Boot Linux Console Test: update the x86_64 kernel" - Changed the commit message to be self contained and not use the commit title (Phillipe) * On "scripts/qemu.py: support adding a console with the default serial device": - Changed the wording on the commit message (Cornelia) - Fixed a typo (s/my/by/) (Cornelia) - Dropped the CONSOLE_DEV_TYPES dict and the logic to use, as all targets and machine types tested now work with "-serial" (Cornelia) * On "Boot Linux Console Test: add a test for mips64el + malta": - Dropped duplicate "import os" line (Phillipe) * Dropped "scripts/qemu.py: log QEMU launch command line", already on master Changes from v2: ================ * On "Acceptance tests: introduce arch parameter and attribute": - Made the documentation on the "arch" attribute and parameter clearer as to what the framework does with them, and what tests may use them for (Cornelia). * On "Acceptance tests: use "arch:" tag to filter target specific tests": - Bumped avocado-framework to the latest release (68.0, instead of 67.0 on v2). * On "Boot Linux Console Test: add a test for mips64el + malta": - Broke down kernel source/doc URLs lines on docstring to make patchew happy about number of columns in line * On "Boot Linux Console Test: add a test for aarch64 + virt" - Broke down "kernel_command_line" assignment line to make patchew happy about number of columns in line * On "Boot Linux Console Test: add a test for arm + virt" - Broke down "kernel_command_line" assignment line to make patchew happy about number of columns in line Open issues during v2: ====================== * The timeout change to 90s may not be necessary, or the best idea, given that a possible tcg+ppc64 performance regression has been identified. - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00338.html * A possible race condition has been identified in the aarch64 target, when running on an environment with more than 1 CPUs (initially reported by Wainer). - https://lists.gnu.org/archive/html/qemu-devel/2019-02/msg00192.html Changes from v1: ================ * On "Acceptance tests: introduce arch parameter and attribute": - Added explicit *host system architecture* to the "arch" parameter behavior documentation (Caio / Philippe) - Added explicit arch parameter name in call to pick_default_qemu_bin() (Caio) - Fixed the documentation about the value of the "arch" attribute when a parameter is not given (Wainer). * On "Acceptance tests: use "arch:" tag to filter target specific tests": - Updated "arch" tag on tests "linux_initrd.py" and "virtio_version.py" (Cornelia) - Fixed the documentation about the value of the "arch" attribute (Wainer) * On "Acceptance tests: look for target architecture in test tags first" - Fixed the documentation given that starting with this patch, the "arch" attribute can also be depend on the "arch:" tag (based on previous patch point raised by Wainer). * New patch "Boot Linux Console Test: add common kernel command line options" (Philippe) * New patch "Boot Linux Console Test: increase timeout" (based on reports from Alex) * On "scripts/qemu.py: support adding a console with the default serial device": - Removed all non-essential console device type defintions from CONSOLE_DEV_TYPES. At this point, setting the console will attempt to use "-serial" on all but the "s390-ccw-virtio*" machines (Philippe). - Updated documentation to make it clear that a specific device type will only be picked from CONSOLE_DEV_TYPES if the machine type is set, and a matching entry exists (Wainer). * On "Boot Linux Console Test: add a test for mips + malta": - Replaced "not easily reproducible kernel" for a kernel built and packaged by the Debian distro (Philippe, and previously Alex) - With the change to a kernel within a Debian package, and its use on another commit, added an utility function, "extract_from_deb()", to extract the kernel from the package (Philippe) - Switched to using common kernel command line options (Philippe) * On "Boot Linux Console Test: add a test for mips64el + malta": - Used the "extract_from_deb()" utility function introduced in the previous patch (Philippe) - Switched to using common kernel command line options (Philippe) - Fixed numbered references on the docstring (Wainer) - Fixed tag name on commit message (s/mips/mips64el/) (Wainer) * On "Boot Linux Console Test: add a test for ppc64 + pseries": - Switched to using common kernel command line options (Philippe) * On "Boot Linux Console Test: add a test for aarch64 + virt": - Switched to using common kernel command line options (Philippe) * On "Boot Linux Console Test: add a test for arm + virt" - Switched to using common kernel command line options (Philippe) * On "Boot Linux Console Test: add a test for s390x + s390-ccw-virtio": - Switched to using common kernel command line options (Philippe) * On "Boot Linux Console Test: add a test for alpha + clipper": - Switched to using common kernel command line options (Philippe) --- Git Info: - URI: https://github.com/clebergnu/qemu/tree/sent/target_arch_v5 - Remote: https://github.com/clebergnu/qemu - Branch: sent/target_arch_v5 Travis CI: - Build: https://travis-ci.org/clebergnu/qemu/builds/505337647 Cleber Rosa (19): Acceptance tests: show avocado test execution by default Acceptance tests: improve docstring on pick_default_qemu_bin() Acceptance tests: fix doc reference to avocado_qemu directory Acceptance tests: introduce arch parameter and attribute Acceptance tests: use "arch:" tag to filter target specific tests Acceptance tests: look for target architecture in test tags first Boot Linux Console Test: rename the x86_64 after the arch and machine Boot Linux Console Test: update the x86_64 kernel Boot Linux Console Test: add common kernel command line options Boot Linux Console Test: increase timeout Boot Linux Console Test: refactor the console watcher into utility method scripts/qemu.py: support adding a console with the default serial device Boot Linux Console Test: add a test for mips64el + malta Boot Linux Console Test: add a test for ppc64 + pseries Boot Linux Console Test for ppc64 + pseries: use power8 CPU model Boot Linux Console Test: add a test for aarch64 + virt Boot Linux Console Test: add a test for arm + virt Boot Linux Console Test: add a test for s390x + s390-ccw-virtio Boot Linux Console Test: add a test for alpha + clipper Philippe Mathieu-Daudé (1): Boot Linux Console Test: add a test for mips + malta .travis.yml | 4 +- docs/devel/testing.rst | 35 +++- python/qemu/__init__.py | 48 ++--- tests/Makefile.include | 5 +- tests/acceptance/avocado_qemu/__init__.py | 23 ++- tests/acceptance/boot_linux_console.py | 231 ++++++++++++++++++++-- tests/acceptance/linux_initrd.py | 2 +- tests/acceptance/virtio_version.py | 2 +- 8 files changed, 289 insertions(+), 61 deletions(-)