| Message ID | 20180525005839.11556-1-crosa@redhat.com |
|---|---|
| Headers | show |
| Series | Acceptance/functional tests | expand |
On 05/24/2018 08:58 PM, Cleber Rosa wrote: > TL;DR > ===== > > Another version, with a minimalist approach, to the acceptance tests > infrastructure for QEMU, based on the Avocado Testing Framework. > > Background > ========== > > The previous version, still considered an RFC, was sent to the list by > Eduardo[1] was based on the work held in Amador's branch[2]. After > reviewing in under a different light, including the experiences > done and reported by Philippe[3]. > (major sigh for killing a line, writing a non-sense sentence) ... it was clear to me that a different approach would be better. - Cleber. > Differences from previous versions > ================================== > > The main difference is that this series include only the minimal > changes deemed necessary to have a starting point. I like to think > that it's better connected to the QEMU community and project needs, > and will hopefully allow for a more organic growth. > > Since this version has less features than the previous versions, > provided it's accepted, these are the next probable development tasks: > > * Provide a simple variants mechanism to allow the same tests to be > run under different targets, machine models and devices (present on > the previous versions as a "YAML to Mux" file with architecture > definitions) > * Implement QEMUMachine migration support (present on the previous > version in the "avocado_qemu.test._VM" class) > * Implement Guest OS image selection and download (mostly an Avocado > feature, paired with a parameter convention and cloud-init support > code) > * Implement interactive support for Guest OS sessions (present on > the previous versions, supported by the aexpect Python module) > > Even though this version shares very little (if any) code with the > previous versions, the following is a list of noteworthy changes: > > * Tests directory is now "tests/acceptance" (was "tests/avocado") > * Base test class is now "avocado_qemu.Test" (was > "avocado_qemu.test.QemuTest") > * Base test class is now hosted in "avocado_qemu/__init__.py" (was > "avocado_qemu/test.py") > * Direct use of "qemu.QEMUMachine", that is, the > avocado_qemu.test._VM class is gone > * avocado_qemu.Test won't search for QEMU binaries on $PATH. To use > QEMU binary on a custom system location it's necessary to use the > "qemu_bin" parameter > * Example test in README.rst is distributed as a real test > ("test_version.py") > * A new "Linux boot console" test, loosely modeled after Phillipe's > use case > > Commit summary > ============== > > Cleber Rosa (5): > Add functional/acceptance tests infrastructure > scripts/qemu.py: allow adding to the list of extra arguments > Acceptance tests: add quick VNC tests > scripts/qemu.py: introduce set_console() method > Acceptance tests: add Linux kernel boot and console checking test > > scripts/qemu.py | 103 +++++++++++++++- > scripts/test_qemu.py | 176 +++++++++++++++++++++++++++ > tests/acceptance/README.rst | 141 +++++++++++++++++++++ > tests/acceptance/avocado_qemu/__init__.py | 45 +++++++ > tests/acceptance/test_boot_linux_console.py | 37 ++++++ > tests/acceptance/test_version.py | 13 ++ > tests/acceptance/test_vnc.py | 50 ++++++++ > 7 files changed, 564 insertions(+), 1 deletion(-) > create mode 100644 scripts/test_qemu.py > create mode 100644 tests/acceptance/README.rst > create mode 100644 tests/acceptance/avocado_qemu/__init__.py > create mode 100644 tests/acceptance/test_boot_linux_console.py > create mode 100644 tests/acceptance/test_version.py > create mode 100644 tests/acceptance/test_vnc.py > > --- > > [1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03443.html > [2] https://github.com/apahim/qemu/commits/avocado_qemu > [3] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03076.html > >
On Thu, 05/24 20:58, Cleber Rosa wrote: > TL;DR > ===== > > Another version, with a minimalist approach, to the acceptance tests > infrastructure for QEMU, based on the Avocado Testing Framework. > > Background > ========== > > The previous version, still considered an RFC, was sent to the list by > Eduardo[1] was based on the work held in Amador's branch[2]. After > reviewing in under a different light, including the experiences > done and reported by Philippe[3]. > > Differences from previous versions > ================================== > > The main difference is that this series include only the minimal > changes deemed necessary to have a starting point. I like to think > that it's better connected to the QEMU community and project needs, > and will hopefully allow for a more organic growth. > > Since this version has less features than the previous versions, > provided it's accepted, these are the next probable development tasks: > > * Provide a simple variants mechanism to allow the same tests to be > run under different targets, machine models and devices (present on > the previous versions as a "YAML to Mux" file with architecture > definitions) > * Implement QEMUMachine migration support (present on the previous > version in the "avocado_qemu.test._VM" class) > * Implement Guest OS image selection and download (mostly an Avocado > feature, paired with a parameter convention and cloud-init support > code) > * Implement interactive support for Guest OS sessions (present on > the previous versions, supported by the aexpect Python module) > > Even though this version shares very little (if any) code with the > previous versions, the following is a list of noteworthy changes: > > * Tests directory is now "tests/acceptance" (was "tests/avocado") > * Base test class is now "avocado_qemu.Test" (was > "avocado_qemu.test.QemuTest") > * Base test class is now hosted in "avocado_qemu/__init__.py" (was > "avocado_qemu/test.py") > * Direct use of "qemu.QEMUMachine", that is, the > avocado_qemu.test._VM class is gone > * avocado_qemu.Test won't search for QEMU binaries on $PATH. To use > QEMU binary on a custom system location it's necessary to use the > "qemu_bin" parameter > * Example test in README.rst is distributed as a real test > ("test_version.py") > * A new "Linux boot console" test, loosely modeled after Phillipe's > use case I like the direction this series is leading us to. Making it easy to write 'correct' and 'effective' tests is the most important thing IMO, and the few tests you add here look promising. I'm also curious to see a test case involving a guest OS image. (I'm also open to a re-implementation of tests/vm/basevm.py with Avocado, if you find it worthwhile.) Fam > > Commit summary > ============== > > Cleber Rosa (5): > Add functional/acceptance tests infrastructure > scripts/qemu.py: allow adding to the list of extra arguments > Acceptance tests: add quick VNC tests > scripts/qemu.py: introduce set_console() method > Acceptance tests: add Linux kernel boot and console checking test > > scripts/qemu.py | 103 +++++++++++++++- > scripts/test_qemu.py | 176 +++++++++++++++++++++++++++ > tests/acceptance/README.rst | 141 +++++++++++++++++++++ > tests/acceptance/avocado_qemu/__init__.py | 45 +++++++ > tests/acceptance/test_boot_linux_console.py | 37 ++++++ > tests/acceptance/test_version.py | 13 ++ > tests/acceptance/test_vnc.py | 50 ++++++++ > 7 files changed, 564 insertions(+), 1 deletion(-) > create mode 100644 scripts/test_qemu.py > create mode 100644 tests/acceptance/README.rst > create mode 100644 tests/acceptance/avocado_qemu/__init__.py > create mode 100644 tests/acceptance/test_boot_linux_console.py > create mode 100644 tests/acceptance/test_version.py > create mode 100644 tests/acceptance/test_vnc.py > > --- > > [1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03443.html > [2] https://github.com/apahim/qemu/commits/avocado_qemu > [3] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03076.html >
On 05/25/2018 02:08 AM, Fam Zheng wrote: > On Thu, 05/24 20:58, Cleber Rosa wrote: >> TL;DR >> ===== >> >> Another version, with a minimalist approach, to the acceptance tests >> infrastructure for QEMU, based on the Avocado Testing Framework. >> >> Background >> ========== >> >> The previous version, still considered an RFC, was sent to the list by >> Eduardo[1] was based on the work held in Amador's branch[2]. After >> reviewing in under a different light, including the experiences >> done and reported by Philippe[3]. >> >> Differences from previous versions >> ================================== >> >> The main difference is that this series include only the minimal >> changes deemed necessary to have a starting point. I like to think >> that it's better connected to the QEMU community and project needs, >> and will hopefully allow for a more organic growth. >> >> Since this version has less features than the previous versions, >> provided it's accepted, these are the next probable development tasks: >> >> * Provide a simple variants mechanism to allow the same tests to be >> run under different targets, machine models and devices (present on >> the previous versions as a "YAML to Mux" file with architecture >> definitions) >> * Implement QEMUMachine migration support (present on the previous >> version in the "avocado_qemu.test._VM" class) >> * Implement Guest OS image selection and download (mostly an Avocado >> feature, paired with a parameter convention and cloud-init support >> code) >> * Implement interactive support for Guest OS sessions (present on >> the previous versions, supported by the aexpect Python module) >> >> Even though this version shares very little (if any) code with the >> previous versions, the following is a list of noteworthy changes: >> >> * Tests directory is now "tests/acceptance" (was "tests/avocado") >> * Base test class is now "avocado_qemu.Test" (was >> "avocado_qemu.test.QemuTest") >> * Base test class is now hosted in "avocado_qemu/__init__.py" (was >> "avocado_qemu/test.py") >> * Direct use of "qemu.QEMUMachine", that is, the >> avocado_qemu.test._VM class is gone >> * avocado_qemu.Test won't search for QEMU binaries on $PATH. To use >> QEMU binary on a custom system location it's necessary to use the >> "qemu_bin" parameter >> * Example test in README.rst is distributed as a real test >> ("test_version.py") >> * A new "Linux boot console" test, loosely modeled after Phillipe's >> use case > > I like the direction this series is leading us to. Making it easy to write > 'correct' and 'effective' tests is the most important thing IMO, and the few > tests you add here look promising. > The soft targets that these adjectives ("easy", "correct" and "effective") bring, are kind of hard to get right, but I'm really glad you like what you see. > I'm also curious to see a test case involving a guest OS image. (I'm also open to > a re-implementation of tests/vm/basevm.py with Avocado, if you find it > worthwhile.) > We had such tests in the previous incarnations of this work[1]. But by doing it in smaller steps, I believe we can achieve a better result, specially when it comes to the exposed API. - Cleber. [1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03466.html > Fam > >> >> Commit summary >> ============== >> >> Cleber Rosa (5): >> Add functional/acceptance tests infrastructure >> scripts/qemu.py: allow adding to the list of extra arguments >> Acceptance tests: add quick VNC tests >> scripts/qemu.py: introduce set_console() method >> Acceptance tests: add Linux kernel boot and console checking test >> >> scripts/qemu.py | 103 +++++++++++++++- >> scripts/test_qemu.py | 176 +++++++++++++++++++++++++++ >> tests/acceptance/README.rst | 141 +++++++++++++++++++++ >> tests/acceptance/avocado_qemu/__init__.py | 45 +++++++ >> tests/acceptance/test_boot_linux_console.py | 37 ++++++ >> tests/acceptance/test_version.py | 13 ++ >> tests/acceptance/test_vnc.py | 50 ++++++++ >> 7 files changed, 564 insertions(+), 1 deletion(-) >> create mode 100644 scripts/test_qemu.py >> create mode 100644 tests/acceptance/README.rst >> create mode 100644 tests/acceptance/avocado_qemu/__init__.py >> create mode 100644 tests/acceptance/test_boot_linux_console.py >> create mode 100644 tests/acceptance/test_version.py >> create mode 100644 tests/acceptance/test_vnc.py >> >> --- >> >> [1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03443.html >> [2] https://github.com/apahim/qemu/commits/avocado_qemu >> [3] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03076.html >> >
TL;DR ===== Another version, with a minimalist approach, to the acceptance tests infrastructure for QEMU, based on the Avocado Testing Framework. Background ========== The previous version, still considered an RFC, was sent to the list by Eduardo[1] was based on the work held in Amador's branch[2]. After reviewing in under a different light, including the experiences done and reported by Philippe[3]. Differences from previous versions ================================== The main difference is that this series include only the minimal changes deemed necessary to have a starting point. I like to think that it's better connected to the QEMU community and project needs, and will hopefully allow for a more organic growth. Since this version has less features than the previous versions, provided it's accepted, these are the next probable development tasks: * Provide a simple variants mechanism to allow the same tests to be run under different targets, machine models and devices (present on the previous versions as a "YAML to Mux" file with architecture definitions) * Implement QEMUMachine migration support (present on the previous version in the "avocado_qemu.test._VM" class) * Implement Guest OS image selection and download (mostly an Avocado feature, paired with a parameter convention and cloud-init support code) * Implement interactive support for Guest OS sessions (present on the previous versions, supported by the aexpect Python module) Even though this version shares very little (if any) code with the previous versions, the following is a list of noteworthy changes: * Tests directory is now "tests/acceptance" (was "tests/avocado") * Base test class is now "avocado_qemu.Test" (was "avocado_qemu.test.QemuTest") * Base test class is now hosted in "avocado_qemu/__init__.py" (was "avocado_qemu/test.py") * Direct use of "qemu.QEMUMachine", that is, the avocado_qemu.test._VM class is gone * avocado_qemu.Test won't search for QEMU binaries on $PATH. To use QEMU binary on a custom system location it's necessary to use the "qemu_bin" parameter * Example test in README.rst is distributed as a real test ("test_version.py") * A new "Linux boot console" test, loosely modeled after Phillipe's use case Commit summary ============== Cleber Rosa (5): Add functional/acceptance tests infrastructure scripts/qemu.py: allow adding to the list of extra arguments Acceptance tests: add quick VNC tests scripts/qemu.py: introduce set_console() method Acceptance tests: add Linux kernel boot and console checking test scripts/qemu.py | 103 +++++++++++++++- scripts/test_qemu.py | 176 +++++++++++++++++++++++++++ tests/acceptance/README.rst | 141 +++++++++++++++++++++ tests/acceptance/avocado_qemu/__init__.py | 45 +++++++ tests/acceptance/test_boot_linux_console.py | 37 ++++++ tests/acceptance/test_version.py | 13 ++ tests/acceptance/test_vnc.py | 50 ++++++++ 7 files changed, 564 insertions(+), 1 deletion(-) create mode 100644 scripts/test_qemu.py create mode 100644 tests/acceptance/README.rst create mode 100644 tests/acceptance/avocado_qemu/__init__.py create mode 100644 tests/acceptance/test_boot_linux_console.py create mode 100644 tests/acceptance/test_version.py create mode 100644 tests/acceptance/test_vnc.py --- [1] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03443.html [2] https://github.com/apahim/qemu/commits/avocado_qemu [3] https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg03076.html