| Message ID | 20180813101453.10200-1-e.emanuelegiuseppe@gmail.com |
|---|---|
| Headers | show |
| Series | Qtest driver framework | expand |
On Mon, Aug 13, 2018 at 12:14:20PM +0200, Emanuele Giuseppe Esposito wrote: > Qgraph API for the qtest driver framework > > This series of patches introduce a different qtest driver > organization, viewing machines, drivers and tests as node in a > graph, each having one or multiple edges relations. > > The idea is to have a framework where each test asks for a specific > driver, and the framework takes care of allocating the proper devices > required and passing the correct command line arguments to QEMU. > > A node can be of four types: > - MACHINE: for example "arm/raspi2" > - DRIVER: for example "generic-sdhci" > - INTERFACE: for example "sdhci" (interface for all "-sdhci" drivers) > - TEST: for example "sdhci-test", consumes an interface and tests > the functions provided > > An edge relation between two nodes (drivers or machines) X and Y can be: > - X CONSUMES Y: Y can be plugged into X > - X PRODUCES Y: X provides the interface Y > - X CONTAINS Y: Y is part of X component > > Basic framework steps are the following: > - All nodes and edges are created in their respective machine/driver/test files > - The framework starts QEMU and asks for a list of available devices > and machines > - The framework walks the graph starting from the available machines and > performs a Depth First Search for tests > - Once a test is found, the path is walked again and all drivers are > allocated accordingly and the final interface is passed to the test > - The test is executed > - Unused objects are cleaned and the path discovery is continued > > Depending on the QEMU binary used, only some drivers/machines will be available > and only test that are reached by them will be executed. > > This work is being done as Google Summer of Code 2018 project for QEMU, > my mentors are Paolo Bonzini and Laurent Vivier. > Additional infos on the project can be found at: > https://wiki.qemu.org/Features/qtest_driver_framework Seems pretty clean to me. Except that patch 33 needs to be fixed and fix squashed: Acked-by: Michael S. Tsirkin <mst@redhat.com> > v3: > - Minor fixes regarding memory leaks and naming > > Signed-off-by: Emanuele Giuseppe Esposito <e.emanuelegiuseppe@gmail.com> > > Emanuele Giuseppe Esposito (32): > tests: qgraph API for the qtest driver framework > tests/qgraph: rename qpci_init_pc and qpci_init_spapr functions > tests/qgraph: pci-pc driver and interface nodes > tests/qgraph: x86_64/pc machine node > tests/qgraph: sdhci driver and interface nodes > tests/qgraph: sdhci test node > tests/qgraph: arm/raspi2 machine node > tests/qgraph: pci-spapr driver and interface nodes > tests/qgraph: ppc64/pseries machine node > tests/qgraph: has_buggy_msi flag > tests/qgraph: e1000e driver and interface nodes > tests/qgraph: e1000e-test node > tests/qgraph: virtio_start_device function > tests/qgraph: virtio-pci driver and interface nodes > tests/qgraph: virtio-mmio driver and interface nodes > tests/qgraph: arm/virt machine node > tests/qgraph: virtio-serial driver and interface nodes > tests/qgraph: virtio-console test node > tests/qgraph: virtio-serial test node > tests/qgraph: virtio-9p driver and interface nodes > tests/qgraph: virtio-9p test node > tests/qgraph: virtio-balloon driver and interface nodes > tests/qgraph: virtio-balloon test node > tests/qgraph: virtio-rng driver and interface nodes > tests/qgraph: virtio-rng test node > tests/qgraph: virtio-blk driver and interface nodes > tests/qgraph: virtio-blk test node > tests/qgraph: virtio-net driver and interface nodes > tests/qgraph: virtio-net test node > tests/qgraph: virtio-scsi driver and interface nodes > tests/qgraph: virtio-scsi test node > tests/qgraph: temporarly commented vhost-user-test > > Paolo Bonzini (1): > tests: virtio: separate ccw tests from libqos > > configure | 2 +- > include/qemu/module.h | 2 + > tests/Makefile.include | 79 +-- > tests/e1000e-test.c | 354 +++---------- > tests/i440fx-test.c | 2 +- > tests/ide-test.c | 2 +- > tests/libqos/ahci.c | 2 +- > tests/libqos/e1000e.c | 262 ++++++++++ > tests/libqos/e1000e.h | 53 ++ > tests/libqos/libqos-pc.c | 2 +- > tests/libqos/libqos-spapr.c | 2 +- > tests/libqos/libqos.c | 2 +- > tests/libqos/libqos.h | 2 +- > tests/libqos/pci-pc.c | 86 ++-- > tests/libqos/pci-pc.h | 22 +- > tests/libqos/pci-spapr.c | 119 +++-- > tests/libqos/pci-spapr.h | 26 +- > tests/libqos/pci.c | 37 +- > tests/libqos/pci.h | 15 + > tests/libqos/ppc64_pseries-machine.c | 111 +++++ > tests/libqos/qgraph.c | 721 +++++++++++++++++++++++++++ > tests/libqos/qgraph.h | 515 +++++++++++++++++++ > tests/libqos/qgraph_extra.h | 263 ++++++++++ > tests/libqos/raspi2-machine.c | 82 +++ > tests/libqos/sdhci.c | 163 ++++++ > tests/libqos/sdhci.h | 69 +++ > tests/libqos/virt-machine.c | 90 ++++ > tests/libqos/virtio-9p.c | 165 ++++++ > tests/libqos/virtio-9p.h | 42 ++ > tests/libqos/virtio-balloon.c | 111 +++++ > tests/libqos/virtio-balloon.h | 39 ++ > tests/libqos/virtio-blk.c | 126 +++++ > tests/libqos/virtio-blk.h | 40 ++ > tests/libqos/virtio-mmio.c | 66 ++- > tests/libqos/virtio-mmio.h | 5 +- > tests/libqos/virtio-net.c | 179 +++++++ > tests/libqos/virtio-net.h | 41 ++ > tests/libqos/virtio-pci.c | 80 ++- > tests/libqos/virtio-pci.h | 12 + > tests/libqos/virtio-rng.c | 108 ++++ > tests/libqos/virtio-rng.h | 39 ++ > tests/libqos/virtio-scsi.c | 117 +++++ > tests/libqos/virtio-scsi.h | 39 ++ > tests/libqos/virtio-serial.c | 109 ++++ > tests/libqos/virtio-serial.h | 39 ++ > tests/libqos/virtio.c | 8 + > tests/libqos/virtio.h | 1 + > tests/libqos/x86_64_pc-machine.c | 110 ++++ > tests/libqtest.h | 3 + > tests/q35-test.c | 4 +- > tests/qos-test.c | 481 ++++++++++++++++++ > tests/rtl8139-test.c | 2 +- > tests/sdhci-test.c | 222 +++------ > tests/tco-test.c | 2 +- > tests/test-qgraph.c | 446 +++++++++++++++++ > tests/usb-hcd-ehci-test.c | 2 +- > tests/vhost-user-test.c | 6 +- > tests/virtio-9p-test.c | 221 +++----- > tests/virtio-balloon-test.c | 22 +- > tests/virtio-blk-test.c | 473 +++++++----------- > tests/virtio-ccw-test.c | 121 +++++ > tests/virtio-console-test.c | 30 +- > tests/virtio-net-test.c | 163 ++---- > tests/virtio-rng-test.c | 25 +- > tests/virtio-scsi-test.c | 155 +++--- > tests/virtio-serial-test.c | 27 +- > 66 files changed, 5674 insertions(+), 1292 deletions(-) > create mode 100644 tests/libqos/e1000e.c > create mode 100644 tests/libqos/e1000e.h > create mode 100644 tests/libqos/ppc64_pseries-machine.c > create mode 100644 tests/libqos/qgraph.c > create mode 100644 tests/libqos/qgraph.h > create mode 100644 tests/libqos/qgraph_extra.h > create mode 100644 tests/libqos/raspi2-machine.c > create mode 100644 tests/libqos/sdhci.c > create mode 100644 tests/libqos/sdhci.h > create mode 100644 tests/libqos/virt-machine.c > create mode 100644 tests/libqos/virtio-9p.c > create mode 100644 tests/libqos/virtio-9p.h > create mode 100644 tests/libqos/virtio-balloon.c > create mode 100644 tests/libqos/virtio-balloon.h > create mode 100644 tests/libqos/virtio-blk.c > create mode 100644 tests/libqos/virtio-blk.h > create mode 100644 tests/libqos/virtio-net.c > create mode 100644 tests/libqos/virtio-net.h > create mode 100644 tests/libqos/virtio-rng.c > create mode 100644 tests/libqos/virtio-rng.h > create mode 100644 tests/libqos/virtio-scsi.c > create mode 100644 tests/libqos/virtio-scsi.h > create mode 100644 tests/libqos/virtio-serial.c > create mode 100644 tests/libqos/virtio-serial.h > create mode 100644 tests/libqos/x86_64_pc-machine.c > create mode 100644 tests/qos-test.c > create mode 100644 tests/test-qgraph.c > create mode 100644 tests/virtio-ccw-test.c > > -- > 2.17.1
Emanuele Giuseppe Esposito <e.emanuelegiuseppe@gmail.com> writes: > Qgraph API for the qtest driver framework > > This series of patches introduce a different qtest driver > organization, viewing machines, drivers and tests as node in a > graph, each having one or multiple edges relations. > > The idea is to have a framework where each test asks for a specific > driver, and the framework takes care of allocating the proper devices > required and passing the correct command line arguments to QEMU. > > A node can be of four types: > - MACHINE: for example "arm/raspi2" > - DRIVER: for example "generic-sdhci" > - INTERFACE: for example "sdhci" (interface for all "-sdhci" drivers) > - TEST: for example "sdhci-test", consumes an interface and tests > the functions provided > > An edge relation between two nodes (drivers or machines) X and Y can be: > - X CONSUMES Y: Y can be plugged into X > - X PRODUCES Y: X provides the interface Y > - X CONTAINS Y: Y is part of X component > > Basic framework steps are the following: > - All nodes and edges are created in their respective machine/driver/test files > - The framework starts QEMU and asks for a list of available devices > and machines > - The framework walks the graph starting from the available machines and > performs a Depth First Search for tests > - Once a test is found, the path is walked again and all drivers are > allocated accordingly and the final interface is passed to the test > - The test is executed > - Unused objects are cleaned and the path discovery is continued > > Depending on the QEMU binary used, only some drivers/machines will be available > and only test that are reached by them will be executed. > > This work is being done as Google Summer of Code 2018 project for QEMU, > my mentors are Paolo Bonzini and Laurent Vivier. > Additional infos on the project can be found at: > https://wiki.qemu.org/Features/qtest_driver_framework You've likely answered this question somewhere already, but it should be answered right here: what does this framework buy us? I figure the main benefit is running tests in all possible contexts, with automated context setup. Instead, you feed the framework declarations of how things are related (the graph stuff described above). Taking a step back: your cover letter describes a solution at a high level. That's good. But it should *first* describe the problem you're trying to solve. > v3: > - Minor fixes regarding memory leaks and naming > > Signed-off-by: Emanuele Giuseppe Esposito <e.emanuelegiuseppe@gmail.com>
On 15/08/2018 14:38, Markus Armbruster wrote: > Emanuele Giuseppe Esposito <e.emanuelegiuseppe@gmail.com> writes: > >> Qgraph API for the qtest driver framework >> >> This series of patches introduce a different qtest driver >> organization, viewing machines, drivers and tests as node in a >> graph, each having one or multiple edges relations. >> >> The idea is to have a framework where each test asks for a specific >> driver, and the framework takes care of allocating the proper devices >> required and passing the correct command line arguments to QEMU. >> >> A node can be of four types: >> - MACHINE: for example "arm/raspi2" >> - DRIVER: for example "generic-sdhci" >> - INTERFACE: for example "sdhci" (interface for all "-sdhci" drivers) >> - TEST: for example "sdhci-test", consumes an interface and tests >> the functions provided >> >> An edge relation between two nodes (drivers or machines) X and Y can be: >> - X CONSUMES Y: Y can be plugged into X >> - X PRODUCES Y: X provides the interface Y >> - X CONTAINS Y: Y is part of X component >> >> Basic framework steps are the following: >> - All nodes and edges are created in their respective machine/driver/test files >> - The framework starts QEMU and asks for a list of available devices >> and machines >> - The framework walks the graph starting from the available machines and >> performs a Depth First Search for tests >> - Once a test is found, the path is walked again and all drivers are >> allocated accordingly and the final interface is passed to the test >> - The test is executed >> - Unused objects are cleaned and the path discovery is continued >> >> Depending on the QEMU binary used, only some drivers/machines will be available >> and only test that are reached by them will be executed. >> >> This work is being done as Google Summer of Code 2018 project for QEMU, >> my mentors are Paolo Bonzini and Laurent Vivier. >> Additional infos on the project can be found at: >> https://wiki.qemu.org/Features/qtest_driver_framework > You've likely answered this question somewhere already, but it should be > answered right here: what does this framework buy us? > > I figure the main benefit is running tests in all possible contexts, > with automated context setup. Instead, you feed the framework > declarations of how things are related (the graph stuff described > above). You're right, that is the main benefit: running tests in all possible contexts, with automated context setup. Before, tests where basically hard-coded to run on a single architecture, or even a single machine type. Now, qgraph allows this kind of higher level view of the devices and their relationships, where tests generally consume an interface that hides the underneath implementation (for example, for sdhci-pci device, it consumes a "pci-bus", without knowing that the bus can be a pci-pc or pci-spapr). This allows developers to add test without knowing or forcing the machine/architecture/additional required devices, they just need to know what interface the test can consume (and if not present, implement a node representing the device they want to test). For example, if you look at virtio-9p-test, you can see that the old test was just testing "virtio-9p-pci", while using the qgraph I was able to add "virtio-9p-device" too, without any additional change to the test, improving test coverage. The other benefit is the automated context setup: as Paolo said once, every test basically re-invented the wheel in its own way. Every test had a main function, a section where qemu was started with proper devices, and a section where everything was cleaned up and shut down. Qgraph handles all these operations automatically, so the developer does not need to care about that. Just add the test function to the graph, and it will take care of running the test with the proper setup. Of course this requires a minimal knowledge of the graph and the relationships you are going to use, since each test will receive structures representing device from the other nodes in the graph. > Taking a step back: your cover letter describes a solution at a high > level. That's good. But it should *first* describe the problem you're > trying to solve. Do you think this explanation of the problem is fine? I am not very experienced with patches, so in case it is, should I re-write the cover letter adding the above explanations or is fine like this? In case I should do it, should I submit the whole series again? >> v3: >> - Minor fixes regarding memory leaks and naming >> >> Signed-off-by: Emanuele Giuseppe Esposito <e.emanuelegiuseppe@gmail.com>
Qgraph API for the qtest driver framework This series of patches introduce a different qtest driver organization, viewing machines, drivers and tests as node in a graph, each having one or multiple edges relations. The idea is to have a framework where each test asks for a specific driver, and the framework takes care of allocating the proper devices required and passing the correct command line arguments to QEMU. A node can be of four types: - MACHINE: for example "arm/raspi2" - DRIVER: for example "generic-sdhci" - INTERFACE: for example "sdhci" (interface for all "-sdhci" drivers) - TEST: for example "sdhci-test", consumes an interface and tests the functions provided An edge relation between two nodes (drivers or machines) X and Y can be: - X CONSUMES Y: Y can be plugged into X - X PRODUCES Y: X provides the interface Y - X CONTAINS Y: Y is part of X component Basic framework steps are the following: - All nodes and edges are created in their respective machine/driver/test files - The framework starts QEMU and asks for a list of available devices and machines - The framework walks the graph starting from the available machines and performs a Depth First Search for tests - Once a test is found, the path is walked again and all drivers are allocated accordingly and the final interface is passed to the test - The test is executed - Unused objects are cleaned and the path discovery is continued Depending on the QEMU binary used, only some drivers/machines will be available and only test that are reached by them will be executed. This work is being done as Google Summer of Code 2018 project for QEMU, my mentors are Paolo Bonzini and Laurent Vivier. Additional infos on the project can be found at: https://wiki.qemu.org/Features/qtest_driver_framework v3: - Minor fixes regarding memory leaks and naming Signed-off-by: Emanuele Giuseppe Esposito <e.emanuelegiuseppe@gmail.com> Emanuele Giuseppe Esposito (32): tests: qgraph API for the qtest driver framework tests/qgraph: rename qpci_init_pc and qpci_init_spapr functions tests/qgraph: pci-pc driver and interface nodes tests/qgraph: x86_64/pc machine node tests/qgraph: sdhci driver and interface nodes tests/qgraph: sdhci test node tests/qgraph: arm/raspi2 machine node tests/qgraph: pci-spapr driver and interface nodes tests/qgraph: ppc64/pseries machine node tests/qgraph: has_buggy_msi flag tests/qgraph: e1000e driver and interface nodes tests/qgraph: e1000e-test node tests/qgraph: virtio_start_device function tests/qgraph: virtio-pci driver and interface nodes tests/qgraph: virtio-mmio driver and interface nodes tests/qgraph: arm/virt machine node tests/qgraph: virtio-serial driver and interface nodes tests/qgraph: virtio-console test node tests/qgraph: virtio-serial test node tests/qgraph: virtio-9p driver and interface nodes tests/qgraph: virtio-9p test node tests/qgraph: virtio-balloon driver and interface nodes tests/qgraph: virtio-balloon test node tests/qgraph: virtio-rng driver and interface nodes tests/qgraph: virtio-rng test node tests/qgraph: virtio-blk driver and interface nodes tests/qgraph: virtio-blk test node tests/qgraph: virtio-net driver and interface nodes tests/qgraph: virtio-net test node tests/qgraph: virtio-scsi driver and interface nodes tests/qgraph: virtio-scsi test node tests/qgraph: temporarly commented vhost-user-test Paolo Bonzini (1): tests: virtio: separate ccw tests from libqos configure | 2 +- include/qemu/module.h | 2 + tests/Makefile.include | 79 +-- tests/e1000e-test.c | 354 +++---------- tests/i440fx-test.c | 2 +- tests/ide-test.c | 2 +- tests/libqos/ahci.c | 2 +- tests/libqos/e1000e.c | 262 ++++++++++ tests/libqos/e1000e.h | 53 ++ tests/libqos/libqos-pc.c | 2 +- tests/libqos/libqos-spapr.c | 2 +- tests/libqos/libqos.c | 2 +- tests/libqos/libqos.h | 2 +- tests/libqos/pci-pc.c | 86 ++-- tests/libqos/pci-pc.h | 22 +- tests/libqos/pci-spapr.c | 119 +++-- tests/libqos/pci-spapr.h | 26 +- tests/libqos/pci.c | 37 +- tests/libqos/pci.h | 15 + tests/libqos/ppc64_pseries-machine.c | 111 +++++ tests/libqos/qgraph.c | 721 +++++++++++++++++++++++++++ tests/libqos/qgraph.h | 515 +++++++++++++++++++ tests/libqos/qgraph_extra.h | 263 ++++++++++ tests/libqos/raspi2-machine.c | 82 +++ tests/libqos/sdhci.c | 163 ++++++ tests/libqos/sdhci.h | 69 +++ tests/libqos/virt-machine.c | 90 ++++ tests/libqos/virtio-9p.c | 165 ++++++ tests/libqos/virtio-9p.h | 42 ++ tests/libqos/virtio-balloon.c | 111 +++++ tests/libqos/virtio-balloon.h | 39 ++ tests/libqos/virtio-blk.c | 126 +++++ tests/libqos/virtio-blk.h | 40 ++ tests/libqos/virtio-mmio.c | 66 ++- tests/libqos/virtio-mmio.h | 5 +- tests/libqos/virtio-net.c | 179 +++++++ tests/libqos/virtio-net.h | 41 ++ tests/libqos/virtio-pci.c | 80 ++- tests/libqos/virtio-pci.h | 12 + tests/libqos/virtio-rng.c | 108 ++++ tests/libqos/virtio-rng.h | 39 ++ tests/libqos/virtio-scsi.c | 117 +++++ tests/libqos/virtio-scsi.h | 39 ++ tests/libqos/virtio-serial.c | 109 ++++ tests/libqos/virtio-serial.h | 39 ++ tests/libqos/virtio.c | 8 + tests/libqos/virtio.h | 1 + tests/libqos/x86_64_pc-machine.c | 110 ++++ tests/libqtest.h | 3 + tests/q35-test.c | 4 +- tests/qos-test.c | 481 ++++++++++++++++++ tests/rtl8139-test.c | 2 +- tests/sdhci-test.c | 222 +++------ tests/tco-test.c | 2 +- tests/test-qgraph.c | 446 +++++++++++++++++ tests/usb-hcd-ehci-test.c | 2 +- tests/vhost-user-test.c | 6 +- tests/virtio-9p-test.c | 221 +++----- tests/virtio-balloon-test.c | 22 +- tests/virtio-blk-test.c | 473 +++++++----------- tests/virtio-ccw-test.c | 121 +++++ tests/virtio-console-test.c | 30 +- tests/virtio-net-test.c | 163 ++---- tests/virtio-rng-test.c | 25 +- tests/virtio-scsi-test.c | 155 +++--- tests/virtio-serial-test.c | 27 +- 66 files changed, 5674 insertions(+), 1292 deletions(-) create mode 100644 tests/libqos/e1000e.c create mode 100644 tests/libqos/e1000e.h create mode 100644 tests/libqos/ppc64_pseries-machine.c create mode 100644 tests/libqos/qgraph.c create mode 100644 tests/libqos/qgraph.h create mode 100644 tests/libqos/qgraph_extra.h create mode 100644 tests/libqos/raspi2-machine.c create mode 100644 tests/libqos/sdhci.c create mode 100644 tests/libqos/sdhci.h create mode 100644 tests/libqos/virt-machine.c create mode 100644 tests/libqos/virtio-9p.c create mode 100644 tests/libqos/virtio-9p.h create mode 100644 tests/libqos/virtio-balloon.c create mode 100644 tests/libqos/virtio-balloon.h create mode 100644 tests/libqos/virtio-blk.c create mode 100644 tests/libqos/virtio-blk.h create mode 100644 tests/libqos/virtio-net.c create mode 100644 tests/libqos/virtio-net.h create mode 100644 tests/libqos/virtio-rng.c create mode 100644 tests/libqos/virtio-rng.h create mode 100644 tests/libqos/virtio-scsi.c create mode 100644 tests/libqos/virtio-scsi.h create mode 100644 tests/libqos/virtio-serial.c create mode 100644 tests/libqos/virtio-serial.h create mode 100644 tests/libqos/x86_64_pc-machine.c create mode 100644 tests/qos-test.c create mode 100644 tests/test-qgraph.c create mode 100644 tests/virtio-ccw-test.c