| Message ID | 4EF73EF5.8050606@redhat.com |
|---|---|
| State | New |
| Headers | show |
Hi Dor, On 12/25/2011 09:19 AM, Dor Laor wrote: > On 12/19/2011 07:13 PM, Anthony Liguori wrote: > > Well, I'm still not convinced that a new standalone package should handle these > cases instead of kvm autotest. I'll be happy to integrate the tests to kvm > autotest anyway and the more the merrier but imho it's a duplicate. I'm sure kvm autotest could be taught to do exactly what qemu-test is doing. But why does kvm autotest have to do everything? I doubt there would be much code reuse. I think it's not a bad thing to have multiple test suites when there isn't considerable overlap. >> It has the following characteristics: >> >> 1) It builds a custom kernel and initramfs based on busybox. This is >> fairly important to ensure that we can run tests with no device >> pre-requisites. > > This can be done easily w/ autotest too. Okay, please demonstrate :-) The procedure to do this with qemu test is: $ git clone git://git.qemu.org/qemu-test.git $ cd qemu-test $ git submodule update --init $ make How would one do this with kvm autotest? >> 2) Tests are scripts that are launched in the initramfs >> >> 3) The test controls exactly how QEMU is launched which allows easy >> testing of various QEMU options > > afaik kvm autotest do the same. It's true that kvm autotest might look less > friendly but its may only be this way since it has lots of options. Actually, kvm-autotest has various layers of abstraction in how QEMU ends up being launched. As you mention below, those layers are there to allow for things like using libvirt. That's desirable when you're doing "virt testing", but not so desirably when you're trying to write specific unit tests against QEMU. >> 5) The tests execute very quickly, can be run stand alone, and do not >> require root privileges > > ditto for kvm auotest. It's possible to configure it w/o root too which is not a > huge issue. When I say, "run quickly", I mean, they execute very quickly. $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 tests/virtio-serial.sh real 0m4.385s user 0m1.460s sys 0m1.860s I've used kvm-autotest a lot, there is no test in kvm-autotest that is even close to completing in 4 seconds start to finish. I don't think kvm-autotest can even run without installing a guest first which puts puts a simple test more at the 30 minute mark. If we're talking about TCG testing, then we're in hours territory. If there's a way to run kvm-autotest as a non-privileged user, I've not figured it out yet. Of course, kvm-autotest can test a lot more things than qemu-test can since the tests are guest agnostic. That is the primary architectural difference. By writing the tests to one specific guest (busybox initrd), they can be extremely fast in execution, but the tests are less general purpose. That makes qemu-test very useful for development testing (such as long automated bisect runs) but less useful for acceptance testing. > Please compare your own virtio-serial test w/ the autotest version of it: > https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py > > > This single file tests functionality, limits, console, live migration and > performance. Of course one can add a very basic 'hello world' sanity test too > that will run quickly and will identify basic breakage fast. Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52. There is a lot of value in being able to write simple tests in 50 lines of code. In fact, the largest qemu-test at the moment is only ~70 lines of code. And besides finding regressions in my own code (which is the primary use atm), I've found and fixed a few real bugs in upstream QEMU. For a couple hundred lines of bash script, how can it be anything but a good thing :-) > Noways we abstract kvm autotest so libvirt will be optionally tested too w/ the > same tests. > > Again, I'm not advocating that kvm autotest is a solution for anything but for > plain guest-host communication, monitor commands, etc it's a really good tool. > > I agree that kvm autotest may be less friendly for developer users since it > carries allot of options for testing a huge matrix. Lucas and Cleber are working > these days to add a make kvmautotest target to qemu so one would be able to > quickly execute autotest and we can think of additional parameters like > sanity-set, migration-set, etc. We (the QEMU project) need to get more serious about testing. We need to (quickly) get to a point where we can mandate that features come with test cases. It is extremely hard to implement touch everything features with no good way to test all of the different things we support. kvm autotest cannot fill that role because it's too complicated and too many layers removed from QEMU. virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not about to mandate that someone writes 2.2k LOC in a python test framework in order to get 1.4k of code merged in QEMU. But 50 lines of bash seems like a more than reasonable requirement. Regards, Anthony Liguori >> >> 6) They are random by nature with the ability to fix the seed in order >> to be used in git-bisect. >> >> I think Gerd had been looking at doing something similar with a custom >> initrd. >> >> I've tried to consider other architectures and had hoped that we could >> commit the vmlinuz and initramfs into git so that it was easy to test >> other architectures without having a full build environment. >> Unfortunately, busybox doesn't link statically with glibc and I can't >> see an obvious way to commit binaries while respecting the GPL since we >> need to pull glibc into the initramfs. >> >> I know buildroot exists specifically to deal with this but in my >> experience, buildroot is very unreliable and extremely heavy weight >> since it rebuilds gcc multiple times in order to bootstrap a ulibc >> environment. >> >> Anyway, the code is available at: >> >> http://git.qemu.org/qemu-test.git >> >> See the README for instructions on how to use it. >> >> Regards, >> >> Anthony Liguori >> >
On 12/26/2011 05:12 PM, Anthony Liguori wrote: > Hi Dor, > Merry Christmas Anthony, > On 12/25/2011 09:19 AM, Dor Laor wrote: >> On 12/19/2011 07:13 PM, Anthony Liguori wrote: >> >> Well, I'm still not convinced that a new standalone package should >> handle these >> cases instead of kvm autotest. I'll be happy to integrate the tests to >> kvm >> autotest anyway and the more the merrier but imho it's a duplicate. > > I'm sure kvm autotest could be taught to do exactly what qemu-test is > doing. But why does kvm autotest have to do everything? I doubt there > would be much code reuse. > > I think it's not a bad thing to have multiple test suites when there > isn't considerable overlap. I agree but in this case, it loos to me that qemu-test is likely to do a subset of what kvm autotest do in a potentially less elegant way but due to the influence of a key maintainer may gain traction and dilute kvm autotest.. > >>> It has the following characteristics: >>> >>> 1) It builds a custom kernel and initramfs based on busybox. This is >>> fairly important to ensure that we can run tests with no device >>> pre-requisites. >> >> This can be done easily w/ autotest too. > > Okay, please demonstrate :-) The procedure to do this with qemu test is: > > $ git clone git://git.qemu.org/qemu-test.git > $ cd qemu-test > $ git submodule update --init > $ make > > How would one do this with kvm autotest? git clone git://github.com/autotest/autotest.git cd autotest/client/tests/kvm sudo ./get_started.py One command shorter and the above downloads F15 automatically and runs it. (from https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript) >>> 2) Tests are scripts that are launched in the initramfs >>> >>> 3) The test controls exactly how QEMU is launched which allows easy >>> testing of various QEMU options >> >> afaik kvm autotest do the same. It's true that kvm autotest might look >> less >> friendly but its may only be this way since it has lots of options. > > Actually, kvm-autotest has various layers of abstraction in how QEMU > ends up being launched. As you mention below, those layers are there to > allow for things like using libvirt. It goes beyond that, since it also related to the monitor interface as well. > That's desirable when you're doing "virt testing", but not so desirably > when you're trying to write specific unit tests against QEMU. True, one may not need it at all but it's nice that a test for migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ the same effort. > >>> 5) The tests execute very quickly, can be run stand alone, and do not >>> require root privileges >> >> ditto for kvm auotest. It's possible to configure it w/o root too >> which is not a >> huge issue. > > When I say, "run quickly", I mean, they execute very quickly. /me too > > $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 > tests/virtio-serial.sh > > real 0m4.385s > user 0m1.460s > sys 0m1.860s That's impressive but it's more of a function of the guest being used - if instead of running a full Fedora install, you'll chose your busybox image w/ -kernel/initrd you'll get a similar result. > > I've used kvm-autotest a lot, there is no test in kvm-autotest that is > even close to completing in 4 seconds start to finish. I don't think > kvm-autotest can even run without installing a guest first which puts > puts a simple test more at the 30 minute mark. If we're talking about > TCG testing, then we're in hours territory. Autotest today may be too verbose by default and will call echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way and Lucas can easily create a slim mode that does not even need to be root. > > If there's a way to run kvm-autotest as a non-privileged user, I've not > figured it out yet. Again, there is no magic in a test suite so if required we can wave it (maybe we need to do that regardless) > > Of course, kvm-autotest can test a lot more things than qemu-test can > since the tests are guest agnostic. That is the primary architectural > difference. By writing the tests to one specific guest (busybox initrd), > they can be extremely fast in execution, but the tests are less general > purpose. > > That makes qemu-test very useful for development testing (such as long > automated bisect runs) but less useful for acceptance testing. Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and the tests will be slimmer or each test will have such a mode. > >> Please compare your own virtio-serial test w/ the autotest version of it: >> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py >> >> >> >> This single file tests functionality, limits, console, live migration and >> performance. Of course one can add a very basic 'hello world' sanity >> test too >> that will run quickly and will identify basic breakage fast. > > Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52. I bet that virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be less than 52 lines. It depends what you like to test. as noted, the above test fully covers all aspects of virtio-serial. It does not have to be this way and I do agree it is nicer to split it to multiple sub files. > > There is a lot of value in being able to write simple tests in 50 lines > of code. In fact, the largest qemu-test at the moment is only ~70 lines > of code. And besides finding regressions in my own code (which is the > primary use atm), I've found and fixed a few real bugs in upstream QEMU. Now imagine what virtio_console.py buys you :) > > For a couple hundred lines of bash script, how can it be anything but a > good thing :-) > >> Noways we abstract kvm autotest so libvirt will be optionally tested >> too w/ the >> same tests. >> >> Again, I'm not advocating that kvm autotest is a solution for anything >> but for >> plain guest-host communication, monitor commands, etc it's a really >> good tool. >> >> I agree that kvm autotest may be less friendly for developer users >> since it >> carries allot of options for testing a huge matrix. Lucas and Cleber >> are working >> these days to add a make kvmautotest target to qemu so one would be >> able to >> quickly execute autotest and we can think of additional parameters like >> sanity-set, migration-set, etc. > > We (the QEMU project) need to get more serious about testing. We need to Like > (quickly) get to a point where we can mandate that features come with > test cases. It is extremely hard to implement touch everything features > with no good way to test all of the different things we support. > > kvm autotest cannot fill that role because it's too complicated and too > many layers removed from QEMU. > > virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not > about to mandate that someone writes 2.2k LOC in a python test framework > in order to get 1.4k of code merged in QEMU. > > But 50 lines of bash seems like a more than reasonable requirement. Hmm, what's the coverage of these 50 LOC? Maybe when your maintainer hat is on, it is indeed a huge task to keep all of qemu platforms/architectures to compile and run so that's enough but when I wear my Red Hat manager role, I wish that it will not only compile and boot but will test as much functionality as possible and will be easily re-used by R&D, QA, libvirt, while all supported guests are _potentially_ exercised, not just busybox. Since kvm autotest will be there anyway, I wish that developers will contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will justify it. I agree autotest is not perfect but it likes to be such. If you wish, you can challenge Lucas and Cleber w/ these type of requirements and we'll all improve as a result. Cheers, Dor > > Regards, > > Anthony Liguori > >>> >>> 6) They are random by nature with the ability to fix the seed in order >>> to be used in git-bisect. >>> >>> I think Gerd had been looking at doing something similar with a custom >>> initrd. >>> >>> I've tried to consider other architectures and had hoped that we could >>> commit the vmlinuz and initramfs into git so that it was easy to test >>> other architectures without having a full build environment. >>> Unfortunately, busybox doesn't link statically with glibc and I can't >>> see an obvious way to commit binaries while respecting the GPL since we >>> need to pull glibc into the initramfs. >>> >>> I know buildroot exists specifically to deal with this but in my >>> experience, buildroot is very unreliable and extremely heavy weight >>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>> environment. >>> >>> Anyway, the code is available at: >>> >>> http://git.qemu.org/qemu-test.git >>> >>> See the README for instructions on how to use it. >>> >>> Regards, >>> >>> Anthony Liguori >>> >> >
On 12/26/2011 05:00 PM, Dor Laor wrote: > On 12/26/2011 05:12 PM, Anthony Liguori wrote: >> Hi Dor, >> > > Merry Christmas Anthony, Happy Hanukkah! >> I think it's not a bad thing to have multiple test suites when there >> isn't considerable overlap. > > I agree but in this case, it loos to me that qemu-test is likely to do a subset > of what kvm autotest do in a potentially less elegant way but due to the > influence of a key maintainer may gain traction and dilute kvm autotest.. I'm worried that if we take an attitude of doing what we've always done, we'll get the same results we always have. kvm-autotest is a great test suite, but I'm not happy about the number of regressions that were present in 1.0 even after a month of release candidates. We need to write a lot more tests than we are right now that cover a much broader set of functionality. >> >>>> It has the following characteristics: >>>> >>>> 1) It builds a custom kernel and initramfs based on busybox. This is >>>> fairly important to ensure that we can run tests with no device >>>> pre-requisites. >>> >>> This can be done easily w/ autotest too. >> >> Okay, please demonstrate :-) The procedure to do this with qemu test is: >> >> $ git clone git://git.qemu.org/qemu-test.git >> $ cd qemu-test >> $ git submodule update --init >> $ make >> >> How would one do this with kvm autotest? > > git clone git://github.com/autotest/autotest.git > cd autotest/client/tests/kvm > sudo ./get_started.py > > One command shorter and the above downloads F15 automatically and runs it. > > (from https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript) I know how to install kvm-autotest :-) You claimed that autotest could easily "build a custom kernel and initramfs based on busybox." I've studied kvm-autotest extensively to determine if it could do what I want it to do and came to the conclusion that it couldn't without significant effort. The infrastructure assumes that you have a full OS available in the guest. The tests are written in Python and make a variety of assumptions. To my knowledge, it's not very practical to build a busybox environment with Python embedded in it. This means you need to install a distro and have a disk based install. Not requiring this was one of my primary requirements in creating qemu-test mainly since the resulting images are very large and installing a guest with TCG is extremely slow. OTOH, you can build a kernel and initrd with a cross compiler and booting a kernel, while not fast under TCG, is not all that slow. >> >>>> 5) The tests execute very quickly, can be run stand alone, and do not >>>> require root privileges >>> >>> ditto for kvm auotest. It's possible to configure it w/o root too >>> which is not a >>> huge issue. >> >> When I say, "run quickly", I mean, they execute very quickly. > > /me too > >> >> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 >> tests/virtio-serial.sh >> >> real 0m4.385s >> user 0m1.460s >> sys 0m1.860s > > That's impressive but it's more of a function of the guest being used - if > instead of running a full Fedora install, you'll chose your busybox image w/ > -kernel/initrd you'll get a similar result. I'm fairly certain that this is not easy to do with kvm-autotest. I'm happy to be proven wrong but I didn't just create qemu-test in a vacuum. >> I've used kvm-autotest a lot, there is no test in kvm-autotest that is >> even close to completing in 4 seconds start to finish. I don't think >> kvm-autotest can even run without installing a guest first which puts >> puts a simple test more at the 30 minute mark. If we're talking about >> TCG testing, then we're in hours territory. > > Autotest today may be too verbose by default and will call > echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way and Lucas > can easily create a slim mode that does not even need to be root. If it's easy, then by all means, please demonstrate. If qemu-test only serves to show what my requirements for kvm-autotest are and Lucas can implement it all very easily, then I would be happy to discontinue qemu-test immediately. But as I mentioned above, I don't think it's that simple. You basically can't have python-based guest tests without having a full guest distribution. That means in the very least, every kvm-autotest test will need to be rewritten. >> If there's a way to run kvm-autotest as a non-privileged user, I've not >> figured it out yet. > > Again, there is no magic in a test suite so if required we can wave it (maybe we > need to do that regardless) I cannot stress enough how important this is. The entire process from git fetch to running the test should not require root privileges. If you are targeting developers, you can't assume that the box is dedicated to testing as you would if you're doing QE testing. That means you need to co-exist with the existing environment and not muck around with network setting and stuff like that. FWIW, I gave this feedback during Lucas' talk at KVM Forum 2011 so this shouldn't be new news to anyone. >> Of course, kvm-autotest can test a lot more things than qemu-test can >> since the tests are guest agnostic. That is the primary architectural >> difference. By writing the tests to one specific guest (busybox initrd), >> they can be extremely fast in execution, but the tests are less general >> purpose. >> >> That makes qemu-test very useful for development testing (such as long >> automated bisect runs) but less useful for acceptance testing. > > Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and the tests > will be slimmer or each test will have such a mode. I've requested a simple config for autotest in the past and even offered to include it in qemu.git as a standard profile. The simple config that I use for regression testing takes about two hours to complete. I'd prefer to have something that took a few minutes to complete... >>> Please compare your own virtio-serial test w/ the autotest version of it: >>> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py >>> >>> >>> >>> >>> This single file tests functionality, limits, console, live migration and >>> performance. Of course one can add a very basic 'hello world' sanity >>> test too >>> that will run quickly and will identify basic breakage fast. >> >> Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52. > > I bet that virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will > be less than 52 lines. Okay, then please do demonstrate. I don't mean to be coy here but as I mentioned earlier, I've studied kvm-autotest and based on my understanding of it, I don't think it's possible. I'd be thrilled to be proven wrong though. > It depends what you like to test. as noted, the above > test fully covers all aspects of virtio-serial. It does not have to be this way > and I do agree it is nicer to split it to multiple sub files. Looking closely, I think the main difference is that the kvm-autotest test also tests guest semantics. It tests the behavior of module unload, SIGIO, poll, etc. While those tests are important for distro QE testing, they are not useful for QEMU testing as there's almost nothing that can change in QEMU that's going to affect that behavior. Those are really Linux kernel tests, not QEMU tests. >> There is a lot of value in being able to write simple tests in 50 lines >> of code. In fact, the largest qemu-test at the moment is only ~70 lines >> of code. And besides finding regressions in my own code (which is the >> primary use atm), I've found and fixed a few real bugs in upstream QEMU. > > Now imagine what virtio_console.py buys you :) Actually, the only thing that it tests that's a QEMU function is having multiple ports per PCI device. That's something I could add to the qemu-test test with a couple dozen lines of code I suspect. >> (quickly) get to a point where we can mandate that features come with >> test cases. It is extremely hard to implement touch everything features >> with no good way to test all of the different things we support. >> >> kvm autotest cannot fill that role because it's too complicated and too >> many layers removed from QEMU. >> >> virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not >> about to mandate that someone writes 2.2k LOC in a python test framework >> in order to get 1.4k of code merged in QEMU. >> >> But 50 lines of bash seems like a more than reasonable requirement. > > Hmm, what's the coverage of these 50 LOC? It tests device creation (including via alias), device fingerprinting (although that's a different test), and basic I/O functionality. > Maybe when your maintainer hat is on, it is indeed a huge task to keep all of > qemu platforms/architectures to compile and run so that's enough but when I wear > my Red Hat manager role, I wish that it will not only compile and boot but will > test as much functionality as possible and will be easily re-used by R&D, QA, > libvirt, while all supported guests are _potentially_ exercised, not just busybox. Right, there are two different use cases here: 1) light, QEMU specific testing, that can be used as a QEMU regression suite. Essentially, the problem to solve is that when doing changes like the memory API, just trying to figure out how to boot a cris guest to confirm that it didn't fall apart is surprisingly time consuming. This is the kind of test suite that a developer would run more or less after every local commit. 2) Thorough integration tests that test the whole stack including the kernel, libvirt, QEMU, and guest software. This is the sort of suite that a distribution would run as part of their QE procedures. I think these goals are different enough that they deserve to be separate projects. I don't think kvm-autotest would be improved by also taking on (1) as it would force lots of change for very little benefit to (2). > Since kvm autotest will be there anyway, I wish that developers will contribute > 50 LOC to autotest (and not 2.2kloc) and hope the ROI will justify it. > > I agree autotest is not perfect but it likes to be such. > If you wish, you can challenge Lucas and Cleber w/ these type of requirements > and we'll all improve as a result. Well consider qemu-test the challenge. It's an existence proof that we can have a very easy to use framework for testing that runs extremely fast and is very easy to write tests for. I don't think qemu-test would ever be appropriate for distro QE testing as kvm-autotest does an excellent job of this. I really think that this are two fundamentally different use-cases that require specialized tools. Of course, I'm happy to be proven wrong. Regards, Anthony Liguori > > Cheers, > Dor > >> >> Regards, >> >> Anthony Liguori >> >>>> >>>> 6) They are random by nature with the ability to fix the seed in order >>>> to be used in git-bisect. >>>> >>>> I think Gerd had been looking at doing something similar with a custom >>>> initrd. >>>> >>>> I've tried to consider other architectures and had hoped that we could >>>> commit the vmlinuz and initramfs into git so that it was easy to test >>>> other architectures without having a full build environment. >>>> Unfortunately, busybox doesn't link statically with glibc and I can't >>>> see an obvious way to commit binaries while respecting the GPL since we >>>> need to pull glibc into the initramfs. >>>> >>>> I know buildroot exists specifically to deal with this but in my >>>> experience, buildroot is very unreliable and extremely heavy weight >>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>>> environment. >>>> >>>> Anyway, the code is available at: >>>> >>>> http://git.qemu.org/qemu-test.git >>>> >>>> See the README for instructions on how to use it. >>>> >>>> Regards, >>>> >>>> Anthony Liguori >>>> >>> >> > >
On 12/27/2011 05:22 PM, Anthony Liguori wrote: > > The infrastructure assumes that you have a full OS available in the > guest. The tests are written in Python and make a variety of > assumptions. To my knowledge, it's not very practical to build a > busybox environment with Python embedded in it. You could move whatever infrastructure qemu-test uses to kvm-autotest, at which point kvm-autotest will know everything qemu-test knows. But there's zero reason to do that, autotest is designed to drive external tests and in fact most of the tests it supports are not in the autotest repository.
On 12/27/2011 09:58 AM, Avi Kivity wrote: > On 12/27/2011 05:22 PM, Anthony Liguori wrote: >> >> The infrastructure assumes that you have a full OS available in the >> guest. The tests are written in Python and make a variety of >> assumptions. To my knowledge, it's not very practical to build a >> busybox environment with Python embedded in it. > > You could move whatever infrastructure qemu-test uses to kvm-autotest, > at which point kvm-autotest will know everything qemu-test knows. But > there's zero reason to do that, autotest is designed to drive external > tests and in fact most of the tests it supports are not in the autotest > repository. Yes. I think having a qemu-test driver for kvm-autotest that knows enough to invoke the qemu-tests and integrate the results in autotest results reporting is the right thing to do. Regards, Anthony Liguori >
On 12/27/2011 02:40 PM, Anthony Liguori wrote: > On 12/27/2011 09:58 AM, Avi Kivity wrote: >> On 12/27/2011 05:22 PM, Anthony Liguori wrote: >>> >>> The infrastructure assumes that you have a full OS available in the >>> guest. The tests are written in Python and make a variety of >>> assumptions. To my knowledge, it's not very practical to build a >>> busybox environment with Python embedded in it. >> >> You could move whatever infrastructure qemu-test uses to kvm-autotest, >> at which point kvm-autotest will know everything qemu-test knows. But >> there's zero reason to do that, autotest is designed to drive external >> tests and in fact most of the tests it supports are not in the autotest >> repository. > > Yes. I think having a qemu-test driver for kvm-autotest that knows > enough to invoke the qemu-tests and integrate the results in autotest > results reporting is the right thing to do. I am happy with that too. There's a slight inaccuracy when you've mentioned that KVM autotest mandates guest OS installs to perform tests. It's possible to use pre-installed guest images, making the time of execution of a test job shorter. But anyway, your reasoning is sound. Considering that's not a lot of code overlap between the 2 infrastructures, as long as we can make them interact well, all is good. Plus, the way qemu-test is structured integrates well with the workflow most qemu developers are used to. In my point of view, it's a win-win situation, as I expect more developers to invest some time writing test automation. Ideally it would be nice to have more people contributing tests to both qemu-test *and* kvm autotest, no doubt, but this is something we (kvm autotest team) still have to figure how to accomplish.
On 12/26/2011 08:00 PM, Dor Laor wrote: > On 12/26/2011 05:12 PM, Anthony Liguori wrote: >> Hi Dor, >> > > Merry Christmas Anthony, > >> On 12/25/2011 09:19 AM, Dor Laor wrote: >>> On 12/19/2011 07:13 PM, Anthony Liguori wrote: >>> >>> Well, I'm still not convinced that a new standalone package should >>> handle these >>> cases instead of kvm autotest. I'll be happy to integrate the tests to >>> kvm >>> autotest anyway and the more the merrier but imho it's a duplicate. >> >> I'm sure kvm autotest could be taught to do exactly what qemu-test is >> doing. But why does kvm autotest have to do everything? I doubt there >> would be much code reuse. >> >> I think it's not a bad thing to have multiple test suites when there >> isn't considerable overlap. I think the main goal of qemu-tests (may be implicit) is to be quick and simple. That is indeed great, but if one thinks that's all we'll ever going to need, that thought is pretty naive. And it may be true that there's room for both test suites... or that, as busy developers, we're refusing to deal with the added complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I believe on the later. One example: kvm-autotest has a complex configuration file format with a steep learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have to be tweaked if somehow the kernel detects the first ethernet interface as em1 (such as recent Fedora systems do). Simple, but not scalable. Other example: kvm-autotest can easily implement (copy?) the functionality to build a busybox initramfs. We already have code to build from sources such as git, local source directories, remote tarballs, you named it. And the recipe to build a given package is also configurable/extensible so we could even leverage that. qemu-tests, on the other hand, would need lots of code to test the same range of guests that kvm-autotest tests. Simple, but limited. So, it looks like, if we add the missing features to both test suites, we'll end up with two implementations, using different languages/approaches, but with very little added value. > > I agree but in this case, it loos to me that qemu-test is likely to do > a subset of what kvm autotest do in a potentially less elegant way but > due to the influence of a key maintainer may gain traction and dilute > kvm autotest.. > >> >>>> It has the following characteristics: >>>> >>>> 1) It builds a custom kernel and initramfs based on busybox. This is >>>> fairly important to ensure that we can run tests with no device >>>> pre-requisites. >>> >>> This can be done easily w/ autotest too. The Python requirement inside the guest is true *if* we want to run regular autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and this accounts for very very little of kvm autotest usage. All other tests interact with the monitor directly and with the guest via ssh/telnet/serial. So, I see no reason for not using a more expressive language, having an API, etc, to do that's done in in_host(). In the guest side, what changes is that we send one command at a time, checking/collecting results, instead of pushing the complete tests files into the initramfs. >> >> Okay, please demonstrate :-) The procedure to do this with qemu test is: >> >> $ git clone git://git.qemu.org/qemu-test.git >> $ cd qemu-test >> $ git submodule update --init >> $ make >> >> How would one do this with kvm autotest? > > git clone git://github.com/autotest/autotest.git > cd autotest/client/tests/kvm > sudo ./get_started.py > > One command shorter and the above downloads F15 automatically and runs > it. > > (from > https://github.com/autotest/autotest/wiki/KVMAutotest-RunGetStartedScript) > >>>> 2) Tests are scripts that are launched in the initramfs >>>> >>>> 3) The test controls exactly how QEMU is launched which allows easy >>>> testing of various QEMU options >>> >>> afaik kvm autotest do the same. It's true that kvm autotest might look >>> less >>> friendly but its may only be this way since it has lots of options. >> >> Actually, kvm-autotest has various layers of abstraction in how QEMU >> ends up being launched. As you mention below, those layers are there to >> allow for things like using libvirt. Indeed the qemu command line parameters gets generated depending on many configuration parameters. It'd be *really* simple to add a configuration parameters that overwrites the qemu command with an static one. > > It goes beyond that, since it also related to the monitor interface as > well. > >> That's desirable when you're doing "virt testing", but not so desirably >> when you're trying to write specific unit tests against QEMU. > > True, one may not need it at all but it's nice that a test for > migration/stress/hotplug will be tested directly w/ qemu and libvirt > w/ the same effort. > >> >>>> 5) The tests execute very quickly, can be run stand alone, and do not >>>> require root privileges >>> >>> ditto for kvm auotest. It's possible to configure it w/o root too >>> which is not a >>> huge issue. >> >> When I say, "run quickly", I mean, they execute very quickly. > > /me too > >> >> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 >> tests/virtio-serial.sh >> >> real 0m4.385s >> user 0m1.460s >> sys 0m1.860s > > That's impressive but it's more of a function of the guest being used > - if instead of running a full Fedora install, you'll chose your > busybox image w/ -kernel/initrd you'll get a similar result. I also think so. Maybe kvm-autotest would take a little more time because of the different approach we take when communicating with the guest, but I bet it'd be irrelevant. > >> >> I've used kvm-autotest a lot, there is no test in kvm-autotest that is >> even close to completing in 4 seconds start to finish. I don't think >> kvm-autotest can even run without installing a guest first which puts >> puts a simple test more at the 30 minute mark. If we're talking about >> TCG testing, then we're in hours territory. > > Autotest today may be too verbose by default and will call > echo 3 > /proc/sys/vm/drop_caches but it does not have to be this way > and Lucas can easily create a slim mode that does not even need to be > root. The drop_caches thing has been configurable for ages. > >> >> If there's a way to run kvm-autotest as a non-privileged user, I've not >> figured it out yet. > > Again, there is no magic in a test suite so if required we can wave it > (maybe we need to do that regardless) Some of the things we do in kvm autotest (and are not currently done in qemu-tests) indeed need root privileges, but we could add granularity to that. > >> >> Of course, kvm-autotest can test a lot more things than qemu-test can >> since the tests are guest agnostic. That is the primary architectural >> difference. By writing the tests to one specific guest (busybox initrd), >> they can be extremely fast in execution, but the tests are less general >> purpose. >> >> That makes qemu-test very useful for development testing (such as long >> automated bisect runs) but less useful for acceptance testing. > > > Autotest too can define a family of 'sanity' or 'rapid-bisec-mode' and > the tests will be slimmer or each test will have such a mode. > >> >>> Please compare your own virtio-serial test w/ the autotest version >>> of it: >>> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py >>> >>> >>> >>> >>> This single file tests functionality, limits, console, live >>> migration and >>> performance. Of course one can add a very basic 'hello world' sanity >>> test too >>> that will run quickly and will identify basic breakage fast. >> >> Note that virtio_console.py is 2175 LOC whereas virtio-serial.sh is 52. > > I bet that > virtio_console_super_slim_and_equal_to_qemu_unittest_serial.py will be > less than 52 lines. It depends what you like to test. as noted, the > above test fully covers all aspects of virtio-serial. It does not have > to be this way and I do agree it is nicer to split it to multiple sub > files. > >> >> There is a lot of value in being able to write simple tests in 50 lines >> of code. In fact, the largest qemu-test at the moment is only ~70 lines >> of code. And besides finding regressions in my own code (which is the >> primary use atm), I've found and fixed a few real bugs in upstream QEMU. > > Now imagine what virtio_console.py buys you :) > >> >> For a couple hundred lines of bash script, how can it be anything but a >> good thing :-) >> >>> Noways we abstract kvm autotest so libvirt will be optionally tested >>> too w/ the >>> same tests. >>> >>> Again, I'm not advocating that kvm autotest is a solution for anything >>> but for >>> plain guest-host communication, monitor commands, etc it's a really >>> good tool. >>> >>> I agree that kvm autotest may be less friendly for developer users >>> since it >>> carries allot of options for testing a huge matrix. Lucas and Cleber >>> are working >>> these days to add a make kvmautotest target to qemu so one would be >>> able to >>> quickly execute autotest and we can think of additional parameters like >>> sanity-set, migration-set, etc. >> >> We (the QEMU project) need to get more serious about testing. We need to > > Like > >> (quickly) get to a point where we can mandate that features come with >> test cases. It is extremely hard to implement touch everything features >> with no good way to test all of the different things we support. >> >> kvm autotest cannot fill that role because it's too complicated and too >> many layers removed from QEMU. >> >> virtio-console/virtio-serial-bus in about 1400 LOC in QEMU. I'm not >> about to mandate that someone writes 2.2k LOC in a python test framework >> in order to get 1.4k of code merged in QEMU. >> >> But 50 lines of bash seems like a more than reasonable requirement. > > Hmm, what's the coverage of these 50 LOC? > Maybe when your maintainer hat is on, it is indeed a huge task to keep > all of qemu platforms/architectures to compile and run so that's > enough but when I wear my Red Hat manager role, I wish that it will > not only compile and boot but will test as much functionality as > possible and will be easily re-used by R&D, QA, libvirt, while all > supported guests are _potentially_ exercised, not just busybox. > > Since kvm autotest will be there anyway, I wish that developers will > contribute 50 LOC to autotest (and not 2.2kloc) and hope the ROI will > justify it. > > I agree autotest is not perfect but it likes to be such. > If you wish, you can challenge Lucas and Cleber w/ these type of > requirements and we'll all improve as a result. Yes, I believe it'd be a nice exercise for all of us. The only thing I ask is that we bear at least with some of the complexity that kvm-autotest inherently holds... the last requests we had was to get rid of all the complexity, while retaining all the other nice characteristics. Pretty hard, so I think we failed, or maybe only half-succeeded at it. > > Cheers, > Dor > >> >> Regards, >> >> Anthony Liguori >> >>>> >>>> 6) They are random by nature with the ability to fix the seed in order >>>> to be used in git-bisect. >>>> >>>> I think Gerd had been looking at doing something similar with a custom >>>> initrd. >>>> >>>> I've tried to consider other architectures and had hoped that we could >>>> commit the vmlinuz and initramfs into git so that it was easy to test >>>> other architectures without having a full build environment. >>>> Unfortunately, busybox doesn't link statically with glibc and I can't >>>> see an obvious way to commit binaries while respecting the GPL >>>> since we >>>> need to pull glibc into the initramfs. >>>> >>>> I know buildroot exists specifically to deal with this but in my >>>> experience, buildroot is very unreliable and extremely heavy weight >>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>>> environment. >>>> >>>> Anyway, the code is available at: >>>> >>>> http://git.qemu.org/qemu-test.git >>>> >>>> See the README for instructions on how to use it. >>>> >>>> Regards, >>>> >>>> Anthony Liguori >>>> >>> >> > >
On 12/27/2011 04:35 PM, Cleber Rosa wrote: > On 12/26/2011 08:00 PM, Dor Laor wrote: >> On 12/26/2011 05:12 PM, Anthony Liguori wrote: >>> Hi Dor, >>> >> >> Merry Christmas Anthony, >> >>> On 12/25/2011 09:19 AM, Dor Laor wrote: >>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote: >>>> >>>> Well, I'm still not convinced that a new standalone package should >>>> handle these >>>> cases instead of kvm autotest. I'll be happy to integrate the tests to >>>> kvm >>>> autotest anyway and the more the merrier but imho it's a duplicate. >>> >>> I'm sure kvm autotest could be taught to do exactly what qemu-test is >>> doing. But why does kvm autotest have to do everything? I doubt there >>> would be much code reuse. >>> >>> I think it's not a bad thing to have multiple test suites when there >>> isn't considerable overlap. > > I think the main goal of qemu-tests (may be implicit) is to be quick and simple. qemu-test doesn't have a main goal. My goal is to improve QEMU's quality. qemu-test is just a tool to help achieve that goal. > That is indeed great, but if one thinks that's all we'll ever going to need, > that thought is pretty naive. I don't know who "we" is, but I can tell you that qemu-test is exactly what *I* need. Consider that I spent a good portion of every single day testing QEMU with either my own or other people's patches, making that job easier and more automated is fairly important to me. I'm sharing it because I suspect that a lot of other developers have a similar need. > And it may be true that there's room for both test > suites... or that, as busy developers, we're refusing to deal with the added > complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I > believe on the later. > > One example: kvm-autotest has a complex configuration file format with a steep > learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have > to be tweaked if somehow the kernel detects the first ethernet interface as em1 > (such as recent Fedora systems do). Simple, but not scalable. I can tell by this comment that you don't actually understand how qemu-test works. Please take a look at it before jumping to conclusions about whether it should or shouldn't be part of kvm-autotest. Hint: qemu-test always uses the same kernel because it builds it as part of the test suite. The behavior of how a nic test will never change unless someone explicitly changes the kernel. >>>>> 1) It builds a custom kernel and initramfs based on busybox. This is >>>>> fairly important to ensure that we can run tests with no device >>>>> pre-requisites. >>>> >>>> This can be done easily w/ autotest too. > > The Python requirement inside the guest is true *if* we want to run regular > autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and > this accounts for very very little of kvm autotest usage. All other tests > interact with the monitor directly and with the guest via ssh/telnet/serial. qemu-test does not require any specific hardware to be used in the guest which lets it test a wider variety of scenarios in QEMU. So you cannot assume there is ssh/telnet/serial available. > > So, I see no reason for not using a more expressive language, I seriously doubt you can build a useful initramfs that contains python without doing something crazy like livecd-tools does.... >>> Actually, kvm-autotest has various layers of abstraction in how QEMU >>> ends up being launched. As you mention below, those layers are there to >>> allow for things like using libvirt. > > Indeed the qemu command line parameters gets generated depending on many > configuration parameters. It'd be *really* simple to add a configuration > parameters that overwrites the qemu command with an static one. But if you're a QEMU developer, you want to have as much control of the command line as possible. For instance, one of the tests in qemu-test makes sure to test invocations without -device as this triggers a different code path (there was a recent regression in this too). You can't just add arguments to reproduce this behavior. >> >> It goes beyond that, since it also related to the monitor interface as well. >> >>> That's desirable when you're doing "virt testing", but not so desirably >>> when you're trying to write specific unit tests against QEMU. >> >> True, one may not need it at all but it's nice that a test for >> migration/stress/hotplug will be tested directly w/ qemu and libvirt w/ the >> same effort. >> >>> >>>>> 5) The tests execute very quickly, can be run stand alone, and do not >>>>> require root privileges >>>> >>>> ditto for kvm auotest. It's possible to configure it w/o root too >>>> which is not a >>>> huge issue. >>> >>> When I say, "run quickly", I mean, they execute very quickly. >> >> /me too >> >>> >>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 >>> tests/virtio-serial.sh >>> >>> real 0m4.385s >>> user 0m1.460s >>> sys 0m1.860s >> >> That's impressive but it's more of a function of the guest being used - if >> instead of running a full Fedora install, you'll chose your busybox image w/ >> -kernel/initrd you'll get a similar result. > > I also think so. Maybe kvm-autotest would take a little more time because of the > different approach we take when communicating with the guest, but I bet it'd be > irrelevant. I don't see any reason why everything needs to live in kvm-autotest... but if you really feel that way, please provide patches that demonstrate how this would work. We could argue indefinitely about how things could work, it's much better to compare how things actually do work :-) >> I agree autotest is not perfect but it likes to be such. >> If you wish, you can challenge Lucas and Cleber w/ these type of requirements >> and we'll all improve as a result. > > Yes, I believe it'd be a nice exercise for all of us. > > The only thing I ask is that we bear at least with some of the complexity that > kvm-autotest inherently holds... I think there's something of a knee jerk reaction here. The existence of qemu-test does not take anything away from kvm-autotest. It's just another tool in our arsenal to achieve our joint goal of making QEMU (and KVM) higher quality. autotest is made to invoke third party tests so the two tools can co-exist in a complimentary way. Regards, Anthony Liguori > the last requests we had was to get rid of all > the complexity, while retaining all the other nice characteristics. Pretty hard, > so I think we failed, or maybe only half-succeeded at it. > >> >> Cheers, >> Dor >> >>> >>> Regards, >>> >>> Anthony Liguori >>> >>>>> >>>>> 6) They are random by nature with the ability to fix the seed in order >>>>> to be used in git-bisect. >>>>> >>>>> I think Gerd had been looking at doing something similar with a custom >>>>> initrd. >>>>> >>>>> I've tried to consider other architectures and had hoped that we could >>>>> commit the vmlinuz and initramfs into git so that it was easy to test >>>>> other architectures without having a full build environment. >>>>> Unfortunately, busybox doesn't link statically with glibc and I can't >>>>> see an obvious way to commit binaries while respecting the GPL since we >>>>> need to pull glibc into the initramfs. >>>>> >>>>> I know buildroot exists specifically to deal with this but in my >>>>> experience, buildroot is very unreliable and extremely heavy weight >>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>>>> environment. >>>>> >>>>> Anyway, the code is available at: >>>>> >>>>> http://git.qemu.org/qemu-test.git >>>>> >>>>> See the README for instructions on how to use it. >>>>> >>>>> Regards, >>>>> >>>>> Anthony Liguori >>>>> >>>> >>> >> >> > >
On 12/27/2011 11:37 PM, Anthony Liguori wrote: > On 12/27/2011 04:35 PM, Cleber Rosa wrote: >> On 12/26/2011 08:00 PM, Dor Laor wrote: >>> On 12/26/2011 05:12 PM, Anthony Liguori wrote: >>>> Hi Dor, >>>> >>> >>> Merry Christmas Anthony, >>> >>>> On 12/25/2011 09:19 AM, Dor Laor wrote: >>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote: >>>>> >>>>> Well, I'm still not convinced that a new standalone package should >>>>> handle these >>>>> cases instead of kvm autotest. I'll be happy to integrate the >>>>> tests to >>>>> kvm >>>>> autotest anyway and the more the merrier but imho it's a duplicate. >>>> >>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is >>>> doing. But why does kvm autotest have to do everything? I doubt there >>>> would be much code reuse. >>>> >>>> I think it's not a bad thing to have multiple test suites when there >>>> isn't considerable overlap. >> >> I think the main goal of qemu-tests (may be implicit) is to be quick >> and simple. > > qemu-test doesn't have a main goal. My goal is to improve QEMU's > quality. qemu-test is just a tool to help achieve that goal. Maybe I've used the wrong wording. Besides the different approach (simpler requirements, initramfs with busybox, etc), it looks to me that keeping it simple played an important role in your decision to write qemu-tests, which is indeed a sign of good design. But read on... > >> That is indeed great, but if one thinks that's all we'll ever going >> to need, >> that thought is pretty naive. > > I don't know who "we" is, but I can tell you that qemu-test is exactly > what *I* need. Consider that I spent a good portion of every single > day testing QEMU with either my own or other people's patches, making > that job easier and more automated is fairly important to me. "We" is everyone that contributes one way or another to qemu. So if you only concerned with your needs, you're definitely on the right track. If not, I think (trying to think as a community) it'd be beneficial to concentrate all efforts, unless it's not possible to do so. It's as simple as that. All my reasoning here revolves around this. > > I'm sharing it because I suspect that a lot of other developers have a > similar need. > >> And it may be true that there's room for both test >> suites... or that, as busy developers, we're refusing to deal with >> the added >> complexity (qemu alone accounts for a lot) and delaying to fix the >> fixable. I >> believe on the later. >> >> One example: kvm-autotest has a complex configuration file format >> with a steep >> learning curve, while a test such as qemu-tests/tests/simple-ping.sh >> would have >> to be tweaked if somehow the kernel detects the first ethernet >> interface as em1 >> (such as recent Fedora systems do). Simple, but not scalable. > > I can tell by this comment that you don't actually understand how > qemu-test works. Please take a look at it before jumping to > conclusions about whether it should or shouldn't be part of kvm-autotest. I can tell you did not grasp my point: kvm autotest is more complex, but more capable and flexible. And I did *not* come to a conclusion, I'm giving examples in an attempt to enrich the discussion. > > Hint: qemu-test always uses the same kernel because it builds it as > part of the test suite. The behavior of how a nic test will never > change unless someone explicitly changes the kernel. Hopefully you understand by now that I'm giving some reasons of why kvm-autotest does some things the way it does (usually more complex), and how qemu-tests, because of its approach, does not have to deal with that. > >>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is >>>>>> fairly important to ensure that we can run tests with no device >>>>>> pre-requisites. >>>>> >>>>> This can be done easily w/ autotest too. >> >> The Python requirement inside the guest is true *if* we want to run >> regular >> autotest tests inside the guest (see >> autotest/client/virt/tests/autotest.py) and >> this accounts for very very little of kvm autotest usage. All other >> tests >> interact with the monitor directly and with the guest via >> ssh/telnet/serial. > > qemu-test does not require any specific hardware to be used in the > guest which lets it test a wider variety of scenarios in QEMU. So you > cannot assume there is ssh/telnet/serial available. I was assuming that we could count on at least a serial port in the guest. If not, then current kvm-autotest can not absorb the same functionality of qemu-tests without re-writing it. A > >> >> So, I see no reason for not using a more expressive language, > > I seriously doubt you can build a useful initramfs that contains > python without doing something crazy like livecd-tools does.... > >>>> Actually, kvm-autotest has various layers of abstraction in how QEMU >>>> ends up being launched. As you mention below, those layers are >>>> there to >>>> allow for things like using libvirt. >> >> Indeed the qemu command line parameters gets generated depending on many >> configuration parameters. It'd be *really* simple to add a configuration >> parameters that overwrites the qemu command with an static one. > > But if you're a QEMU developer, you want to have as much control of > the command line as possible. For instance, one of the tests in > qemu-test makes sure to test invocations without -device as this > triggers a different code path (there was a recent regression in this > too). You can't just add arguments to reproduce this behavior. > >>> >>> It goes beyond that, since it also related to the monitor interface >>> as well. >>> >>>> That's desirable when you're doing "virt testing", but not so >>>> desirably >>>> when you're trying to write specific unit tests against QEMU. >>> >>> True, one may not need it at all but it's nice that a test for >>> migration/stress/hotplug will be tested directly w/ qemu and libvirt >>> w/ the >>> same effort. >>> >>>> >>>>>> 5) The tests execute very quickly, can be run stand alone, and do >>>>>> not >>>>>> require root privileges >>>>> >>>>> ditto for kvm auotest. It's possible to configure it w/o root too >>>>> which is not a >>>>> huge issue. >>>> >>>> When I say, "run quickly", I mean, they execute very quickly. >>> >>> /me too >>> >>>> >>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 >>>> tests/virtio-serial.sh >>>> >>>> real 0m4.385s >>>> user 0m1.460s >>>> sys 0m1.860s >>> >>> That's impressive but it's more of a function of the guest being >>> used - if >>> instead of running a full Fedora install, you'll chose your busybox >>> image w/ >>> -kernel/initrd you'll get a similar result. >> >> I also think so. Maybe kvm-autotest would take a little more time >> because of the >> different approach we take when communicating with the guest, but I >> bet it'd be >> irrelevant. > > I don't see any reason why everything needs to live in kvm-autotest... > but if you really feel that way, please provide patches that > demonstrate how this would work. We could argue indefinitely about > how things could work, it's much better to compare how things actually > do work :-) > >>> I agree autotest is not perfect but it likes to be such. >>> If you wish, you can challenge Lucas and Cleber w/ these type of >>> requirements >>> and we'll all improve as a result. >> >> Yes, I believe it'd be a nice exercise for all of us. >> >> The only thing I ask is that we bear at least with some of the >> complexity that >> kvm-autotest inherently holds... > > I think there's something of a knee jerk reaction here. The existence > of qemu-test does not take anything away from kvm-autotest. It's just > another tool in our arsenal to achieve our joint goal of making QEMU > (and KVM) higher quality. > > autotest is made to invoke third party tests so the two tools can > co-exist in a complimentary way. > > Regards, > > Anthony Liguori > >> the last requests we had was to get rid of all >> the complexity, while retaining all the other nice characteristics. >> Pretty hard, >> so I think we failed, or maybe only half-succeeded at it. >> >>> >>> Cheers, >>> Dor >>> >>>> >>>> Regards, >>>> >>>> Anthony Liguori >>>> >>>>>> >>>>>> 6) They are random by nature with the ability to fix the seed in >>>>>> order >>>>>> to be used in git-bisect. >>>>>> >>>>>> I think Gerd had been looking at doing something similar with a >>>>>> custom >>>>>> initrd. >>>>>> >>>>>> I've tried to consider other architectures and had hoped that we >>>>>> could >>>>>> commit the vmlinuz and initramfs into git so that it was easy to >>>>>> test >>>>>> other architectures without having a full build environment. >>>>>> Unfortunately, busybox doesn't link statically with glibc and I >>>>>> can't >>>>>> see an obvious way to commit binaries while respecting the GPL >>>>>> since we >>>>>> need to pull glibc into the initramfs. >>>>>> >>>>>> I know buildroot exists specifically to deal with this but in my >>>>>> experience, buildroot is very unreliable and extremely heavy weight >>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>>>>> environment. >>>>>> >>>>>> Anyway, the code is available at: >>>>>> >>>>>> http://git.qemu.org/qemu-test.git >>>>>> >>>>>> See the README for instructions on how to use it. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Anthony Liguori >>>>>> >>>>> >>>> >>> >>> >> >> >
On 12/27/2011 11:37 PM, Anthony Liguori wrote: > On 12/27/2011 04:35 PM, Cleber Rosa wrote: >> On 12/26/2011 08:00 PM, Dor Laor wrote: >>> On 12/26/2011 05:12 PM, Anthony Liguori wrote: >>>> Hi Dor, >>>> >>> >>> Merry Christmas Anthony, >>> >>>> On 12/25/2011 09:19 AM, Dor Laor wrote: >>>>> On 12/19/2011 07:13 PM, Anthony Liguori wrote: >>>>> >>>>> Well, I'm still not convinced that a new standalone package should >>>>> handle these >>>>> cases instead of kvm autotest. I'll be happy to integrate the >>>>> tests to >>>>> kvm >>>>> autotest anyway and the more the merrier but imho it's a duplicate. >>>> >>>> I'm sure kvm autotest could be taught to do exactly what qemu-test is >>>> doing. But why does kvm autotest have to do everything? I doubt there >>>> would be much code reuse. >>>> >>>> I think it's not a bad thing to have multiple test suites when there >>>> isn't considerable overlap. >> >> I think the main goal of qemu-tests (may be implicit) is to be quick >> and simple. > > qemu-test doesn't have a main goal. My goal is to improve QEMU's > quality. qemu-test is just a tool to help achieve that goal. Maybe I've used the wrong wording. I got the feeling that, besides testing qemu the way you need it, keeping qemu-test simple was really important. If not, I'd imagine you'd try to re-use something that already exists (yes, kvm autotest). > >> That is indeed great, but if one thinks that's all we'll ever going >> to need, >> that thought is pretty naive. > > I don't know who "we" is, but I can tell you that qemu-test is exactly > what *I* need. Consider that I spent a good portion of every single > day testing QEMU with either my own or other people's patches, making > that job easier and more automated is fairly important to me. "We" is everyone that somehow contributes to QEMU, that is, the QEMU community. If you're only concerned about what *you* need, then you're on the right track. If, besides that, you feel it'd be nice to *try to* concentrate our efforts, then we're all on the same track. > > I'm sharing it because I suspect that a lot of other developers have a > similar need. > >> And it may be true that there's room for both test >> suites... or that, as busy developers, we're refusing to deal with >> the added >> complexity (qemu alone accounts for a lot) and delaying to fix the >> fixable. I >> believe on the later. >> >> One example: kvm-autotest has a complex configuration file format >> with a steep >> learning curve, while a test such as qemu-tests/tests/simple-ping.sh >> would have >> to be tweaked if somehow the kernel detects the first ethernet >> interface as em1 >> (such as recent Fedora systems do). Simple, but not scalable. > > I can tell by this comment that you don't actually understand how > qemu-test works. Please take a look at it before jumping to > conclusions about whether it should or shouldn't be part of kvm-autotest. > > Hint: qemu-test always uses the same kernel because it builds it as > part of the test suite. The behavior of how a nic test will never > change unless someone explicitly changes the kernel. I can tell you did not get my point: I'm giving some reasons of why current kvm autotest is somehow complex, and how qemu-tests gets away and keeps it simple. BTW, I did not jumped at any conclusion. I'm just trying to enrich the discussion, which may end up proving that there's no other way to have what qemu-tests does. > >>>>>> 1) It builds a custom kernel and initramfs based on busybox. This is >>>>>> fairly important to ensure that we can run tests with no device >>>>>> pre-requisites. >>>>> >>>>> This can be done easily w/ autotest too. >> >> The Python requirement inside the guest is true *if* we want to run >> regular >> autotest tests inside the guest (see >> autotest/client/virt/tests/autotest.py) and >> this accounts for very very little of kvm autotest usage. All other >> tests >> interact with the monitor directly and with the guest via >> ssh/telnet/serial. > > qemu-test does not require any specific hardware to be used in the > guest which lets it test a wider variety of scenarios in QEMU. So you > cannot assume there is ssh/telnet/serial available. I really thought we could rely on, at least, a serial connection. If not, then indeed the current kvm autotest approach is not compatible with that test environment. That is not to say that kvm autotest couldn't incorporate the qemu-tests approach/functionality. BTW, I just don't like the idea of having lots of functionalities/tests implemented on two test suites for a single piece of software, unless proven that there's no way around it. To me, this is the whole point of this discussion. > >> >> So, I see no reason for not using a more expressive language, > > I seriously doubt you can build a useful initramfs that contains > python without doing something crazy like livecd-tools does.... You're right. Again, I was thinking we could rely at least on a serial connection. Can we not? > >>>> Actually, kvm-autotest has various layers of abstraction in how QEMU >>>> ends up being launched. As you mention below, those layers are >>>> there to >>>> allow for things like using libvirt. >> >> Indeed the qemu command line parameters gets generated depending on many >> configuration parameters. It'd be *really* simple to add a configuration >> parameters that overwrites the qemu command with an static one. > > But if you're a QEMU developer, you want to have as much control of > the command line as possible. For instance, one of the tests in > qemu-test makes sure to test invocations without -device as this > triggers a different code path (there was a recent regression in this > too). You can't just add arguments to reproduce this behavior. > >>> >>> It goes beyond that, since it also related to the monitor interface >>> as well. >>> >>>> That's desirable when you're doing "virt testing", but not so >>>> desirably >>>> when you're trying to write specific unit tests against QEMU. >>> >>> True, one may not need it at all but it's nice that a test for >>> migration/stress/hotplug will be tested directly w/ qemu and libvirt >>> w/ the >>> same effort. >>> >>>> >>>>>> 5) The tests execute very quickly, can be run stand alone, and do >>>>>> not >>>>>> require root privileges >>>>> >>>>> ditto for kvm auotest. It's possible to configure it w/o root too >>>>> which is not a >>>>> huge issue. >>>> >>>> When I say, "run quickly", I mean, they execute very quickly. >>> >>> /me too >>> >>>> >>>> $ time ./qemu-test ~/build/qemu/x86_64-softmmu/qemu-system-x86_64 >>>> tests/virtio-serial.sh >>>> >>>> real 0m4.385s >>>> user 0m1.460s >>>> sys 0m1.860s >>> >>> That's impressive but it's more of a function of the guest being >>> used - if >>> instead of running a full Fedora install, you'll chose your busybox >>> image w/ >>> -kernel/initrd you'll get a similar result. >> >> I also think so. Maybe kvm-autotest would take a little more time >> because of the >> different approach we take when communicating with the guest, but I >> bet it'd be >> irrelevant. > > I don't see any reason why everything needs to live in kvm-autotest... > but if you really feel that way, please provide patches that > demonstrate how this would work. If it's technically viable, I think that having it as part of kvm autotest, shows that the project is more cohesive, attracts more contributions, and makes better use of our efforts. > We could argue indefinitely about how things could work, it's much > better to compare how things actually do work :-) Sure, that's also what I suggested when I mentioned it'd be a "nice exercise for all of us". > >>> I agree autotest is not perfect but it likes to be such. >>> If you wish, you can challenge Lucas and Cleber w/ these type of >>> requirements >>> and we'll all improve as a result. >> >> Yes, I believe it'd be a nice exercise for all of us. >> >> The only thing I ask is that we bear at least with some of the >> complexity that >> kvm-autotest inherently holds... > > I think there's something of a knee jerk reaction here. The existence > of qemu-test does not take anything away from kvm-autotest. It's just > another tool in our arsenal to achieve our joint goal of making QEMU > (and KVM) higher quality. You're right, It does not take anything away from kvm autotest today. But suppose we can prove that kvm autotest can indeed absorve all of qemu-tests functionalities, it'd be itself a reason for doing so. It'd avoid finding ourselves with two evolved test tools that do some of the same things, but are separate implementations. > > autotest is made to invoke third party tests so the two tools can > co-exist in a complimentary way. That's a no brainier and everyone so far agrees on that. The delicate issue is whether qemu-tests functionality could live in kvm autotest in a nice way. > > Regards, > > Anthony Liguori > >> the last requests we had was to get rid of all >> the complexity, while retaining all the other nice characteristics. >> Pretty hard, >> so I think we failed, or maybe only half-succeeded at it. >> >>> >>> Cheers, >>> Dor >>> >>>> >>>> Regards, >>>> >>>> Anthony Liguori >>>> >>>>>> >>>>>> 6) They are random by nature with the ability to fix the seed in >>>>>> order >>>>>> to be used in git-bisect. >>>>>> >>>>>> I think Gerd had been looking at doing something similar with a >>>>>> custom >>>>>> initrd. >>>>>> >>>>>> I've tried to consider other architectures and had hoped that we >>>>>> could >>>>>> commit the vmlinuz and initramfs into git so that it was easy to >>>>>> test >>>>>> other architectures without having a full build environment. >>>>>> Unfortunately, busybox doesn't link statically with glibc and I >>>>>> can't >>>>>> see an obvious way to commit binaries while respecting the GPL >>>>>> since we >>>>>> need to pull glibc into the initramfs. >>>>>> >>>>>> I know buildroot exists specifically to deal with this but in my >>>>>> experience, buildroot is very unreliable and extremely heavy weight >>>>>> since it rebuilds gcc multiple times in order to bootstrap a ulibc >>>>>> environment. >>>>>> >>>>>> Anyway, the code is available at: >>>>>> >>>>>> http://git.qemu.org/qemu-test.git >>>>>> >>>>>> See the README for instructions on how to use it. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Anthony Liguori >>>>>> >>>>> >>>> >>> >>> >> >> > >
On 12/27/2011 11:01 PM, Cleber Rosa wrote: > On 12/27/2011 11:37 PM, Anthony Liguori wrote: >>> I think the main goal of qemu-tests (may be implicit) is to be quick and simple. >> >> qemu-test doesn't have a main goal. My goal is to improve QEMU's quality. >> qemu-test is just a tool to help achieve that goal. > > Maybe I've used the wrong wording. I got the feeling that, besides testing qemu > the way you need it, keeping qemu-test simple was really important. Simple is always important. In the case of qemu-test, there are some important trade offs made to keep things simple and fast. It doesn't try to work with arbitrary guests which means the tests need to handle only one environment. The guest is pre-made to have exactly what is needed for the tests so there is no setup required. The guest is as small as possible such that the test can run as quickly as possible. >>> That is indeed great, but if one thinks that's all we'll ever going to need, >>> that thought is pretty naive. >> >> I don't know who "we" is, but I can tell you that qemu-test is exactly what >> *I* need. Consider that I spent a good portion of every single day testing >> QEMU with either my own or other people's patches, making that job easier and >> more automated is fairly important to me. > > "We" is everyone that somehow contributes to QEMU, that is, the QEMU community. > If you're only concerned about what *you* need, then you're on the right track. > If, besides that, you feel it'd be nice to *try to* concentrate our efforts, > then we're all on the same track. There is no need to have a single tool that meets every possible need. In fact, the Unix tradition is to have separate single purposed tools. Having two test tools it not a bad thing provided that the overlap isn't significant. We shouldn't be discussing whether it's possible to merge the two tools, but rather what the technical benefits of doing so would be. Since at this point, there is almost no overlap between the two, I don't see any actual technical benefit to merging them. I see benefit to autotest executing qemu-test, of course. >> I'm sharing it because I suspect that a lot of other developers have a similar >> need. >> >>> And it may be true that there's room for both test >>> suites... or that, as busy developers, we're refusing to deal with the added >>> complexity (qemu alone accounts for a lot) and delaying to fix the fixable. I >>> believe on the later. >>> >>> One example: kvm-autotest has a complex configuration file format with a steep >>> learning curve, while a test such as qemu-tests/tests/simple-ping.sh would have >>> to be tweaked if somehow the kernel detects the first ethernet interface as em1 >>> (such as recent Fedora systems do). Simple, but not scalable. >> >> I can tell by this comment that you don't actually understand how qemu-test >> works. Please take a look at it before jumping to conclusions about whether it >> should or shouldn't be part of kvm-autotest. >> >> Hint: qemu-test always uses the same kernel because it builds it as part of >> the test suite. The behavior of how a nic test will never change unless >> someone explicitly changes the kernel. > > I can tell you did not get my point: I'm giving some reasons of why current kvm > autotest is somehow complex, and how qemu-tests gets away and keeps it simple. You're claiming that "we're refusing to deal with the added complexity (qemu alone accounts for a lot) and delaying to fix the fixable". There is no way that qemu-test would ever need to deal with how Fedora configures udev to name ethernet devices without becoming something totally different than it is. So there's no "delaying to fix the fixable" here. qemu-test makes a simplifying assumption. By restricting the guest to a fixed environment (initramfs w/busybox), things inherently become much, much simpler. Of course, this is not an adequate assumption to make if it were our only test tool but fortunately, we have an existing test suite that does a very good job at testing a wide variety of guests :-) >>> >>> The Python requirement inside the guest is true *if* we want to run regular >>> autotest tests inside the guest (see autotest/client/virt/tests/autotest.py) and >>> this accounts for very very little of kvm autotest usage. All other tests >>> interact with the monitor directly and with the guest via ssh/telnet/serial. >> >> qemu-test does not require any specific hardware to be used in the guest which >> lets it test a wider variety of scenarios in QEMU. So you cannot assume there >> is ssh/telnet/serial available. > > I really thought we could rely on, at least, a serial connection. If not, then > indeed the current kvm autotest approach is not compatible with that test > environment. That is not to say that kvm autotest couldn't incorporate the > qemu-tests approach/functionality. With the scope that qemu-test has, it cannot assume any hardware is present because it wants to test every piece of hardware. > BTW, I just don't like the idea of having lots of functionalities/tests > implemented on two test suites for a single piece of software, unless proven > that there's no way around it. To me, this is the whole point of this discussion. I actually disagree on principle. Two tools shouldn't be combined unless there's a proven reason that they benefit from being together. You get more flexibility at the end of the day having strong, independent components that can be combined together. >>> So, I see no reason for not using a more expressive language, >> >> I seriously doubt you can build a useful initramfs that contains python >> without doing something crazy like livecd-tools does.... > > You're right. Again, I was thinking we could rely at least on a serial > connection. Can we not? No. How would you test creating a guest with no serial devices? The nice thing about delivering a test via the initramfs is that you're only injecting software into the guest (usually via some non guest visible mechanism). >> I don't see any reason why everything needs to live in kvm-autotest... but if >> you really feel that way, please provide patches that demonstrate how this >> would work. > > If it's technically viable, I think that having it as part of kvm autotest, > shows that the project is more cohesive, attracts more contributions, and makes > better use of our efforts. Just putting the code in kvm-autotest.git in a directory doesn't make sense to me. Beyond the lack of a technical reason to do so, logistically, it makes it harder for me to ask people to submit test cases with a patch series if I can't actually apply the test case when I'm applying the patches to qemu.git. If qemu-test didn't use large submodules (like linux.git), I would have made qemu-test part of qemu.git. As far as I'm concerned, qemu-test.git is just an extension to qemu.git. >> I think there's something of a knee jerk reaction here. The existence of >> qemu-test does not take anything away from kvm-autotest. It's just another >> tool in our arsenal to achieve our joint goal of making QEMU (and KVM) higher >> quality. > > You're right, It does not take anything away from kvm autotest today. But > suppose we can prove that kvm autotest can indeed absorve all of qemu-tests > functionalities, it'd be itself a reason for doing so. That's not obvious to me. > It'd avoid finding > ourselves with two evolved test tools that do some of the same things, but are > separate implementations. It's just too easy to argue about abstract things like this. This discussion about testing has come up many times and we've talked to death how much we need to do better testing and get to test driven development. Yet, not much has materialized. I'm not interested in talking about things in abstract anymore. That's why I announced qemu-test. If folks think it should behave differently, then patches are welcome. If you think that kvm-autotest can do everything that qemu-test does, just make the changes to kvm-autotest to make it behave the same way and show me. If it makes sense, I'll happily embrace it. Regards, Anthony Liguori
On 12/28/2011 04:27 PM, Anthony Liguori wrote: >> Maybe I've used the wrong wording. I got the feeling that, besides >> testing qemu >> the way you need it, keeping qemu-test simple was really important. > > > Simple is always important. In the case of qemu-test, there are some > important trade offs made to keep things simple and fast. It doesn't > try to work with arbitrary guests which means the tests need to handle > only one environment. The guest is pre-made to have exactly what is > needed for the tests so there is no setup required. The guest is as > small as possible such that the test can run as quickly as possible. In fact using linux as a guest negates that. First of all, which linux version? if it's fixed, you'll eventually miss functionality and need to migrate. If it keeps changing, so does your test, and it will keep breaking. Using Linux as a guest allows you to do system tests (ping was an example) but doesn't allow you to do unit tests (test regression where if this bit in that register was set, but if them bits in thar registers were clear, then the packet would be encoded with ROT-26 before being sent out). I think qemu-test needs to use its own drivers which allow full control over what you do with the hardware. Otherwise, a regression that only shows up in non-Linux guests will not be testable with qemu-test. > There is no need to have a single tool that meets every possible > need. In fact, the Unix tradition is to have separate single purposed > tools. Those single purpose tools, if successful, tend to grow more purposes (cf. qemu), and if unsuccessful, tend to lose purpose. > Having two test tools it not a bad thing provided that the overlap > isn't significant. This is important, if the boundary isn't clear, then it will grow more fuzzy in time. I suggest the following: - qemu-test: qemu unit tests - kvm-unit-tests: kvm unit tests - kvm-autotest: unit test drivers + system tests > We shouldn't be discussing whether it's possible to merge the two > tools, but rather what the technical benefits of doing so would be. > > Since at this point, there is almost no overlap between the two, I > don't see any actual technical benefit to merging them. I see benefit > to autotest executing qemu-test, of course. I'd say that running a ping test is a weak version of kvm-autotest's system tests. Running a synthetic test that pokes values into memory and mmio and sees a packet coming out is a unit test (the latter can in fact be executed without a guest at all, just a table driving calls to the memory and irq APIs). > Just putting the code in kvm-autotest.git in a directory doesn't make > sense to me. Beyond the lack of a technical reason to do so, > logistically, it makes it harder for me to ask people to submit test > cases with a patch series if I can't actually apply the test case when > I'm applying the patches to qemu.git. > > If qemu-test didn't use large submodules (like linux.git), I would > have made qemu-test part of qemu.git. As far as I'm concerned, > qemu-test.git is just an extension to qemu.git. Why not just put it in qemu.git?
On 12/28/2011 05:01 PM, Avi Kivity wrote: > I'd say that running a ping test is a weak version of kvm-autotest's > system tests. Running a synthetic test that pokes values into memory > and mmio and sees a packet coming out is a unit test (the latter can in > fact be executed without a guest at all, just a table driving calls to > the memory and irq APIs). > Consider 98d23704138e0 7b4252e83f6f7d f7e80adf3cc4 c16ada980f43 4abf12f4ea8 (found by looking for 'fix' in the commit log and filtering out the commits that don't support my case) how can you reject such patches on the grounds that they're not accompanied by unit tests? only by making it easy to add tests for them. I think it would be hard/impossible to test them with linux-as-a-guest, since they fix edge cases that linux doesn't invoke. But by having our own driver (often just using qmp to poke at memory), we can easily generate the sequence that triggers the error. We'd probably need a library to support setting up a pci device's BARs, but that's easy with qmp/python integration. You can even poke a small executable into memory and execute it directly, if you really need guest cpu interaction.
On 12/28/2011 09:01 AM, Avi Kivity wrote: > On 12/28/2011 04:27 PM, Anthony Liguori wrote: >>> Maybe I've used the wrong wording. I got the feeling that, besides >>> testing qemu >>> the way you need it, keeping qemu-test simple was really important. >> >> >> Simple is always important. In the case of qemu-test, there are some >> important trade offs made to keep things simple and fast. It doesn't >> try to work with arbitrary guests which means the tests need to handle >> only one environment. The guest is pre-made to have exactly what is >> needed for the tests so there is no setup required. The guest is as >> small as possible such that the test can run as quickly as possible. > > In fact using linux as a guest negates that. First of all, which linux > version? if it's fixed, you'll eventually miss functionality and need to > migrate. If it keeps changing, so does your test, and it will keep > breaking. The kernel is a git submodule so it's a very specific version. Yes, we may need to bump the version down the road and obviously, if we have to change any tests in the process, we can. > > Using Linux as a guest allows you to do system tests (ping was an > example) but doesn't allow you to do unit tests (test regression where > if this bit in that register was set, but if them bits in thar registers > were clear, then the packet would be encoded with ROT-26 before being > sent out). > > I think qemu-test needs to use its own drivers which allow full control > over what you do with the hardware. Otherwise, a regression that only > shows up in non-Linux guests will not be testable with qemu-test. I think you're advocating for qtest. This is another important part of my testing strategy. I haven't received a lot of input on that RFC... http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com But there's certain things that I still consider to be unit testing (like basic networking tests) that I don't want to have to write with qtest. I'm not up for writing a TCP/IP stack in Python... >> There is no need to have a single tool that meets every possible >> need. In fact, the Unix tradition is to have separate single purposed >> tools. > > Those single purpose tools, if successful, tend to grow more purposes > (cf. qemu), and if unsuccessful, tend to lose purpose. > >> Having two test tools it not a bad thing provided that the overlap >> isn't significant. > > This is important, if the boundary isn't clear, then it will grow more > fuzzy in time. > > I suggest the following: > > - qemu-test: qemu unit tests > - kvm-unit-tests: kvm unit tests > - kvm-autotest: unit test drivers + system tests I would counter with: - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest) - qtest: low level, single device functional tests - kvm-unit-tests: low level, instruction-set level functional tests - qemu-test: higher level functional/coverage tests (multiple device interaction) - kvm-autotest: unit/functional test drivers + acceptance testing Note that the line I'm drawing is acceptance vs. functional testing, not unit vs. integration testing. Technically, our unit tests are things like test-visitor. Everything else is an integration test. But the separation between kvm-autotest is acceptance testing vs. functional testing. Acceptance testing is, "does Windows boot", "can I create three virtio-serial devices". Obviously, part of acceptance testing is, "does this set of functional tests pass". >> We shouldn't be discussing whether it's possible to merge the two >> tools, but rather what the technical benefits of doing so would be. >> >> Since at this point, there is almost no overlap between the two, I >> don't see any actual technical benefit to merging them. I see benefit >> to autotest executing qemu-test, of course. > > I'd say that running a ping test is a weak version of kvm-autotest's > system tests. Consider the Linux kernel to be our library of functionality to write our unit tests. I don't want to write a TCP/IP stack. We aren't just grabbing a random distro kernel. We're building one from scratch configured in a specific way. In theory, we could even host the kernel source on git.qemu.org and fork it to add more interesting things in the kernel (although I'd prefer not to do this, obviously). > Running a synthetic test that pokes values into memory > and mmio and sees a packet coming out is a unit test (the latter can in > fact be executed without a guest at all, just a table driving calls to > the memory and irq APIs). > > >> Just putting the code in kvm-autotest.git in a directory doesn't make >> sense to me. Beyond the lack of a technical reason to do so, >> logistically, it makes it harder for me to ask people to submit test >> cases with a patch series if I can't actually apply the test case when >> I'm applying the patches to qemu.git. >> >> If qemu-test didn't use large submodules (like linux.git), I would >> have made qemu-test part of qemu.git. As far as I'm concerned, >> qemu-test.git is just an extension to qemu.git. > > Why not just put it in qemu.git? Actually, having thought a bit about it, I think we can move qemu-test into qemu.git and just build the initramfs in a separate repository. That will make it easier to host binary packages of the later on qemu.org. Regards, Anthony Liguori >
On 12/28/2011 09:28 AM, Avi Kivity wrote: > On 12/28/2011 05:01 PM, Avi Kivity wrote: >> I'd say that running a ping test is a weak version of kvm-autotest's >> system tests. Running a synthetic test that pokes values into memory >> and mmio and sees a packet coming out is a unit test (the latter can in >> fact be executed without a guest at all, just a table driving calls to >> the memory and irq APIs). >> > > Consider > 98d23704138e0 > 7b4252e83f6f7d > f7e80adf3cc4 > c16ada980f43 > 4abf12f4ea8 > > (found by looking for 'fix' in the commit log and filtering out the > commits that don't support my case) > > how can you reject such patches on the grounds that they're not > accompanied by unit tests? That's why I've also proposed qtest. But having written quite a few qtest unit tests by now, you hit the limits of this type of testing pretty quickly. > only by making it easy to add tests for > them. I think it would be hard/impossible to test them with > linux-as-a-guest, since they fix edge cases that linux doesn't invoke. > But by having our own driver (often just using qmp to poke at memory), > we can easily generate the sequence that triggers the error. > > We'd probably need a library to support setting up a pci device's BARs, > but that's easy with qmp/python integration. You can even poke a small > executable into memory and execute it directly, if you really need guest > cpu interaction. Please review the qtest series. I think it offers a pretty good approach to writing this style of test. But as I mentioned, you hit the limits pretty quickly. Regards, Anthony Liguori >
On 12/28/2011 06:42 PM, Anthony Liguori wrote: >> In fact using linux as a guest negates that. First of all, which linux >> version? if it's fixed, you'll eventually miss functionality and need to >> migrate. If it keeps changing, so does your test, and it will keep >> breaking. > > > The kernel is a git submodule so it's a very specific version. Yes, > we may need to bump the version down the road and obviously, if we > have to change any tests in the process, we can. Having a full linux source as part of the build process detracts somewhat from the advantages here. > >> >> Using Linux as a guest allows you to do system tests (ping was an >> example) but doesn't allow you to do unit tests (test regression where >> if this bit in that register was set, but if them bits in thar registers >> were clear, then the packet would be encoded with ROT-26 before being >> sent out). >> >> I think qemu-test needs to use its own drivers which allow full control >> over what you do with the hardware. Otherwise, a regression that only >> shows up in non-Linux guests will not be testable with qemu-test. > > I think you're advocating for qtest. This is another important part > of my testing strategy. I haven't received a lot of input on that RFC... > > http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com > > But there's certain things that I still consider to be unit testing > (like basic networking tests) that I don't want to have to write with > qtest. I'm not up for writing a TCP/IP stack in Python... A ping test is not a unit test. The ping test is already covered by kvm-autotest; just set up a config that runs just that; after the initial run you'll have a guest installed so it'll be quick. If we have a DSL guest it'll even be very quick. To test the various NIC emulations, you don't need a full TCP stack, just like you didn't need to write an NTP implementation for qtest's rtc test. Instead you just poke values until it sends out a packet. If you want to test virtio-net with both direct and indirect buffers, you can only do that with a unit test, you can't do it using a full linux guest since it has its own ideas of when to use indirects and when to avoid them (for example, it may choose to always avoid them). >> I suggest the following: >> >> - qemu-test: qemu unit tests >> - kvm-unit-tests: kvm unit tests >> - kvm-autotest: unit test drivers + system tests > > > I would counter with: > > - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest) > - qtest: low level, single device functional tests > - kvm-unit-tests: low level, instruction-set level functional tests Not really. kvm-unit-tests tests things specific to kvm. > - qemu-test: higher level functional/coverage tests (multiple device > interaction) > - kvm-autotest: unit/functional test drivers + acceptance testing > > Note that the line I'm drawing is acceptance vs. functional testing, > not unit vs. integration testing. Technically, our unit tests are > things like test-visitor. Everything else is an integration test. > > But the separation between kvm-autotest is acceptance testing vs. > functional testing. > > Acceptance testing is, "does Windows boot", "can I create three > virtio-serial devices". > > Obviously, part of acceptance testing is, "does this set of functional > tests pass". Seems like a very blurry line. Especially as the functional test is weaker than either qtest and kvm-autotest. I now have to agree with the others that it's duplicate functionality. Does it really matter whether you're creating an image by compiling Linux and assembling and initramfs, or by downloading Fedora.iso and installing it? It's testing exactly the same thing, guest boot and functionality. Would you add live migration testing to qemu-test? If yes, you're duplicating some more. If not, you're not doing functional or coverage tests for that functionality. > >>> We shouldn't be discussing whether it's possible to merge the two >>> tools, but rather what the technical benefits of doing so would be. >>> >>> Since at this point, there is almost no overlap between the two, I >>> don't see any actual technical benefit to merging them. I see benefit >>> to autotest executing qemu-test, of course. >> >> I'd say that running a ping test is a weak version of kvm-autotest's >> system tests. > > Consider the Linux kernel to be our library of functionality to write > our unit tests. I was about to say the same thing, but with a negative implication. Using Linux restricts your tests to what Linux does with the devices. > I don't want to write a TCP/IP stack. We aren't just grabbing a > random distro kernel. We're building one from scratch configured in a > specific way. How does that help? > In theory, we could even host the kernel source on git.qemu.org and > fork it to add more interesting things in the kernel (although I'd > prefer not to do this, obviously). That way lies madness, though it could be argued that it lies all around us.
On 12/28/2011 06:44 PM, Anthony Liguori wrote: > On 12/28/2011 09:28 AM, Avi Kivity wrote: >> On 12/28/2011 05:01 PM, Avi Kivity wrote: >>> I'd say that running a ping test is a weak version of kvm-autotest's >>> system tests. Running a synthetic test that pokes values into memory >>> and mmio and sees a packet coming out is a unit test (the latter can in >>> fact be executed without a guest at all, just a table driving calls to >>> the memory and irq APIs). >>> >> >> Consider >> 98d23704138e0 >> 7b4252e83f6f7d >> f7e80adf3cc4 >> c16ada980f43 >> 4abf12f4ea8 >> >> (found by looking for 'fix' in the commit log and filtering out the >> commits that don't support my case) >> >> how can you reject such patches on the grounds that they're not >> accompanied by unit tests? > > That's why I've also proposed qtest. But having written quite a few > qtest unit tests by now, you hit the limits of this type of testing > pretty quickly. Can you describe those limits? > >> only by making it easy to add tests for >> them. I think it would be hard/impossible to test them with >> linux-as-a-guest, since they fix edge cases that linux doesn't invoke. >> But by having our own driver (often just using qmp to poke at memory), >> we can easily generate the sequence that triggers the error. >> >> We'd probably need a library to support setting up a pci device's BARs, >> but that's easy with qmp/python integration. You can even poke a small >> executable into memory and execute it directly, if you really need guest >> cpu interaction. > > Please review the qtest series. I think it offers a pretty good > approach to writing this style of test. But as I mentioned, you hit > the limits pretty quickly. I think it's great, it looks like exactly what I wanted, except it's been delivered on time. I'd really like to see it integrated quickly with some flesh around it, then replying -ENOTEST to all patches. This will improve qemu's quality a lot more than guest boot/ping tests, which we do regularly with kvm-autotest anyway. Think of how new instruction emulations are always accompanied by new kvm-unit-tests tests, I often don't even have to ask for them.
On 12/28/2011 07:21 PM, Avi Kivity wrote: >> I think you're advocating for qtest. This is another important part >> > of my testing strategy. I haven't received a lot of input on that RFC... >> > >> > http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com >> > >> > But there's certain things that I still consider to be unit testing >> > (like basic networking tests) that I don't want to have to write with >> > qtest. I'm not up for writing a TCP/IP stack in Python... > A ping test is not a unit test. > > The ping test is already covered by kvm-autotest; just set up a config > that runs just that; after the initial run you'll have a guest installed > so it'll be quick. If we have a DSL guest it'll even be very quick. > > To test the various NIC emulations, you don't need a full TCP stack, > just like you didn't need to write an NTP implementation for qtest's rtc > test. Instead you just poke values until it sends out a packet. If you > want to test virtio-net with both direct and indirect buffers, you can > only do that with a unit test, you can't do it using a full linux guest > since it has its own ideas of when to use indirects and when to avoid > them (for example, it may choose to always avoid them). > >>> >> I suggest the following: >>> >> >>> >> - qemu-test: qemu unit tests >>> >> - kvm-unit-tests: kvm unit tests >>> >> - kvm-autotest: unit test drivers + system tests >> > >> > >> > I would counter with: >> > >> > - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest) +1 for qtester >> > - qtest: low level, single device functional tests +1 for it >> > - kvm-unit-tests: low level, instruction-set level functional tests > Not really. kvm-unit-tests tests things specific to kvm. > >> > - qemu-test: higher level functional/coverage tests (multiple device >> > interaction) >> > - kvm-autotest: unit/functional test drivers + acceptance testing >> > >> > Note that the line I'm drawing is acceptance vs. functional testing, >> > not unit vs. integration testing. Technically, our unit tests are >> > things like test-visitor. Everything else is an integration test. >> > >> > But the separation between kvm-autotest is acceptance testing vs. >> > functional testing. >> > >> > Acceptance testing is, "does Windows boot", "can I create three >> > virtio-serial devices". >> > >> > Obviously, part of acceptance testing is, "does this set of functional >> > tests pass". > Seems like a very blurry line. Especially as the functional test is > weaker than either qtest and kvm-autotest. I now have to agree with the > others that it's duplicate functionality. Does it really matter whether > you're creating an image by compiling Linux and assembling and > initramfs, or by downloading Fedora.iso and installing it? It's testing > exactly the same thing, guest boot and functionality. > > Would you add live migration testing to qemu-test? If yes, you're > duplicating some more. If not, you're not doing functional or coverage > tests for that functionality. From the recent threads it looks to me that the 2 advantages of qemu-test over kvm-autotest are: 1. python is not used within the guest 2. qemu-test is smaller and simpler Except for the above 2, kvm autotest is superior. My main motivation to merge qemu-test and kvm autotest is that I fear that qemu-test will be the only requirement for committing stuff for qemu and developers will be excused from their 'duty' by writing a 50 LOC shell script and assume they done w/ testing. In addition to that, the 50 LOC will be too basic and only provide value for basic functionality tests and kvm autotest folks will need to rewrite all of the matching content and beyond. Those above 2 advantages can be solve: 1. python is not used within the guest - One option is just to use the same shell code in kvm autotest w/o no python code and use the same kernel/initramfs. - Another way is to use a plain linux distro w/ python and boot it into single user test mode and potentially use a S4 guest snapshot or external snapshot to shorten its boot time. You'll get faster boot time and friendlier code. 2. qemu-test is smaller and simpler kvm autotest will have a fast, out of the box mode especially designed to answer your requirements. It's mainly a matter of preparing a fastTestMode.py that will setup all of the above in a similar way that today's https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py Lastly, each test can have a fast|basic|sanity overloaded function that will be called by this. >> > >>>> >>> We shouldn't be discussing whether it's possible to merge the two >>>> >>> tools, but rather what the technical benefits of doing so would be. >>>> >>> >>>> >>> Since at this point, there is almost no overlap between the two, I >>>> >>> don't see any actual technical benefit to merging them. I see benefit >>>> >>> to autotest executing qemu-test, of course. >>> >> >>> >> I'd say that running a ping test is a weak version of kvm-autotest's >>> >> system tests. >> > >> > Consider the Linux kernel to be our library of functionality to write >> > our unit tests. > I was about to say the same thing, but with a negative implication. > Using Linux restricts your tests to what Linux does with the devices. > >> > I don't want to write a TCP/IP stack. We aren't just grabbing a >> > random distro kernel. We're building one from scratch configured in a >> > specific way. > How does that help? > >> > In theory, we could even host the kernel source on git.qemu.org and >> > fork it to add more interesting things in the kernel (although I'd >> > prefer not to do this, obviously). > That way lies madness, though it could be argued that it lies all around us.
On 12/28/2011 11:26 AM, Avi Kivity wrote: > On 12/28/2011 06:44 PM, Anthony Liguori wrote: >> On 12/28/2011 09:28 AM, Avi Kivity wrote: >>> On 12/28/2011 05:01 PM, Avi Kivity wrote: >>>> I'd say that running a ping test is a weak version of kvm-autotest's >>>> system tests. Running a synthetic test that pokes values into memory >>>> and mmio and sees a packet coming out is a unit test (the latter can in >>>> fact be executed without a guest at all, just a table driving calls to >>>> the memory and irq APIs). >>>> >>> >>> Consider >>> 98d23704138e0 >>> 7b4252e83f6f7d >>> f7e80adf3cc4 >>> c16ada980f43 >>> 4abf12f4ea8 >>> >>> (found by looking for 'fix' in the commit log and filtering out the >>> commits that don't support my case) >>> >>> how can you reject such patches on the grounds that they're not >>> accompanied by unit tests? >> >> That's why I've also proposed qtest. But having written quite a few >> qtest unit tests by now, you hit the limits of this type of testing >> pretty quickly. > > Can you describe those limits? I started writing a finger print test. While it's easy to do PCI enumeration, it gets pretty nasty if you want to actually access BARs (to read virtio registers) and forget about trying to get SMBIOS information as that involves mucking around with ACPI. OTOH, it was very simple to write with qemu-test: http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD And I ended up finding some long standing bugs in the process: http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com It's fairly nasty as it would only show up if doing migration from a new QEMU with a new guest to a old QEMU. I think it's a good example of the type of test that I'm targeting at qemu-test. It's really not interesting to run it across multiple distro types. But doing it with qtest would be prohibitively hard. >>> only by making it easy to add tests for >>> them. I think it would be hard/impossible to test them with >>> linux-as-a-guest, since they fix edge cases that linux doesn't invoke. >>> But by having our own driver (often just using qmp to poke at memory), >>> we can easily generate the sequence that triggers the error. >>> >>> We'd probably need a library to support setting up a pci device's BARs, >>> but that's easy with qmp/python integration. You can even poke a small >>> executable into memory and execute it directly, if you really need guest >>> cpu interaction. >> >> Please review the qtest series. I think it offers a pretty good >> approach to writing this style of test. But as I mentioned, you hit >> the limits pretty quickly. > > I think it's great, it looks like exactly what I wanted, except it's > been delivered on time. Can you take a look at how interrupts are handled? From what I gather, the only real limitation of this approach is that we won't be able to simulate MSI vectors but I don't think that's a huge problem. I looked at integrating interrupt handling at CPUs itself and that turned out to be fairly invasive. > I'd really like to see it integrated quickly > with some flesh around it, then replying -ENOTEST to all patches. This > will improve qemu's quality a lot more than guest boot/ping tests, which > we do regularly with kvm-autotest anyway. > > Think of how new instruction emulations are always accompanied by new > kvm-unit-tests tests, I often don't even have to ask for them. Yes, this is exactly where I'm heading with all of this. Regards, Anthony Liguori
On 12/28/2011 11:21 AM, Avi Kivity wrote: > On 12/28/2011 06:42 PM, Anthony Liguori wrote: >>> In fact using linux as a guest negates that. First of all, which linux >>> version? if it's fixed, you'll eventually miss functionality and need to >>> migrate. If it keeps changing, so does your test, and it will keep >>> breaking. >> >> >> The kernel is a git submodule so it's a very specific version. Yes, >> we may need to bump the version down the road and obviously, if we >> have to change any tests in the process, we can. > > Having a full linux source as part of the build process detracts > somewhat from the advantages here. Except that binaries can be made available. The initramfs is currently 512k while the kernel is about 5MB. OTOH, a Fedora install runs in the multiple GB. Having kernels/initramfs for the dozen target types that QEMU supports is reasonable. Having Fedora installs for all of them probably isn't for most people. >> But there's certain things that I still consider to be unit testing >> (like basic networking tests) that I don't want to have to write with >> qtest. I'm not up for writing a TCP/IP stack in Python... > > A ping test is not a unit test. > > The ping test is already covered by kvm-autotest; just set up a config > that runs just that; after the initial run you'll have a guest installed > so it'll be quick. If we have a DSL guest it'll even be very quick. Achieving this with kvm-autotest is not easy. I doubt that DSL has an automated install mechanism and the step file based guest installation is fincky at best. When we had the kvm-autotest day, none of the folks that tried to set it up who had never set it up before had something working by the end of the day. And that was with lmr available on IRC to answer questions. There is a huge setup cost with kvm-autotest. > To test the various NIC emulations, you don't need a full TCP stack, > just like you didn't need to write an NTP implementation for qtest's rtc > test. Instead you just poke values until it sends out a packet. If you > want to test virtio-net with both direct and indirect buffers, you can > only do that with a unit test, you can't do it using a full linux guest > since it has its own ideas of when to use indirects and when to avoid > them (for example, it may choose to always avoid them). I am not advocating that we don't write proper unit tests... >> Acceptance testing is, "does Windows boot", "can I create three >> virtio-serial devices". >> >> Obviously, part of acceptance testing is, "does this set of functional >> tests pass". > > Seems like a very blurry line. Especially as the functional test is > weaker than either qtest and kvm-autotest. I now have to agree with the > others that it's duplicate functionality. Does it really matter whether > you're creating an image by compiling Linux and assembling and > initramfs, or by downloading Fedora.iso and installing it? Yes. One of them is built entirely from source and the other is doing something meant to be done on bare metal. How do you test ARM guests with kvm-autotest? There is no such thing as an ISO installer for ARM. Are we going to download random images from the internet? Even if there is an ISO, installing a guest with TCG will take many hours. My local qemu-test repo can now fully bootstrap an initramfs for non-x86 targets (which involves building gcc and uClibc). Start to finish it takes about 30 minutes regardless of the target (since the build runs natively). > Would you add live migration testing to qemu-test? If yes, you're > duplicating some more. If not, you're not doing functional or coverage > tests for that functionality. I would add live migration schema testing, absolutely. I wouldn't add an acceptance test though. An acceptance test with live migration would be creating a guest, running it, and then seeing it live migration worked. >> I don't want to write a TCP/IP stack. We aren't just grabbing a >> random distro kernel. We're building one from scratch configured in a >> specific way. > > How does that help? Not sure I understand the question. Regards, Anthony Liguori >> In theory, we could even host the kernel source on git.qemu.org and >> fork it to add more interesting things in the kernel (although I'd >> prefer not to do this, obviously). > > That way lies madness, though it could be argued that it lies all around us. > >
On 12/29/2011 06:12 PM, Anthony Liguori wrote: >>> That's why I've also proposed qtest. But having written quite a few >>> qtest unit tests by now, you hit the limits of this type of testing >>> pretty quickly. >> >> Can you describe those limits? > > > I started writing a finger print test. While it's easy to do PCI > enumeration, it gets pretty nasty if you want to actually access BARs > (to read virtio registers) Why is that? it should be pretty easy to poke values into the BAR registers. Something else we can do is access the BARs directly. We have APIs to poke values into mmio, add an api to poke a value into a device's BAR. Now that each BAR is represented by exactly one memory region, known to the pci core even if it is not mapped, it should be easy. > and forget about trying to get SMBIOS information as that involves > mucking around with ACPI. SMBIOS is built by seabios, yes? So fingerprinting it should be done from seabios-test.git. Let's divide it into two problems: - seabios pokes qemu to get information. We should fingerprint this information to make sure different qemu version can interoperate with different seabios versions (needed since we migrate the bios along with the rest of the guest). I'm guessing most of this info is from fwcfg? We can easily fingerprint it like any other device. - we need to make sure seabios generates the same tables when using -M. Here, we're not testing a device, rather we're testing guest code, so it makes sense to use a guest for this. However, I don't think it's even necessary. From a quick read of the manual, SMBIOS is just a set of static tables in memory which are picked up using a signature. So all we need to do is boot an empty guest, read its memory, and look for the tables. > > OTOH, it was very simple to write with qemu-test: > > http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD > Yes; but using Linux limits you to what it exposes (of course Linux exposes quite a lot, so that's mostly a non issue; but we'll have counterexamples). > > And I ended up finding some long standing bugs in the process: > > http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com > > > It's fairly nasty as it would only show up if doing migration from a > new QEMU with a new guest to a old QEMU. Yes, good catch. > > I think it's a good example of the type of test that I'm targeting at > qemu-test. It's really not interesting to run it across multiple > distro types. But doing it with qtest would be prohibitively hard. I don't see why. A python library to read pci config should be a couple of dozens of lines long. Then you iterate over all functions and print selected registers. >>> Please review the qtest series. I think it offers a pretty good >>> approach to writing this style of test. But as I mentioned, you hit >>> the limits pretty quickly. >> >> I think it's great, it looks like exactly what I wanted, except it's >> been delivered on time. > > > Can you take a look at how interrupts are handled? From what I > gather, the only real limitation of this approach is that we won't be > able to simulate MSI vectors but I don't think that's a huge problem. > > I looked at integrating interrupt handling at CPUs itself and that > turned out to be fairly invasive. You mention in the changelog replacing the APIC. Why can't you do that? > >> I'd really like to see it integrated quickly >> with some flesh around it, then replying -ENOTEST to all patches. This >> will improve qemu's quality a lot more than guest boot/ping tests, which >> we do regularly with kvm-autotest anyway. >> >> Think of how new instruction emulations are always accompanied by new >> kvm-unit-tests tests, I often don't even have to ask for them. > > Yes, this is exactly where I'm heading with all of this. It looks great. One thing I'd change is to use the qmp protocol (perhaps not the monitor, just the schema/codegen). Does something prohibit this?
On 12/29/2011 08:38 AM, Dor Laor wrote: > On 12/28/2011 07:21 PM, Avi Kivity wrote: >> Would you add live migration testing to qemu-test? If yes, you're >> duplicating some more. If not, you're not doing functional or coverage >> tests for that functionality. > > From the recent threads it looks to me that the 2 advantages of qemu-test over > kvm-autotest are: > > 1. python is not used within the guest > 2. qemu-test is smaller and simpler > > Except for the above 2, kvm autotest is superior. Sorry, but that's non-sense. qemu-test builds a custom guest, entirely from source. This makes it possible to efficiently test non-native architectures. The tests are written for this minimal environment (which is not straight forward to do). This tests will never run on Windows as they are 100% tied to their environment. autotest (let's be clear, there is no such thing as kvm autotest anymore) is a framework for executing third party tests. It's got fancy magic to execute in client mode vs. server mode. There is a suite of tests that are integrated in autotest that exercise various types of virtualization functionality. This suite of tests use a special config format to determine what they do. Included in this, is the ability to create a guest from an ISO either using step files or via a guest-specific mechanism (kickstart, etc.). The tests are written to be mostly guest agnostic and are therefore written in Python in a cross platform way. There is essentially no overlap between the two and calling kvm autotest superior is like calling the Sun superior to the Moon. > My main motivation to merge > qemu-test and kvm autotest is that I fear that qemu-test will be the only > requirement for committing stuff for qemu and developers will be excused from > their 'duty' by writing a 50 LOC shell script and assume they done w/ testing. There is no requirement to write autotest tests to get things merged into QEMU. That's is how things are today. And I don't think there will ever a requirement to write anything that isn't merged directly into qemu.git. I'm not going to hold up merging a feature until another project merges something into their tree. So unless we're going to merge KVM autotest into qemu.git, I don't think there's every going to be a requirement to write a KVM autotest test in order to get something merged into qemu.git. But since autotest is a framework for running third-party tests, it seems to make sense for qemu.git to have a test framework that autotest can execute. And since what we call KVM autotest actually tests a heck of a lot more than just QEMU, it makes sense for KVM autotest to continue to focus on full stack testing where QEMU is but one of the many components that it tests. > In addition to that, the 50 LOC will be too basic and only provide value for > basic functionality tests and kvm autotest folks will need to rewrite all of the > matching content and beyond. > > Those above 2 advantages can be solve: > > 1. python is not used within the guest > - One option is just to use the same shell code in kvm autotest w/o > no python code and use the same kernel/initramfs. Yes, you can make a directory in kvm autotest that just imports qemu-test, but what's the point of doing that? Isn't it better for this to live in qemu.git? > - Another way is to use a plain linux distro w/ python and boot it > into single user test mode and potentially use a S4 guest snapshot > or external snapshot to shorten its boot time. You cannot easily create a "plain linux distro" for an ARM target. If you don't believe me, add ARM guest support to KVM autotest and see how well it works out :-) > You'll get faster boot time and friendlier code. Adding S4 resume for "simple tests" seems to be a bit odd to me... > 2. qemu-test is smaller and simpler > kvm autotest will have a fast, out of the box mode especially > designed to answer your requirements. > It's mainly a matter of preparing a fastTestMode.py that will setup > all of the above in a similar way that today's > https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py I hope you see from the above that this isn't just about having something new that has fewer features and is therefore simpler for the time being. The approach is fundamentally different. Regards, Anthony Liguori
On 12/29/2011 06:26 PM, Anthony Liguori wrote: > On 12/28/2011 11:21 AM, Avi Kivity wrote: >> On 12/28/2011 06:42 PM, Anthony Liguori wrote: >>>> In fact using linux as a guest negates that. First of all, which >>>> linux >>>> version? if it's fixed, you'll eventually miss functionality and >>>> need to >>>> migrate. If it keeps changing, so does your test, and it will keep >>>> breaking. >>> >>> >>> The kernel is a git submodule so it's a very specific version. Yes, >>> we may need to bump the version down the road and obviously, if we >>> have to change any tests in the process, we can. >> >> Having a full linux source as part of the build process detracts >> somewhat from the advantages here. > > Except that binaries can be made available. The initramfs is > currently 512k while the kernel is about 5MB. > > OTOH, a Fedora install runs in the multiple GB. Having > kernels/initramfs for the dozen target types that QEMU supports is > reasonable. Having Fedora installs for all of them probably isn't for > most people. Then use DSL for the ping test. > >>> But there's certain things that I still consider to be unit testing >>> (like basic networking tests) that I don't want to have to write with >>> qtest. I'm not up for writing a TCP/IP stack in Python... >> >> A ping test is not a unit test. >> >> The ping test is already covered by kvm-autotest; just set up a config >> that runs just that; after the initial run you'll have a guest installed >> so it'll be quick. If we have a DSL guest it'll even be very quick. > > Achieving this with kvm-autotest is not easy. I doubt that DSL has an > automated install mechanism and the step file based guest installation > is fincky at best. > > When we had the kvm-autotest day, none of the folks that tried to set > it up who had never set it up before had something working by the end > of the day. And that was with lmr available on IRC to answer questions. > > There is a huge setup cost with kvm-autotest. That needs to be fixed, not workarounded. I previously advised Lucas not to spend too much time on this, since the audience is limited and it would be hard to amortize the effort. Now I see that this was wrong. > >> To test the various NIC emulations, you don't need a full TCP stack, >> just like you didn't need to write an NTP implementation for qtest's rtc >> test. Instead you just poke values until it sends out a packet. If you >> want to test virtio-net with both direct and indirect buffers, you can >> only do that with a unit test, you can't do it using a full linux guest >> since it has its own ideas of when to use indirects and when to avoid >> them (for example, it may choose to always avoid them). > > I am not advocating that we don't write proper unit tests... Right, and qtest is dangerously close to making writing tests fun. But qemu-test is a wierd cross between qtest and kvm-autotest. > >>> Acceptance testing is, "does Windows boot", "can I create three >>> virtio-serial devices". >>> >>> Obviously, part of acceptance testing is, "does this set of functional >>> tests pass". >> >> Seems like a very blurry line. Especially as the functional test is >> weaker than either qtest and kvm-autotest. I now have to agree with the >> others that it's duplicate functionality. Does it really matter whether >> you're creating an image by compiling Linux and assembling and >> initramfs, or by downloading Fedora.iso and installing it? > > Yes. One of them is built entirely from source and the other is doing > something meant to be done on bare metal. > > How do you test ARM guests with kvm-autotest? There is no such thing > as an ISO installer for ARM. Are we going to download random images > from the internet? Do exactly what you plan to do with qemu-test. > > Even if there is an ISO, installing a guest with TCG will take many > hours. My local qemu-test repo can now fully bootstrap an initramfs > for non-x86 targets (which involves building gcc and uClibc). Start > to finish it takes about 30 minutes regardless of the target (since > the build runs natively). You can skip the install test and only do the boot test. Seed your disk with images from qemu.org or by running the install. kvm-autotest does this. > >> Would you add live migration testing to qemu-test? If yes, you're >> duplicating some more. If not, you're not doing functional or coverage >> tests for that functionality. > > I would add live migration schema testing, absolutely. I wouldn't add > an acceptance test though. > > An acceptance test with live migration would be creating a guest, > running it, and then seeing it live migration worked. Then it's duplicate functionality. It's already been done, but the setup procedure is hard, so let's do it again. > >>> I don't want to write a TCP/IP stack. We aren't just grabbing a >>> random distro kernel. We're building one from scratch configured in a >>> specific way. >> >> How does that help? > > Not sure I understand the question. > In what way is your specifically configured kernel's TCP stack better than the random distro's kernel's?
On 12/29/2011 10:36 AM, Avi Kivity wrote: > On 12/29/2011 06:12 PM, Anthony Liguori wrote: >>>> That's why I've also proposed qtest. But having written quite a few >>>> qtest unit tests by now, you hit the limits of this type of testing >>>> pretty quickly. >>> >>> Can you describe those limits? >> >> >> I started writing a finger print test. While it's easy to do PCI >> enumeration, it gets pretty nasty if you want to actually access BARs >> (to read virtio registers) > > Why is that? it should be pretty easy to poke values into the BAR registers. You have to map them which means you need to figure out what the memory layout looks like. Since there's no BIOS, you need to poke fw_cfg to figure all of this out and then program the BARs appropriately. It's definitely non-trivial. > Something else we can do is access the BARs directly. We have APIs to > poke values into mmio, add an api to poke a value into a device's BAR. > Now that each BAR is represented by exactly one memory region, known to > the pci core even if it is not mapped, it should be easy. > >> and forget about trying to get SMBIOS information as that involves >> mucking around with ACPI. > > SMBIOS is built by seabios, yes? I think QEMU actually builds the blob of information and passes the full blob to SeaBIOS. You could read and parse the blob out of fw_cfg I guess but that's still pretty sucky. It's a lot nicer to just poke sysfs. > So fingerprinting it should be done > from seabios-test.git. Let's divide it into two problems: That seems awkward especially since fingerprinting Just Works with qemu-test already (and it was only 50 lines of shell code!). > - seabios pokes qemu to get information. We should fingerprint this > information to make sure different qemu version can interoperate with > different seabios versions (needed since we migrate the bios along with > the rest of the guest). I'm guessing most of this info is from fwcfg? > We can easily fingerprint it like any other device. > - we need to make sure seabios generates the same tables when using -M. > Here, we're not testing a device, rather we're testing guest code, so it > makes sense to use a guest for this. > > However, I don't think it's even necessary. From a quick read of the > manual, SMBIOS is just a set of static tables in memory which are picked > up using a signature. So all we need to do is boot an empty guest, read > its memory, and look for the tables. Doesn't it sound a whole nicer use linux.git to parse this information for us in a friendly, easy to consume fashion? >> OTOH, it was very simple to write with qemu-test: >> >> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD >> > > Yes; but using Linux limits you to what it exposes (of course Linux > exposes quite a lot, so that's mostly a non issue; but we'll have > counterexamples). Maybe. And for those counter examples, we can drill down to qtest. But just because we may need to do fancy things, it doesn't mean we shouldn't take the easy approach 95% of the time. >> And I ended up finding some long standing bugs in the process: >> >> http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com >> >> >> It's fairly nasty as it would only show up if doing migration from a >> new QEMU with a new guest to a old QEMU. > > Yes, good catch. > >> >> I think it's a good example of the type of test that I'm targeting at >> qemu-test. It's really not interesting to run it across multiple >> distro types. But doing it with qtest would be prohibitively hard. > > I don't see why. A python library to read pci config should be a couple > of dozens of lines long. Then you iterate over all functions and print > selected registers. PCI enumeration is easy. It's mapping the bars and then poking other sources of information that's more challenging. Everything is doable by writing a libOS but I'd rather just use Linux than invent a libOS just for testing. >>>> Please review the qtest series. I think it offers a pretty good >>>> approach to writing this style of test. But as I mentioned, you hit >>>> the limits pretty quickly. >>> >>> I think it's great, it looks like exactly what I wanted, except it's >>> been delivered on time. >> >> >> Can you take a look at how interrupts are handled? From what I >> gather, the only real limitation of this approach is that we won't be >> able to simulate MSI vectors but I don't think that's a huge problem. >> >> I looked at integrating interrupt handling at CPUs itself and that >> turned out to be fairly invasive. > > You mention in the changelog replacing the APIC. Why can't you do that? I currently replace the I/O APIC which seems to be limited to 16 IRQs. I think MSI takes a side channel directly to the local APIC, no? >> >>> I'd really like to see it integrated quickly >>> with some flesh around it, then replying -ENOTEST to all patches. This >>> will improve qemu's quality a lot more than guest boot/ping tests, which >>> we do regularly with kvm-autotest anyway. >>> >>> Think of how new instruction emulations are always accompanied by new >>> kvm-unit-tests tests, I often don't even have to ask for them. >> >> Yes, this is exactly where I'm heading with all of this. > > It looks great. One thing I'd change is to use the qmp protocol > (perhaps not the monitor, just the schema/codegen). Does something > prohibit this? Yeah, I thought about using QMP. But events are posted in QMP which means that you never get an explicit ACK. That may not be a problem but it's something I had in mind. It also seemed to be reasonably complicated without a clear advantage. qtest isn't going to be a supported protocol so we can certainly change it down the road if we want to. Regards, Anthony Liguori >
On 12/29/2011 06:39 PM, Anthony Liguori wrote: > > qemu-test builds a custom guest, entirely from source. This makes it > possible to efficiently test non-native architectures. The tests are > written for this minimal environment (which is not straight forward to > do). This tests will never run on Windows as they are 100% tied to > their environment. > > autotest (let's be clear, there is no such thing as kvm autotest > anymore) is a framework for executing third party tests. It's got > fancy magic to execute in client mode vs. server mode. There is a > suite of tests that are integrated in autotest that exercise various > types of virtualization functionality. This suite of tests use a > special config format to determine what they do. > > Included in this, is the ability to create a guest from an ISO either > using step files or via a guest-specific mechanism (kickstart, etc.). > The tests are written to be mostly guest agnostic and are therefore > written in Python in a cross platform way. > > There is essentially no overlap between the two and calling kvm > autotest superior is like calling the Sun superior to the Moon. Why not extend autotest do the same thing. Instead of downloading a fedora iso it would download a kernel tarball and (cross-)build it. > >> My main motivation to merge >> qemu-test and kvm autotest is that I fear that qemu-test will be the >> only >> requirement for committing stuff for qemu and developers will be >> excused from >> their 'duty' by writing a 50 LOC shell script and assume they done w/ >> testing. > > There is no requirement to write autotest tests to get things merged > into QEMU. That's is how things are today. > > And I don't think there will ever a requirement to write anything that > isn't merged directly into qemu.git. I'm not going to hold up merging > a feature until another project merges something into their tree. What about virtio features (we used to depend on the kernel, now on the spec)? Seabios? > > So unless we're going to merge KVM autotest into qemu.git, I don't > think there's every going to be a requirement to write a KVM autotest > test in order to get something merged into qemu.git. Let's consider postcopy live migration being proposed. In addition to various unit tests, it also wants an integration test. So now we need to write a qemu-test postcopy live migration test, and also autotest test that tests non-Linux guests, migration under load, with memory hotplug, etc. > > But since autotest is a framework for running third-party tests, it > seems to make sense for qemu.git to have a test framework that > autotest can execute. > > And since what we call KVM autotest actually tests a heck of a lot > more than just QEMU, it makes sense for KVM autotest to continue to > focus on full stack testing where QEMU is but one of the many > components that it tests. It might have made sense to split the kvm-testing functionality of autotest, and have autotest drive that. We could even have called it qemu-test.
On 12/29/2011 10:46 AM, Avi Kivity wrote: > On 12/29/2011 06:26 PM, Anthony Liguori wrote: >>>> I don't want to write a TCP/IP stack. We aren't just grabbing a >>>> random distro kernel. We're building one from scratch configured in a >>>> specific way. >>> >>> How does that help? >> >> Not sure I understand the question. >> > > In what way is your specifically configured kernel's TCP stack better > than the random distro's kernel's? I firmly believe that with qtest we'll end up eventually building a libOS to make it easier to write qtest tests. Overtime, that libOS will become increasingly complex up until the point where it approaches something that feels like an actual OS. Effort spent developing libOS is a cost to building test cases. By using Linux and a minimal userspace as our libOS, we can avoid spending a lot of time building a sophisticated libOS. If we need advanced libOS features, we just use qemu-test. If it's just a matter of poking some registers on a device along, we just use qtest. Guest neutral tests that are meant to run on Linux, Windows, etc. are in a completely different ballpark. Regards, Anthony Liguori
On 12/29/2011 10:53 AM, Avi Kivity wrote: > On 12/29/2011 06:39 PM, Anthony Liguori wrote: >> >> qemu-test builds a custom guest, entirely from source. This makes it >> possible to efficiently test non-native architectures. The tests are >> written for this minimal environment (which is not straight forward to >> do). This tests will never run on Windows as they are 100% tied to >> their environment. >> >> autotest (let's be clear, there is no such thing as kvm autotest >> anymore) is a framework for executing third party tests. It's got >> fancy magic to execute in client mode vs. server mode. There is a >> suite of tests that are integrated in autotest that exercise various >> types of virtualization functionality. This suite of tests use a >> special config format to determine what they do. >> >> Included in this, is the ability to create a guest from an ISO either >> using step files or via a guest-specific mechanism (kickstart, etc.). >> The tests are written to be mostly guest agnostic and are therefore >> written in Python in a cross platform way. >> >> There is essentially no overlap between the two and calling kvm >> autotest superior is like calling the Sun superior to the Moon. > > Why not extend autotest do the same thing. Instead of downloading a > fedora iso it would download a kernel tarball and (cross-)build it. My plan is to post a qemu-jeos repo that has the build bits for creating the kernel/initramfs pair. qemu-test will then live in qemu.git and just depend on the results of the qemu-jeos.git. I'd maybe even go as far as including the binaries in qemu.git but I don't think that would work so well... >>> My main motivation to merge >>> qemu-test and kvm autotest is that I fear that qemu-test will be the >>> only >>> requirement for committing stuff for qemu and developers will be >>> excused from >>> their 'duty' by writing a 50 LOC shell script and assume they done w/ >>> testing. >> >> There is no requirement to write autotest tests to get things merged >> into QEMU. That's is how things are today. >> >> And I don't think there will ever a requirement to write anything that >> isn't merged directly into qemu.git. I'm not going to hold up merging >> a feature until another project merges something into their tree. > > What about virtio features (we used to depend on the kernel, now on the > spec)? Seabios? Those are rare occurrences and have always been a bit awkward. OTOH, we're talking about the critical path of essentially every single feature that gets merged into QEMU. >> So unless we're going to merge KVM autotest into qemu.git, I don't >> think there's every going to be a requirement to write a KVM autotest >> test in order to get something merged into qemu.git. > > Let's consider postcopy live migration being proposed. In addition to > various unit tests, it also wants an integration test. So now we need > to write a qemu-test postcopy live migration test, and also autotest > test that tests non-Linux guests, migration under load, with memory > hotplug, etc. But the integration test is also going to depend on libvirt support for the feature. So how does this all get orchestrated? kvm-autotest merges the test case first, then QEMU merges the support, then libvirt? What about VDSM and ovirt-engine? Once kvm-autotest starts supporting VDSM that's yet another component to deal with. Realistically, kvm-autotest is going to have to wait for the bits to get merged in all of the necessary pieces and then build a test for it on their own schedule. >> But since autotest is a framework for running third-party tests, it >> seems to make sense for qemu.git to have a test framework that >> autotest can execute. >> >> And since what we call KVM autotest actually tests a heck of a lot >> more than just QEMU, it makes sense for KVM autotest to continue to >> focus on full stack testing where QEMU is but one of the many >> components that it tests. > > It might have made sense to split the kvm-testing functionality of > autotest, and have autotest drive that. We could even have called it > qemu-test. I specifically advocated this during Lucas' KVM Forum talk and he was strongly opposed to it. I think kvm autotest would get a lot more interest if the test cases were pulled out of autotest, made more stand alone. They also should be more Unix like being divided into individual executables with independent command line options over a single do-everything configuration file. But that's all independent of qemu-test. qemu-test is really qtest + Linux/busybox as a libOS. Even if the above happened, I would still think qemu-test needed to exist. Regards, Anthony Liguori
On 12/29/2011 06:49 PM, Anthony Liguori wrote: > On 12/29/2011 10:36 AM, Avi Kivity wrote: >> On 12/29/2011 06:12 PM, Anthony Liguori wrote: >>>>> That's why I've also proposed qtest. But having written quite a few >>>>> qtest unit tests by now, you hit the limits of this type of testing >>>>> pretty quickly. >>>> >>>> Can you describe those limits? >>> >>> >>> I started writing a finger print test. While it's easy to do PCI >>> enumeration, it gets pretty nasty if you want to actually access BARs >>> (to read virtio registers) >> >> Why is that? it should be pretty easy to poke values into the BAR >> registers. > > You have to map them which means you need to figure out what the > memory layout looks like. Since there's no BIOS, you need to poke > fw_cfg to figure all of this out and then program the BARs appropriately. > > It's definitely non-trivial. Or just map sequentially at 0xe0000000 ascending. > >> Something else we can do is access the BARs directly. We have APIs to >> poke values into mmio, add an api to poke a value into a device's BAR. >> Now that each BAR is represented by exactly one memory region, known to >> the pci core even if it is not mapped, it should be easy. >> >>> and forget about trying to get SMBIOS information as that involves >>> mucking around with ACPI. >> >> SMBIOS is built by seabios, yes? > > I think QEMU actually builds the blob of information and passes the > full blob to SeaBIOS. You could read and parse the blob out of fw_cfg > I guess but that's still pretty sucky. It's a lot nicer to just poke > sysfs. Seems like two lines of code, either way. > >> So fingerprinting it should be done >> from seabios-test.git. Let's divide it into two problems: > > That seems awkward especially since fingerprinting Just Works with > qemu-test already (and it was only 50 lines of shell code!). > >> - seabios pokes qemu to get information. We should fingerprint this >> information to make sure different qemu version can interoperate with >> different seabios versions (needed since we migrate the bios along with >> the rest of the guest). I'm guessing most of this info is from fwcfg? >> We can easily fingerprint it like any other device. >> - we need to make sure seabios generates the same tables when using -M. >> Here, we're not testing a device, rather we're testing guest code, so it >> makes sense to use a guest for this. >> >> However, I don't think it's even necessary. From a quick read of the >> manual, SMBIOS is just a set of static tables in memory which are picked >> up using a signature. So all we need to do is boot an empty guest, read >> its memory, and look for the tables. > > Doesn't it sound a whole nicer use linux.git to parse this information > for us in a friendly, easy to consume fashion? Run 'dmidecode -d /path/to/memory/dump', if you must. I don't think the qemu-test approach is bad, per se, it's just that qtest is better. It gives you full control over what to fingerprint and is not reliant on Linux not changing. >> >> Yes; but using Linux limits you to what it exposes (of course Linux >> exposes quite a lot, so that's mostly a non issue; but we'll have >> counterexamples). > > > Maybe. And for those counter examples, we can drill down to qtest. > But just because we may need to do fancy things, it doesn't mean we > shouldn't take the easy approach 95% of the time. I guess we have to see how difficult it will be with qtest. My feeling is that it will be very easy. >> I don't see why. A python library to read pci config should be a couple >> of dozens of lines long. Then you iterate over all functions and print >> selected registers. > > > PCI enumeration is easy. It's mapping the bars and then poking other > sources of information that's more challenging. > > Everything is doable by writing a libOS but I'd rather just use Linux > than invent a libOS just for testing. We will have to write a libOS to test devices. To test, say, virtio-net with indirect buffers, you have to map BARs (or poke unmapped BARs directly). >> >> You mention in the changelog replacing the APIC. Why can't you do that? > > > I currently replace the I/O APIC which seems to be limited to 16 > IRQs. I think MSI takes a side channel directly to the local APIC, no? Yes, writes to the 1MB @ 0xfee00000 gets translated to MSIs. So you just translate them to events. >> It looks great. One thing I'd change is to use the qmp protocol >> (perhaps not the monitor, just the schema/codegen). Does something >> prohibit this? > > > Yeah, I thought about using QMP. But events are posted in QMP which > means that you never get an explicit ACK. That may not be a problem > but it's something I had in mind. > > It also seemed to be reasonably complicated without a clear > advantage. qtest isn't going to be a supported protocol so we can > certainly change it down the road if we want to. It's a pity not to reuse all the tooling? Seems like self-NIH.
On 12/29/2011 07:02 PM, Anthony Liguori wrote: > > Those are rare occurrences and have always been a bit awkward. OTOH, > we're talking about the critical path of essentially every single > feature that gets merged into QEMU. > Won't that be qtest? Major features are rare. Small changes are common. >> Let's consider postcopy live migration being proposed. In addition to >> various unit tests, it also wants an integration test. So now we need >> to write a qemu-test postcopy live migration test, and also autotest >> test that tests non-Linux guests, migration under load, with memory >> hotplug, etc. > > But the integration test is also going to depend on libvirt support > for the feature. So how does this all get orchestrated? kvm-autotest > merges the test case first, then QEMU merges the support, then > libvirt? What about VDSM and ovirt-engine? Once kvm-autotest starts > supporting VDSM that's yet another component to deal with. > > Realistically, kvm-autotest is going to have to wait for the bits to > get merged in all of the necessary pieces and then build a test for it > on their own schedule. It just has to be posted and reviewed, it doesn't all have to be merged at the same time (and the qemu merge has to precede the autotest merge)
On 12/29/2011 06:53 PM, Anthony Liguori wrote: >> In what way is your specifically configured kernel's TCP stack better >> than the random distro's kernel's? > > > I firmly believe that with qtest we'll end up eventually building a > libOS to make it easier to write qtest tests. > > Overtime, that libOS will become increasingly complex up until the > point where it approaches something that feels like an actual OS. > Effort spent developing libOS is a cost to building test cases. > > By using Linux and a minimal userspace as our libOS, we can avoid > spending a lot of time building a sophisticated libOS. If we need > advanced libOS features, we just use qemu-test. If it's just a matter > of poking some registers on a device along, we just use qtest. Would there be device-level tests in qemu-test?
On 12/29/2011 11:03 AM, Avi Kivity wrote: > On 12/29/2011 06:49 PM, Anthony Liguori wrote: >>> However, I don't think it's even necessary. From a quick read of the >>> manual, SMBIOS is just a set of static tables in memory which are picked >>> up using a signature. So all we need to do is boot an empty guest, read >>> its memory, and look for the tables. >> >> Doesn't it sound a whole nicer use linux.git to parse this information >> for us in a friendly, easy to consume fashion? > > Run 'dmidecode -d /path/to/memory/dump', if you must. > > I don't think the qemu-test approach is bad, per se, it's just that > qtest is better. It gives you full control over what to fingerprint and > is not reliant on Linux not changing. Maybe. But I feel a lot more comfortable asking people to write qemu-test's than writing low level logic to do this stuff in qtest. Maybe over time, we'll have a good enough libOS for qtest that qemu-test won't be that useful anymore. But in the short term, I think we'll get more test coverage by having both. >>> >>> You mention in the changelog replacing the APIC. Why can't you do that? >> >> >> I currently replace the I/O APIC which seems to be limited to 16 >> IRQs. I think MSI takes a side channel directly to the local APIC, no? > > Yes, writes to the 1MB @ 0xfee00000 gets translated to MSIs. So you > just translate them to events. Ah, okay, I wasn't thinking of that. I'll clean qtest up and resubmit next week. >>> It looks great. One thing I'd change is to use the qmp protocol >>> (perhaps not the monitor, just the schema/codegen). Does something >>> prohibit this? >> >> >> Yeah, I thought about using QMP. But events are posted in QMP which >> means that you never get an explicit ACK. That may not be a problem >> but it's something I had in mind. >> >> It also seemed to be reasonably complicated without a clear >> advantage. qtest isn't going to be a supported protocol so we can >> certainly change it down the road if we want to. > > It's a pity not to reuse all the tooling? Seems like self-NIH. No, it's just going to be more work to reuse things. I was careful to make sure that it was possible down the road. We have a bit of way to go before the QAPI infrastructure is generic enough to be used for something like this. Regards, Anthony Liguori
On 12/29/2011 11:06 AM, Avi Kivity wrote: > On 12/29/2011 07:02 PM, Anthony Liguori wrote: >> >> Those are rare occurrences and have always been a bit awkward. OTOH, >> we're talking about the critical path of essentially every single >> feature that gets merged into QEMU. >> > > Won't that be qtest? The difference is that the qtest test can be patch N/N in a patch series and be merged at exactly the same time. I think having an in-tree test suite is the only practical way to do TDD. Regards, Anthony Liguori >>> Let's consider postcopy live migration being proposed. In addition to >>> various unit tests, it also wants an integration test. So now we need >>> to write a qemu-test postcopy live migration test, and also autotest >>> test that tests non-Linux guests, migration under load, with memory >>> hotplug, etc. >> >> But the integration test is also going to depend on libvirt support >> for the feature. So how does this all get orchestrated? kvm-autotest >> merges the test case first, then QEMU merges the support, then >> libvirt? What about VDSM and ovirt-engine? Once kvm-autotest starts >> supporting VDSM that's yet another component to deal with. >> >> Realistically, kvm-autotest is going to have to wait for the bits to >> get merged in all of the necessary pieces and then build a test for it >> on their own schedule. > > It just has to be posted and reviewed, it doesn't all have to be merged > at the same time (and the qemu merge has to precede the autotest merge) >
On 12/29/2011 11:08 AM, Avi Kivity wrote: > On 12/29/2011 06:53 PM, Anthony Liguori wrote: >>> In what way is your specifically configured kernel's TCP stack better >>> than the random distro's kernel's? >> >> >> I firmly believe that with qtest we'll end up eventually building a >> libOS to make it easier to write qtest tests. >> >> Overtime, that libOS will become increasingly complex up until the >> point where it approaches something that feels like an actual OS. >> Effort spent developing libOS is a cost to building test cases. >> >> By using Linux and a minimal userspace as our libOS, we can avoid >> spending a lot of time building a sophisticated libOS. If we need >> advanced libOS features, we just use qemu-test. If it's just a matter >> of poking some registers on a device along, we just use qtest. > > Would there be device-level tests in qemu-test? What is a "device-level" test? Take a look at: http://git.qemu.org/?p=qemu-test.git;a=blob;f=scripts/bin/fingerprint;h=2d4202a826917b16856a2acb4617f623fdc4c0d3;hb=HEAD It's reading BAR0 from any virtio devices to determine the guest features exposed. Yes, it's written in sh :-) It uses the BAR mappings that sysfs exposes. Regards, Anthony Liguori >
On 12/29/2011 11:08 AM, Avi Kivity wrote: > On 12/29/2011 06:53 PM, Anthony Liguori wrote: >>> In what way is your specifically configured kernel's TCP stack better >>> than the random distro's kernel's? >> >> >> I firmly believe that with qtest we'll end up eventually building a >> libOS to make it easier to write qtest tests. >> >> Overtime, that libOS will become increasingly complex up until the >> point where it approaches something that feels like an actual OS. >> Effort spent developing libOS is a cost to building test cases. >> >> By using Linux and a minimal userspace as our libOS, we can avoid >> spending a lot of time building a sophisticated libOS. If we need >> advanced libOS features, we just use qemu-test. If it's just a matter >> of poking some registers on a device along, we just use qtest. > > Would there be device-level tests in qemu-test? qemu-jeos has a kernel build environment for the target so it is also possible to build userspace C programs and/or kernel modules so you could also write guest tests in C. So it may actually be reasonable to write a special virtio-test kernel module actually. Regards, Anthony Liguori
On 12/29/2011 07:10 PM, Anthony Liguori wrote: > On 12/29/2011 11:03 AM, Avi Kivity wrote: >> On 12/29/2011 06:49 PM, Anthony Liguori wrote: >>>> However, I don't think it's even necessary. From a quick read of the >>>> manual, SMBIOS is just a set of static tables in memory which are >>>> picked >>>> up using a signature. So all we need to do is boot an empty guest, >>>> read >>>> its memory, and look for the tables. >>> >>> Doesn't it sound a whole nicer use linux.git to parse this information >>> for us in a friendly, easy to consume fashion? >> >> Run 'dmidecode -d /path/to/memory/dump', if you must. >> >> I don't think the qemu-test approach is bad, per se, it's just that >> qtest is better. It gives you full control over what to fingerprint and >> is not reliant on Linux not changing. > > Maybe. But I feel a lot more comfortable asking people to write > qemu-test's than writing low level logic to do this stuff in qtest. Writing the libOS is definitely a job for the (sub-)maintainers. The work to code a unit test for patch X should be O(X). >>> IRQs. I think MSI takes a side channel directly to the local APIC, no? >> >> Yes, writes to the 1MB @ 0xfee00000 gets translated to MSIs. So you >> just translate them to events. > I currently replace the I/O APIC which seems to be limited to 16 > > Ah, okay, I wasn't thinking of that. I'll clean qtest up and resubmit > next week. Great. I'll be happy to help writing libOS. >>> It also seemed to be reasonably complicated without a clear >>> advantage. qtest isn't going to be a supported protocol so we can >>> certainly change it down the road if we want to. >> >> It's a pity not to reuse all the tooling? Seems like self-NIH. > > > No, it's just going to be more work to reuse things. I was careful > to make sure that it was possible down the road. > > We have a bit of way to go before the QAPI infrastructure is generic > enough to be used for something like this. I've started to simplify the build procedure, maybe it will help a tiny bit.
On 29 December 2011 16:36, Avi Kivity <avi@redhat.com> wrote: > Yes; but using Linux limits you to what it exposes (of course Linux > exposes quite a lot, so that's mostly a non issue; but we'll have > counterexamples). Actually IME Linux is pretty conservative about how it uses devices. A lot of the ARM device models have gaping holes in their emulation which we've never needed to fix because Linux simply doesn't use those features (eg the PL080 DMA controller which doesn't actually implement DMA!) I think for devices what would be particularly useful would be if you can write a (simple) test for something at the register level, which generates an image which you can run on the real hardware as well as on QEMU. Then you can confirm that your test case is correct. Otherwise the tests are just going to bake in the same assumptions/misconceptions about what the hardware does as the QEMU model. My guess is that a serious attempt at tests covering all the functionality of a device is probably approximately doubling the effort required for a device model, incidentally. A half-hearted attempt probably doesn't buy you much over automating "boot the guest OS and prod its driver". -- PMM
On 12/29/2011 07:14 PM, Anthony Liguori wrote: > On 12/29/2011 11:08 AM, Avi Kivity wrote: >> On 12/29/2011 06:53 PM, Anthony Liguori wrote: >>>> In what way is your specifically configured kernel's TCP stack better >>>> than the random distro's kernel's? >>> >>> >>> I firmly believe that with qtest we'll end up eventually building a >>> libOS to make it easier to write qtest tests. >>> >>> Overtime, that libOS will become increasingly complex up until the >>> point where it approaches something that feels like an actual OS. >>> Effort spent developing libOS is a cost to building test cases. >>> >>> By using Linux and a minimal userspace as our libOS, we can avoid >>> spending a lot of time building a sophisticated libOS. If we need >>> advanced libOS features, we just use qemu-test. If it's just a matter >>> of poking some registers on a device along, we just use qtest. >> >> Would there be device-level tests in qemu-test? > > What is a "device-level" test? > A test that tests just one device (or a patch to a single device's device emulation). > Take a look at: > > http://git.qemu.org/?p=qemu-test.git;a=blob;f=scripts/bin/fingerprint;h=2d4202a826917b16856a2acb4617f623fdc4c0d3;hb=HEAD > > > It's reading BAR0 from any virtio devices to determine the guest > features exposed. Yes, it's written in sh :-) It would be trivial once libos exists. And we need libos so we can -ENOTEST device patches, yes? > It uses the BAR mappings that sysfs exposes.
On 12/29/2011 07:16 PM, Anthony Liguori wrote: >> Would there be device-level tests in qemu-test? > > > qemu-jeos has a kernel build environment for the target so it is also > possible to build userspace C programs and/or kernel modules so you > could also write guest tests in C. > > So it may actually be reasonable to write a special virtio-test kernel > module actually. Better to use vfio or uio for that, IMO. But qtest is better still.
On 12/29/2011 07:22 PM, Peter Maydell wrote: > On 29 December 2011 16:36, Avi Kivity <avi@redhat.com> wrote: > > Yes; but using Linux limits you to what it exposes (of course Linux > > exposes quite a lot, so that's mostly a non issue; but we'll have > > counterexamples). > > Actually IME Linux is pretty conservative about how it uses devices. > A lot of the ARM device models have gaping holes in their emulation > which we've never needed to fix because Linux simply doesn't use > those features (eg the PL080 DMA controller which doesn't actually > implement DMA!) We were discussing fingerprinting, not actually driving the device. For that, I think we're all agreed that qtest is vastly superior to using an OS driver. > I think for devices what would be particularly useful would be > if you can write a (simple) test for something at the register > level, which generates an image which you can run on the real > hardware as well as on QEMU. Then you can confirm that your test > case is correct. Otherwise the tests are just going to bake in the > same assumptions/misconceptions about what the hardware does as > the QEMU model. That is exactly qtest. > My guess is that a serious attempt at tests covering all the > functionality of a device is probably approximately doubling > the effort required for a device model, incidentally. A > half-hearted attempt probably doesn't buy you much over > automating "boot the guest OS and prod its driver". Agreed.
On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote: > On 12/29/2011 07:22 PM, Peter Maydell wrote: >> I think for devices what would be particularly useful would be >> if you can write a (simple) test for something at the register >> level, which generates an image which you can run on the real >> hardware as well as on QEMU. Then you can confirm that your test >> case is correct. Otherwise the tests are just going to bake in the >> same assumptions/misconceptions about what the hardware does as >> the QEMU model. > > That is exactly qtest. Am I missing something? I looked at the qtest patches and they seem to work by just prodding the device through qemu function calls, not by running a guest CPU binary that prods the device. -- PMM
On 12/29/2011 07:36 PM, Peter Maydell wrote: > On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote: > > On 12/29/2011 07:22 PM, Peter Maydell wrote: > >> I think for devices what would be particularly useful would be > >> if you can write a (simple) test for something at the register > >> level, which generates an image which you can run on the real > >> hardware as well as on QEMU. Then you can confirm that your test > >> case is correct. Otherwise the tests are just going to bake in the > >> same assumptions/misconceptions about what the hardware does as > >> the QEMU model. > > > > That is exactly qtest. > > Am I missing something? I looked at the qtest patches and they seem > to work by just prodding the device through qemu function calls, > not by running a guest CPU binary that prods the device. Sorry, I misread your post. I guess we can point a qtest test at real hardware too, if something like vfio is available to provide access to registers and to hook interrupts.
On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote: > On 12/29/2011 07:22 PM, Peter Maydell wrote: >> My guess is that a serious attempt at tests covering all the >> functionality of a device is probably approximately doubling >> the effort required for a device model, incidentally. A >> half-hearted attempt probably doesn't buy you much over >> automating "boot the guest OS and prod its driver". > > Agreed. The next obvious question is: are we going to make a serious attempt? (For instance, in a hypothetical tests-required world, would we tell those nice folks from Samsung "no you can't land your Exynos patches unless you write 9000+ lines of test cases" ? I suspect that if we set the bar for new board and device models that high then the result will largely be that we don't in fact get new board or device models.) -- PMM
On 12/29/2011 07:49 PM, Peter Maydell wrote: > On 29 December 2011 17:26, Avi Kivity <avi@redhat.com> wrote: > > On 12/29/2011 07:22 PM, Peter Maydell wrote: > >> My guess is that a serious attempt at tests covering all the > >> functionality of a device is probably approximately doubling > >> the effort required for a device model, incidentally. A > >> half-hearted attempt probably doesn't buy you much over > >> automating "boot the guest OS and prod its driver". > > > > Agreed. > > The next obvious question is: are we going to make a serious attempt? > (For instance, in a hypothetical tests-required world, would we > tell those nice folks from Samsung "no you can't land your > Exynos patches unless you write 9000+ lines of test cases" ? Yes. > I suspect that if we set the bar for new board and device models > that high then the result will largely be that we don't in fact > get new board or device models.) If just doubles the effort, I disagree. Even if that turns out to be the case, it's fine. Better to have a few good devices than dozens of bad ones.
On 12/29/2011 11:22 AM, Avi Kivity wrote: > On 12/29/2011 07:14 PM, Anthony Liguori wrote: >> On 12/29/2011 11:08 AM, Avi Kivity wrote: >>> On 12/29/2011 06:53 PM, Anthony Liguori wrote: >>>>> In what way is your specifically configured kernel's TCP stack better >>>>> than the random distro's kernel's? >>>> >>>> >>>> I firmly believe that with qtest we'll end up eventually building a >>>> libOS to make it easier to write qtest tests. >>>> >>>> Overtime, that libOS will become increasingly complex up until the >>>> point where it approaches something that feels like an actual OS. >>>> Effort spent developing libOS is a cost to building test cases. >>>> >>>> By using Linux and a minimal userspace as our libOS, we can avoid >>>> spending a lot of time building a sophisticated libOS. If we need >>>> advanced libOS features, we just use qemu-test. If it's just a matter >>>> of poking some registers on a device along, we just use qtest. >>> >>> Would there be device-level tests in qemu-test? >> >> What is a "device-level" test? >> > > A test that tests just one device (or a patch to a single device's > device emulation). > >> Take a look at: >> >> http://git.qemu.org/?p=qemu-test.git;a=blob;f=scripts/bin/fingerprint;h=2d4202a826917b16856a2acb4617f623fdc4c0d3;hb=HEAD >> >> >> It's reading BAR0 from any virtio devices to determine the guest >> features exposed. Yes, it's written in sh :-) > > It would be trivial once libos exists. And we need libos so we can > -ENOTEST device patches, yes? Or qemu-test. I'd be happy with a test like the virtio-serial test that I posted for qemu-test. I expect we'll have a solid libos for x86 but I doubt we'll ever have a solid libos for other targets. Regards, Anthony Liguori > >> It uses the BAR mappings that sysfs exposes. > > >
On 12/29/2011 11:22 AM, Peter Maydell wrote: > On 29 December 2011 16:36, Avi Kivity<avi@redhat.com> wrote: >> Yes; but using Linux limits you to what it exposes (of course Linux >> exposes quite a lot, so that's mostly a non issue; but we'll have >> counterexamples). > > Actually IME Linux is pretty conservative about how it uses devices. > A lot of the ARM device models have gaping holes in their emulation > which we've never needed to fix because Linux simply doesn't use > those features (eg the PL080 DMA controller which doesn't actually > implement DMA!) > > I think for devices what would be particularly useful would be > if you can write a (simple) test for something at the register > level, which generates an image which you can run on the real > hardware as well as on QEMU. Then you can confirm that your test > case is correct. Otherwise the tests are just going to bake in the > same assumptions/misconceptions about what the hardware does as > the QEMU model. I think that could be useful, but it orthogonal to what we need for Test Driven Development. The problem today is that it is very hard to make broad changes in QEMU simply because it's so difficult to know whether you're breaking another target. Look at the challenges around merging the memory API. We need a simple way that a developer can sanity check that they've not broken other targets and all of the various devices that we support. > > My guess is that a serious attempt at tests covering all the > functionality of a device is probably approximately doubling > the effort required for a device model, incidentally. A > half-hearted attempt probably doesn't buy you much over > automating "boot the guest OS and prod its driver". Right, but "boot the guest OS and prod its driver" is very useful particularly if it can cover all of the various targets we support. I don't think we should focus much on qtest for non-x86 targets. I mean, if you are interested in it for ARM, fantastic, but I don't think we would mandate it. For x86, it's reasonable to have stricter requirements on testing, particularly for new devices. FWIW, I expect virtio-scsi to be the first guinea pig here... I believe Stefan has already started looking at writing some qtest based tests for virtio-scsi. Regards, Anthony Liguori > -- PMM >
On 12/29/2011 11:49 AM, Peter Maydell wrote: > On 29 December 2011 17:26, Avi Kivity<avi@redhat.com> wrote: >> On 12/29/2011 07:22 PM, Peter Maydell wrote: >>> My guess is that a serious attempt at tests covering all the >>> functionality of a device is probably approximately doubling >>> the effort required for a device model, incidentally. A >>> half-hearted attempt probably doesn't buy you much over >>> automating "boot the guest OS and prod its driver". >> >> Agreed. > > The next obvious question is: are we going to make a serious attempt? > (For instance, in a hypothetical tests-required world, would we > tell those nice folks from Samsung "no you can't land your > Exynos patches unless you write 9000+ lines of test cases" ? The virtio-serial test case I posted was 50 lines in qemu-test. The virtio-serial driver is about ~1500 LOC. That's about 3%. I would expect that we at least have some sort of test that could verify that the Exynos platform more or less worked as expected. If that was just booting a Linux kernel, that would be fine by me. > I suspect that if we set the bar for new board and device models > that high then the result will largely be that we don't in fact > get new board or device models.) This is why the barrier needs to be as low as possible. Regards, Anthony Liguori > -- PMM >
On 29 December 2011 18:35, Anthony Liguori <anthony@codemonkey.ws> wrote: > On 12/29/2011 11:49 AM, Peter Maydell wrote: >> The next obvious question is: are we going to make a serious attempt? >> (For instance, in a hypothetical tests-required world, would we >> tell those nice folks from Samsung "no you can't land your >> Exynos patches unless you write 9000+ lines of test cases" ? > > The virtio-serial test case I posted was 50 lines in qemu-test. The > virtio-serial driver is about ~1500 LOC. That's about 3%. This just means it's testing only a fraction of what virtio-serial actually implements (as you note yourself in the comments for the test case). For serious coverage you'd also need to cover reading(!), larger quantities of data than single lines, what happens when one end feeds in data but the other end isn't reading, vmstate save/load, multiple simultaneous ports, behaviour on close-and-reopen, whether it works on bigendian targets, benchmarking to identify possible performance regressions, etc etc. I think that by the time you've done all that you'll be closer to 1500 lines than 50. > I would expect that we at least have some sort of test that could verify > that the Exynos platform more or less worked as expected. If that was just > booting a Linux kernel, that would be fine by me. Yes. There's a large range between "no tests required at all" (essentially what we have now) and "an equivalent level of device testing to what you would carry out before sending your hardware design out to be fabbed into silicon" (which I hope we'd all agree would be ludicrously high for QEMU); I'm trying to establish where we're attempting to set our bar. I agree that we'd get a lot of bang-for-the-buck out of basic automated "boot the guest and prod it" testing that covered most of our platforms. -- PMM
On Thu, Dec 29, 2011 at 19:04, Peter Maydell <peter.maydell@linaro.org> wrote: > On 29 December 2011 18:35, Anthony Liguori <anthony@codemonkey.ws> wrote: >> On 12/29/2011 11:49 AM, Peter Maydell wrote: >>> The next obvious question is: are we going to make a serious attempt? >>> (For instance, in a hypothetical tests-required world, would we >>> tell those nice folks from Samsung "no you can't land your >>> Exynos patches unless you write 9000+ lines of test cases" ? >> >> The virtio-serial test case I posted was 50 lines in qemu-test. The >> virtio-serial driver is about ~1500 LOC. That's about 3%. > > This just means it's testing only a fraction of what virtio-serial > actually implements (as you note yourself in the comments for the > test case). For serious coverage you'd also need to cover reading(!), > larger quantities of data than single lines, what happens when one end > feeds in data but the other end isn't reading, vmstate save/load, > multiple simultaneous ports, behaviour on close-and-reopen, > whether it works on bigendian targets, benchmarking to identify > possible performance regressions, etc etc. I think that by the time > you've done all that you'll be closer to 1500 lines than 50. The coverage point of view is interesting. I'd expect the coverage rate to rise as one moves from qtest via qemu-test to autotest. However, I'd also expect that having qtest and qemu-test would both increase chances that patches also get test coverage. Gathering test coverage data could be a start. >> I would expect that we at least have some sort of test that could verify >> that the Exynos platform more or less worked as expected. If that was just >> booting a Linux kernel, that would be fine by me. > > Yes. There's a large range between "no tests required at all" (essentially > what we have now) and "an equivalent level of device testing to what you > would carry out before sending your hardware design out to be fabbed into > silicon" (which I hope we'd all agree would be ludicrously high for QEMU); > I'm trying to establish where we're attempting to set our bar. > > I agree that we'd get a lot of bang-for-the-buck out of basic automated > "boot the guest and prod it" testing that covered most of our platforms. That may also be too simplistic. For example, sparc-test (boot Sparc32 Linux with busybox) does not catch doubleword access problems or certain bugs in floating point point emulation. For those, I need a boot from an install CD which starts X. Overall, I'd think that all of qtest, qemu-test and autotest have their place. If one brilliant day autotest becomes so lean and mean that qemu-test is unnecessary, it can be deprecated.
On 29 December 2011 17:56, Avi Kivity <avi@redhat.com> wrote: > On 12/29/2011 07:49 PM, Peter Maydell wrote: >> I suspect that if we set the bar for new board and device models >> that high then the result will largely be that we don't in fact >> get new board or device models.) > > If just doubles the effort, I disagree. If you turn a six month project into a twelve month project I suspect that will push it beyond the "worth doing" limit in many cases. (Or more likely, people just won't bother to send their models upstream, and we'll have more forks out there.) > Even if that turns out to be the case, it's fine. Better to > have a few good devices than dozens of bad ones. This is easier to say on the x86 side of the fence, since most of the devices you need are already in the codebase and nobody is going to throw them out again if tests don't get written for them :-) There are lots of things where I'd rather have a "tested by booting a guest OS" implementation than none at all (like audio support for the versatile/etc boards, which went in recently) or TrustZone support (not in yet but may be along later). At least then we have something that works for most people and something we can fix bugs in, rather than a gaping hole in capability. -- PMM
On 12/29/2011 01:04 PM, Peter Maydell wrote: > On 29 December 2011 18:35, Anthony Liguori<anthony@codemonkey.ws> wrote: >> On 12/29/2011 11:49 AM, Peter Maydell wrote: >>> The next obvious question is: are we going to make a serious attempt? >>> (For instance, in a hypothetical tests-required world, would we >>> tell those nice folks from Samsung "no you can't land your >>> Exynos patches unless you write 9000+ lines of test cases" ? >> >> The virtio-serial test case I posted was 50 lines in qemu-test. The >> virtio-serial driver is about ~1500 LOC. That's about 3%. > > This just means it's testing only a fraction of what virtio-serial > actually implements (as you note yourself in the comments for the > test case). For serious coverage you'd also need to cover reading(!), > larger quantities of data than single lines, what happens when one end > feeds in data but the other end isn't reading, vmstate save/load, > multiple simultaneous ports, behaviour on close-and-reopen, > whether it works on bigendian targets, benchmarking to identify > possible performance regressions, etc etc. I think that by the time > you've done all that you'll be closer to 1500 lines than 50. At the end of the day, the right level depth of tests is something we'll need to work out. Realistically, the level of test coverage is directly proportional to how upset will get when someone breaks that particular function. But 0% test coverage is absolutely not enough. So the first step is to get an infrastructure together that we can all live with and start writing tests. Regards, Anthony Liguori > >> I would expect that we at least have some sort of test that could verify >> that the Exynos platform more or less worked as expected. If that was just >> booting a Linux kernel, that would be fine by me. > > Yes. There's a large range between "no tests required at all" (essentially > what we have now) and "an equivalent level of device testing to what you > would carry out before sending your hardware design out to be fabbed into > silicon" (which I hope we'd all agree would be ludicrously high for QEMU); > I'm trying to establish where we're attempting to set our bar. > > I agree that we'd get a lot of bang-for-the-buck out of basic automated > "boot the guest and prod it" testing that covered most of our platforms. > > -- PMM >
On 29 December 2011 21:46, Anthony Liguori <anthony@codemonkey.ws> wrote: > But 0% test coverage is absolutely not enough. So the first step is to get > an infrastructure together that we can all live with and start writing > tests. Agreed. How does your framework deal with non-x86 targets? Finding and installing a working crosscompiler can be painful, especially if you wanted to crosscompile userspace rather than just the kernel... -- PMM
On 12/29/2011 05:04 PM, Peter Maydell wrote: > On 29 December 2011 18:35, Anthony Liguori<anthony@codemonkey.ws> wrote: >> On 12/29/2011 11:49 AM, Peter Maydell wrote: >>> The next obvious question is: are we going to make a serious attempt? >>> (For instance, in a hypothetical tests-required world, would we >>> tell those nice folks from Samsung "no you can't land your >>> Exynos patches unless you write 9000+ lines of test cases" ? >> >> The virtio-serial test case I posted was 50 lines in qemu-test. The >> virtio-serial driver is about ~1500 LOC. That's about 3%. > > This just means it's testing only a fraction of what virtio-serial > actually implements (as you note yourself in the comments for the > test case). For serious coverage you'd also need to cover reading(!), > larger quantities of data than single lines, what happens when one end > feeds in data but the other end isn't reading, vmstate save/load, > multiple simultaneous ports, behaviour on close-and-reopen, > whether it works on bigendian targets, benchmarking to identify > possible performance regressions, etc etc. I think that by the time > you've done all that you'll be closer to 1500 lines than 50. Agreed. There's a reason why virtio-serial testing in kvm autotest is 2.1 k LOC for the test itself and 0.9 k LOC for the auxiliary guest script. Most of the scenarios you've described are covered on kvm autotest version. >> I would expect that we at least have some sort of test that could verify >> that the Exynos platform more or less worked as expected. If that was just >> booting a Linux kernel, that would be fine by me. > > Yes. There's a large range between "no tests required at all" (essentially > what we have now) and "an equivalent level of device testing to what you > would carry out before sending your hardware design out to be fabbed into > silicon" (which I hope we'd all agree would be ludicrously high for QEMU); > I'm trying to establish where we're attempting to set our bar. > > I agree that we'd get a lot of bang-for-the-buck out of basic automated > "boot the guest and prod it" testing that covered most of our platforms. And that sort of automated testing can be done right now with KVM autotest. Granted, I never had time to make it work with ARM, but if people are interested, we are happily accepting patches :)
On 12/29/2011 04:10 PM, Peter Maydell wrote: > On 29 December 2011 21:46, Anthony Liguori<anthony@codemonkey.ws> wrote: >> But 0% test coverage is absolutely not enough. So the first step is to get >> an infrastructure together that we can all live with and start writing >> tests. > > Agreed. > > How does your framework deal with non-x86 targets? http://git.qemu.org/qemu-jeos.git I've already got ppc32 support working. Adding a new arch is just a matter of adding a kernel config and uClibc config to configs/ > Finding and > installing a working crosscompiler can be painful, especially if > you wanted to crosscompile userspace rather than just the kernel... I spent a couple days researching what to do here and ended up settling on: 1) build binutils for desired target 2) build GCC using (1) as a cross compiler. This is a limited form of GCC (no thread support) targeted as uClibc 3) build kernel using (2) and installer headers into our new sysroot 4) build uClibc using (2) and (3) 5) build busybox using (2) and (4) The whole process takes about 30 minutes on my laptop using -j1 which isn't all that bad. It's certainly faster than doing a distro install with TCG :-) qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc, and busybox plus miscellaneous scripts needed to make a working initramfs. Once we get more ARCHs working, I'll add a cronjob to qemu.org to build weekly binary packages so that for most people, all you have to do is grab an RPM/DEB with prebuilt binaries. Regards, Anthony Liguori > > -- PMM >
On 12/29/2011 12:38 PM, Dor Laor wrote: > From the recent threads it looks to me that the 2 advantages of > qemu-test over kvm-autotest are: > > 1. python is not used within the guest ^ Just a (relatively small) subset of KVM autotest tests do require python in the guest (the ones that execute the autotest client inside the guest, virtio-console, and ksm overcommit), so if you don't want py on the guest, just execute the tests that do not require it. For example, we don't rely on any python for the Windows tests. > 2. qemu-test is smaller and simpler > > Except for the above 2, kvm autotest is superior. My main motivation to > merge qemu-test and kvm autotest is that I fear that qemu-test will be > the only requirement for committing stuff for qemu and developers will > be excused from their 'duty' by writing a 50 LOC shell script and assume > they done w/ testing. In addition to that, the 50 LOC will be too basic > and only provide value for basic functionality tests and kvm autotest > folks will need to rewrite all of the matching content and beyond. Requiring people to writing tests for qemu-test is already a improvement over the current situation. It might make people more interested and engaged on test driven development, some of them might want to take the time and learn how to use and develop tests for autotest. We gotta start with something :) > Those above 2 advantages can be solve: > > 1. python is not used within the guest > - One option is just to use the same shell code in kvm autotest w/o > no python code and use the same kernel/initramfs. > - Another way is to use a plain linux distro w/ python and boot it > into single user test mode and potentially use a S4 guest snapshot > or external snapshot to shorten its boot time. > You'll get faster boot time and friendlier code. Yes, slowly we are making autotest and the kvm tests faster by disabling by default some settings that make more sense to be enabled on automated test grids (drop memory caches, take a list of all packages installed on the test machine *between* tests, things like that). Hopefully it'll improve the experience of using it quite a lot. Then, on the virt test side, we can find ways to cut the time to start the functional tests like you suggested (small, pre-installed images, snapshots, you name it). In any case, I believe that qemu-test is positive, since it lowers the barrier and gets people interested in writing tests. As long as we are careful and avoid overlapping features as much as possible, it's fine.
On 12/29/2011 03:02 PM, Anthony Liguori wrote: > On 12/29/2011 10:53 AM, Avi Kivity wrote: >> On 12/29/2011 06:39 PM, Anthony Liguori wrote: >> >> It might have made sense to split the kvm-testing functionality of >> autotest, and have autotest drive that. We could even have called it >> qemu-test. > > I specifically advocated this during Lucas' KVM Forum talk and he was > strongly opposed to it. Ok, this is a point where I have failed in communicating things. I've watched the presentation on video just to see if my memory was not betraying me, and I wouldn't say I was 'strongly opposed', it's just that there are some practical problems, not impossible to overcome of course, but worth thinking. * We rely on a fair bit of autotest libraries/API, so by extracting the APIs we rely on to start a new project we'd be creating some code duplication, and updates/bugfixes on the autotest API wouldn't be easily carried on to the new project. * I'm not sure that this per se would fix the perceived problems with the tests. It'd be nearly the same code, just outside the autotest tree, I fail to see how that would be much better, and therefore, justify the work on doing this. Of course I might be wrong. > I think kvm autotest would get a lot more interest if the test cases > were pulled out of autotest, made more stand alone. They also should be > more Unix like being divided into individual executables with > independent command line options over a single do-everything > configuration file. You have a point with regards to having the test cases more stand alone, but it's not as easy as one would think mainly because of the large amount of user configurable options that we have to pass to the tests. This is a place where the config file format shines, since we just have to override some parameters on different variants to have a test working among Windows and Linux guests, for example, all expressed in a concise way. The config files might look enormous, but they are actually pretty small compared with the amount of test combinations we can get out of it. This is nothing I'd call insurmountable as well. If I get more feedback of people saying "Ok, this would make things better for me and make me more interested", then we'll go that route. I'm sorry if I gave the impression I was strongly opposed to your idea.
On 12/29/2011 05:17 PM, Lucas Meneghel Rodrigues wrote: > On 12/29/2011 03:02 PM, Anthony Liguori wrote: >> On 12/29/2011 10:53 AM, Avi Kivity wrote: >>> On 12/29/2011 06:39 PM, Anthony Liguori wrote: >>> >>> It might have made sense to split the kvm-testing functionality of >>> autotest, and have autotest drive that. We could even have called it >>> qemu-test. >> >> I specifically advocated this during Lucas' KVM Forum talk and he was >> strongly opposed to it. > > Ok, this is a point where I have failed in communicating things. > > I've watched the presentation on video just to see if my memory was not > betraying me, and I wouldn't say I was 'strongly opposed', it's just that there > are some practical problems, not impossible to overcome of course, but worth > thinking. So I decided to do some snooping. Here are some stats: anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py 150 balloon_check.py 68 boot_savevm.py 190 cdrom.py 1875 cgroup.py 111 cpu_hotplug.py 170 enospc.py 71 floppy.py 72 getfd.py 89 hdparm.py 0 __init__.py 615 ksm_overcommit.py 107 migration_multi_host.py 117 migration.py 85 migration_with_file_transfer.py 43 migration_with_reboot.py 138 multi_disk.py 62 nic_bonding.py 104 nic_hotplug.py 60 nmi_watchdog.py 203 pci_hotplug.py 182 physical_resources_check.py 439 qemu_img.py 48 qemu_iotests.py 407 qmp_basic.py 389 qmp_basic_rhel6.py 54 set_link.py 67 smbios_table.py 356 stepmaker.py 247 steps.py 43 stop_continue.py 31 system_reset_bootable.py 181 timedrift.py 96 timedrift_with_migration.py 91 timedrift_with_reboot.py 103 timedrift_with_stop.py 28 unittest_kvmctl.py 121 unittest.py 90 usb.py 2174 virtio_console.py 85 vmstop.py 9562 total anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" kvm | sort -u Amos Kong <akong@redhat.com> Chris Evich <cevich@redhat.com> Cleber Rosa <crosa@redhat.com> Jerry Tang <jtang@suse.com> lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4> Lucas Meneghel Rodrigues <lmr@redhat.com> Lucas Meneghel Rodrigues <lookkas@gmail.com> Lukas Doktor <ldoktor@redhat.com> mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com> pradeep <psuriset@linux.vnet.ibm.com> Qingtang Zhou <qzhou@redhat.com> Thomas Jarosch <thomas.jarosch@intra2net.com> Yiqiao Pu <ypu@redhat.com> Which leads me to the following conclusions: 1) No one outside of autotest developers is actually contributing tests to kvm-autotest. 2) Most of the tests are relatively small and simple enough that they could be trivially ported to a stand alone utility. For instance, looking at the pci_hotplug.py, it just executes monitor commands and does basic pci enumeration in the guest. I don't see a lot of pitfalls here. At the end of the day, we're not talking about all that much code. > > * We rely on a fair bit of autotest libraries/API, so by extracting the APIs we > rely on to start a new project we'd be creating some code duplication, and > updates/bugfixes on the autotest API wouldn't be easily carried on to the new > project. > * I'm not sure that this per se would fix the perceived problems with the tests. > It'd be nearly the same code, just outside the autotest tree, I fail to see how > that would be much better, and therefore, justify the work on doing this. Of > course I might be wrong. It's not about where they live, it's about the ability to execute them in a simple and direct fashion. > >> I think kvm autotest would get a lot more interest if the test cases >> were pulled out of autotest, made more stand alone. They also should be >> more Unix like being divided into individual executables with >> independent command line options over a single do-everything >> configuration file. > > You have a point with regards to having the test cases more stand alone, but > it's not as easy as one would think mainly because of the large amount of user > configurable options that we have to pass to the tests. Configuration options are really the devil when it comes to testing. As a developer, I want my test suite to just run and tell me if I have bugs. I have no interest in configuring it to run a certain way. I don't want to think that much about what I'm testing, I just want the tests to run and tell me if I screwed something up. Regards, Anthony Liguori > This is a place where > the config file format shines, since we just have to override some parameters on > different variants to have a test working among Windows and Linux guests, for > example, all expressed in a concise way. The config files might look enormous, > but they are actually pretty small compared with the amount of test combinations > we can get out of it. This is nothing I'd call insurmountable as well. > > If I get more feedback of people saying "Ok, this would make things better for > me and make me more interested", then we'll go that route. I'm sorry if I gave > the impression I was strongly opposed to your idea. >
On 12/29/2011 10:33 PM, Anthony Liguori wrote: > So I decided to do some snooping. Here are some stats: > > anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py > 150 balloon_check.py > 68 boot_savevm.py > 190 cdrom.py > 1875 cgroup.py > 111 cpu_hotplug.py > 170 enospc.py > 71 floppy.py > 72 getfd.py > 89 hdparm.py > 0 __init__.py > 615 ksm_overcommit.py > 107 migration_multi_host.py > 117 migration.py > 85 migration_with_file_transfer.py > 43 migration_with_reboot.py > 138 multi_disk.py > 62 nic_bonding.py > 104 nic_hotplug.py > 60 nmi_watchdog.py > 203 pci_hotplug.py > 182 physical_resources_check.py > 439 qemu_img.py > 48 qemu_iotests.py > 407 qmp_basic.py > 389 qmp_basic_rhel6.py > 54 set_link.py > 67 smbios_table.py > 356 stepmaker.py > 247 steps.py > 43 stop_continue.py > 31 system_reset_bootable.py > 181 timedrift.py > 96 timedrift_with_migration.py > 91 timedrift_with_reboot.py > 103 timedrift_with_stop.py > 28 unittest_kvmctl.py > 121 unittest.py > 90 usb.py > 2174 virtio_console.py > 85 vmstop.py > 9562 total There are more tests under client/virt/tests: [lmr@freedom tests]$ wc -l *.py 25 autotest.py 100 boot.py 42 build.py 32 clock_getres.py 235 ethtool.py 84 file_transfer.py 53 fillup_disk.py 76 guest_s4.py 80 guest_test.py 45 image_copy.py 0 __init__.py 136 iofuzz.py 31 ioquit.py 40 iozone_windows.py 127 jumbo.py 85 kdump.py 41 linux_s3.py 84 lvm.py 60 mac_change.py 46 module_probe.py 90 multicast.py 106 netperf.py 149 netstress_kill_guest.py 255 nfs_corrupt.py 63 nicdriver_unload.py 39 nic_promisc.py 73 ping.py 29 pxe.py 42 shutdown.py 147 softlockup.py 53 stress_boot.py 164 trans_hugepage_defrag.py 127 trans_hugepage.py 113 trans_hugepage_swapping.py 852 unattended_install.py 175 vlan.py 46 watchdog.py 136 whql_client_install.py 275 whql_submission.py 49 yum_update.py 4405 total Not to mention the infrastructure libraries: [lmr@freedom virt]$ wc -l *.py 1351 aexpect.py 705 base_installer.py 9 common.py 0 __init__.py 195 installer.py 64 installer_unittest.py 240 kvm_installer.py 921 kvm_monitor.py 1635 kvm_vm.py 324 libvirt_monitor.py 1271 libvirt_vm.py 110 passfd.py 237 ppm_utils.py 519 rss_client.py 527 virt_env_process.py 124 virt_http_server.py 51 virt_passfd_setup.py 229 virt_scheduler.py 1401 virt_step_editor.py 131 virt_test.py 315 virt_test_setup.py 813 virt_test_utils.py 3556 virt_utils.py 196 virt_utils_unittest.py 224 virt_video_maker.py 1032 virt_vm.py 16180 total There's a bit more outside this dir, not to mention the auxiliary data files. The support libraries account for roughly for the same number of lines of test code written, so we can say in very rough terms we have around 33 k of python code for kvm/libvirt tests. I agree it is more than an order of magnitude smaller than the qemu code base, but still it demands effort that has to be justified. > anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" > kvm | sort -u > > Amos Kong <akong@redhat.com> > Chris Evich <cevich@redhat.com> > Cleber Rosa <crosa@redhat.com> > Jerry Tang <jtang@suse.com> > lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4> > Lucas Meneghel Rodrigues <lmr@redhat.com> > Lucas Meneghel Rodrigues <lookkas@gmail.com> > Lukas Doktor <ldoktor@redhat.com> > mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> > Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com> > pradeep <psuriset@linux.vnet.ibm.com> > Qingtang Zhou <qzhou@redhat.com> > Thomas Jarosch <thomas.jarosch@intra2net.com> > Yiqiao Pu <ypu@redhat.com> > > Which leads me to the following conclusions: > > 1) No one outside of autotest developers is actually contributing tests > to kvm-autotest. Autotest was maintained using subversion until recently, all commits from people outside autotest were commited by autotest developers, the git tree was just a mirror until recently. Grepping for Signed-off-by: lines will give a much better idea of the contributors: [lmr@freedom autotest.git]$ git log client/virt | grep "Signed-off-by:" | sort -u Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com> Signed-off-by: Alon Levy <alevy@redhat.com> Signed-off-by: Amos Kong <akong@redhat.com> Signed-off-by: Chen Cao <kcao@redhat.com> Signed-off-by: Chris Evich <cevich@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com> Signed-off-by: Feng Yang <fyang@redhat.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> Signed-off-by: Jason D. Gaston <jason.d.gaston@intel.com> Signed-off-by: Jason Wang <jasowang@redhat.com> Signed-off-by: Jiri Zupka <jzupka@redhat.com> Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> Signed-off-by: Lukas Doktor <ldoktor@redhat.com> Signed-off-by: Martin Jenner <mjenner@redhat.com> Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com> Signed-off-by: Pradeep Kumar Surisetty <psuriset@linux.vnet.ibm.com> Signed-off-by: Qingtang Zhou <qzhou@redhat.com> Signed-off-by: Quentin Deldycke <quentindeldycke@gmail.com> Signed-off-by: Ray Chauvin <rchauvin@redhat.com> Signed-off-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> Signed-off-by: Suqin Huang <shuang@redhat.com> Signed-off-by: Wei Yang <weiyang@linux.vnet.ibm.com> Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com> Signed-off-by: Yiqiao Pu <ypu@redhat.com> Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com> Signed-off-by: Yolkfull Chow <yzhou@redhat.com> [lmr@freedom autotest.git]$ git log client/tests/kvm | grep "Signed-off-by:" | sort -u Signed-off-by: Alon Levy <alevy@redhat.com> Signed-off-by: Amos Kong <akong@redhat.com> Signed-off-by: Avi Kivity <avi@redhat.com> Signed-off-by: Bear Yang <byang@redhat.com> Signed-off-by: Cao, Chen <kcao@redhat.com> Signed-off-by: Chen Cao <kcao@redhat.com> Signed-off-by: Chris Evich <cevich@redhat.com> Signed-off-by: Cleber Rosa <crosa@redhat.com> Signed-off-by: David Huff <dhuff@redhat.com> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> Signed-off-by: Feng Yang <fyang@redhat.com> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> Signed-off-by: Jason Wang <jasowang@redhat.com> Signed-off-by: Jerry Tang <jtang@suse.com> Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com> Signed-off-by: Jin Bing Guo <guojb@cn.ibm.com> Signed-off-by: Jiri Zupka <jzupka@redhat.com> Signed-off-by: Ken Cao <kcao@redhat.com> Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com> Signed-off-by: Lukas Doktor <ldoktor@redhat.com> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> Signed-off-by: Martin Jenner <mjenner@redhat.com> Signed-off-by: Michael Goldish <mgoldish@redhat.com> Signed-off-by: Michael S. Tsirkin <mst@redhat.com> Signed-off-by: Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com> Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com> Signed-off-by: Qingtang Zhou <qzhou@redhat.com> Signed-off-by: Ryan Harper <ryanh@us.ibm.com> Signed-off-by: Shuxi Shang <sshang@redhat.com> Signed-off-by: Sudhir Kumar <skumar@linux.vnet.ibm.com> Signed-off-by: Supriya Kannery <supriyak@in.ibm.com> Signed-off-by: Suqin Huang <shuang@redhat.com> Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com> Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com> Signed-off-by: Uri Lublin <uril@redhat.com> Signed-off-by: Yiqiao Pu <ypu@redhat.com> Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com> Signed-off-by: Yolkfull Chow <yzhou@redhat.com> So, not quite what you wanted to demonstrate. > 2) Most of the tests are relatively small and simple enough that they > could be trivially ported to a stand alone utility. For instance, > looking at the pci_hotplug.py, it just executes monitor commands and > does basic pci enumeration in the guest. They do depend on infrastructure code to be simple and small. The infrastructure code was written using autotest libraries, so although it is possible to make it stand alone, still there's the problem of having these library/classes out of sync with autotest evolving. > I don't see a lot of pitfalls here. At the end of the day, we're not > talking about all that much code. Even so (you see it's not so small), it demands effort that has to be well justified, as I pointed out. I'm starting to believe it might be worth the effort. > It's not about where they live, it's about the ability to execute them > in a simple and direct fashion. Ok, fair enough, we'll see what we can do, although I wish it were a simpler problem, see below. >> You have a point with regards to having the test cases more stand >> alone, but >> it's not as easy as one would think mainly because of the large amount >> of user >> configurable options that we have to pass to the tests. > > Configuration options are really the devil when it comes to testing. As > a developer, I want my test suite to just run and tell me if I have > bugs. I have no interest in configuring it to run a certain way. I don't > want to think that much about what I'm testing, I just want the tests to > run and tell me if I screwed something up. Still, you need to tell your tests, say, the format of the terminal prompt for Fedora might be slightly different of the [insert-distro-here], that the src and destination paths on a file transfer test are different for Windows and Linux guests, that the output of a given command when you are running a test with a virtio card differs from the output when you are using rtl8139, that you want to build qemu from git repo X, or from brew build Y, using a custom version of seabios, and so on and so forth.
On 12/29/2011 10:20 PM, Lucas Meneghel Rodrigues wrote: > On 12/29/2011 10:33 PM, Anthony Liguori wrote: >> So I decided to do some snooping. Here are some stats: >> >> anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py >> 150 balloon_check.py >> 68 boot_savevm.py >> 190 cdrom.py >> 1875 cgroup.py >> 111 cpu_hotplug.py >> 170 enospc.py >> 71 floppy.py >> 72 getfd.py >> 89 hdparm.py >> 0 __init__.py >> 615 ksm_overcommit.py >> 107 migration_multi_host.py >> 117 migration.py >> 85 migration_with_file_transfer.py >> 43 migration_with_reboot.py >> 138 multi_disk.py >> 62 nic_bonding.py >> 104 nic_hotplug.py >> 60 nmi_watchdog.py >> 203 pci_hotplug.py >> 182 physical_resources_check.py >> 439 qemu_img.py >> 48 qemu_iotests.py >> 407 qmp_basic.py >> 389 qmp_basic_rhel6.py >> 54 set_link.py >> 67 smbios_table.py >> 356 stepmaker.py >> 247 steps.py >> 43 stop_continue.py >> 31 system_reset_bootable.py >> 181 timedrift.py >> 96 timedrift_with_migration.py >> 91 timedrift_with_reboot.py >> 103 timedrift_with_stop.py >> 28 unittest_kvmctl.py >> 121 unittest.py >> 90 usb.py >> 2174 virtio_console.py >> 85 vmstop.py >> 9562 total > > There are more tests under client/virt/tests: > > [lmr@freedom tests]$ wc -l *.py > 25 autotest.py > 100 boot.py > 42 build.py > 32 clock_getres.py > 235 ethtool.py > 84 file_transfer.py > 53 fillup_disk.py > 76 guest_s4.py > 80 guest_test.py > 45 image_copy.py > 0 __init__.py > 136 iofuzz.py > 31 ioquit.py > 40 iozone_windows.py > 127 jumbo.py > 85 kdump.py > 41 linux_s3.py > 84 lvm.py > 60 mac_change.py > 46 module_probe.py > 90 multicast.py > 106 netperf.py > 149 netstress_kill_guest.py > 255 nfs_corrupt.py > 63 nicdriver_unload.py > 39 nic_promisc.py > 73 ping.py > 29 pxe.py > 42 shutdown.py > 147 softlockup.py > 53 stress_boot.py > 164 trans_hugepage_defrag.py > 127 trans_hugepage.py > 113 trans_hugepage_swapping.py > 852 unattended_install.py > 175 vlan.py > 46 watchdog.py > 136 whql_client_install.py > 275 whql_submission.py > 49 yum_update.py > 4405 total > > Not to mention the infrastructure libraries: > > [lmr@freedom virt]$ wc -l *.py > 1351 aexpect.py > 705 base_installer.py > 9 common.py > 0 __init__.py > 195 installer.py > 64 installer_unittest.py > 240 kvm_installer.py > 921 kvm_monitor.py > 1635 kvm_vm.py > 324 libvirt_monitor.py > 1271 libvirt_vm.py > 110 passfd.py > 237 ppm_utils.py > 519 rss_client.py > 527 virt_env_process.py > 124 virt_http_server.py > 51 virt_passfd_setup.py > 229 virt_scheduler.py > 1401 virt_step_editor.py > 131 virt_test.py > 315 virt_test_setup.py > 813 virt_test_utils.py > 3556 virt_utils.py > 196 virt_utils_unittest.py > 224 virt_video_maker.py > 1032 virt_vm.py > 16180 total > > There's a bit more outside this dir, not to mention the auxiliary data > files. The support libraries account for roughly for the same number > of lines of test code written, so we can say in very rough terms we > have around 33 k of python code for kvm/libvirt tests. I agree it is > more than an order of magnitude smaller than the qemu code base, but > still it demands effort that has to be justified. > >> anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" >> kvm | sort -u >> >> Amos Kong <akong@redhat.com> >> Chris Evich <cevich@redhat.com> >> Cleber Rosa <crosa@redhat.com> >> Jerry Tang <jtang@suse.com> >> lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4> >> Lucas Meneghel Rodrigues <lmr@redhat.com> >> Lucas Meneghel Rodrigues <lookkas@gmail.com> >> Lukas Doktor <ldoktor@redhat.com> >> mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> >> Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com> >> pradeep <psuriset@linux.vnet.ibm.com> >> Qingtang Zhou <qzhou@redhat.com> >> Thomas Jarosch <thomas.jarosch@intra2net.com> >> Yiqiao Pu <ypu@redhat.com> >> >> Which leads me to the following conclusions: >> >> 1) No one outside of autotest developers is actually contributing tests >> to kvm-autotest. > > Autotest was maintained using subversion until recently, all commits > from people outside autotest were commited by autotest developers, the > git tree was just a mirror until recently. Grepping for Signed-off-by: > lines will give a much better idea of the contributors: > > [lmr@freedom autotest.git]$ git log client/virt | grep > "Signed-off-by:" | sort -u > Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com> > Signed-off-by: Alon Levy <alevy@redhat.com> > Signed-off-by: Amos Kong <akong@redhat.com> > Signed-off-by: Chen Cao <kcao@redhat.com> > Signed-off-by: Chris Evich <cevich@redhat.com> > Signed-off-by: Cleber Rosa <crosa@redhat.com> > Signed-off-by: Feng Yang <fyang@redhat.com> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> > Signed-off-by: Jason D. Gaston <jason.d.gaston@intel.com> > Signed-off-by: Jason Wang <jasowang@redhat.com> > Signed-off-by: Jiri Zupka <jzupka@redhat.com> > Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> > Signed-off-by: Lukas Doktor <ldoktor@redhat.com> > Signed-off-by: Martin Jenner <mjenner@redhat.com> > Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com> > Signed-off-by: Pradeep Kumar Surisetty <psuriset@linux.vnet.ibm.com> > Signed-off-by: Qingtang Zhou <qzhou@redhat.com> > Signed-off-by: Quentin Deldycke <quentindeldycke@gmail.com> > Signed-off-by: Ray Chauvin <rchauvin@redhat.com> > Signed-off-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> > Signed-off-by: Suqin Huang <shuang@redhat.com> > Signed-off-by: Wei Yang <weiyang@linux.vnet.ibm.com> > Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com> > Signed-off-by: Yiqiao Pu <ypu@redhat.com> > Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com> > Signed-off-by: Yolkfull Chow <yzhou@redhat.com> > > [lmr@freedom autotest.git]$ git log client/tests/kvm | grep > "Signed-off-by:" | sort -u > Signed-off-by: Alon Levy <alevy@redhat.com> > Signed-off-by: Amos Kong <akong@redhat.com> > Signed-off-by: Avi Kivity <avi@redhat.com> > Signed-off-by: Bear Yang <byang@redhat.com> > Signed-off-by: Cao, Chen <kcao@redhat.com> > Signed-off-by: Chen Cao <kcao@redhat.com> > Signed-off-by: Chris Evich <cevich@redhat.com> > Signed-off-by: Cleber Rosa <crosa@redhat.com> > Signed-off-by: David Huff <dhuff@redhat.com> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> > Signed-off-by: Feng Yang <fyang@redhat.com> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> > Signed-off-by: Jason Wang <jasowang@redhat.com> > Signed-off-by: Jerry Tang <jtang@suse.com> > Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com> > Signed-off-by: Jin Bing Guo <guojb@cn.ibm.com> > Signed-off-by: Jiri Zupka <jzupka@redhat.com> > Signed-off-by: Ken Cao <kcao@redhat.com> > Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com> > Signed-off-by: Lukas Doktor <ldoktor@redhat.com> > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> > Signed-off-by: Martin Jenner <mjenner@redhat.com> > Signed-off-by: Michael Goldish <mgoldish@redhat.com> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > Signed-off-by: Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com> > Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com> > Signed-off-by: Qingtang Zhou <qzhou@redhat.com> > Signed-off-by: Ryan Harper <ryanh@us.ibm.com> > Signed-off-by: Shuxi Shang <sshang@redhat.com> > Signed-off-by: Sudhir Kumar <skumar@linux.vnet.ibm.com> > Signed-off-by: Supriya Kannery <supriyak@in.ibm.com> > Signed-off-by: Suqin Huang <shuang@redhat.com> > Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com> > Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com> > Signed-off-by: Uri Lublin <uril@redhat.com> > Signed-off-by: Yiqiao Pu <ypu@redhat.com> > Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com> > Signed-off-by: Yolkfull Chow <yzhou@redhat.com> > > So, not quite what you wanted to demonstrate. > >> 2) Most of the tests are relatively small and simple enough that they >> could be trivially ported to a stand alone utility. For instance, >> looking at the pci_hotplug.py, it just executes monitor commands and >> does basic pci enumeration in the guest. > > They do depend on infrastructure code to be simple and small. The > infrastructure code was written using autotest libraries, so although > it is possible to make it stand alone, still there's the problem of > having these library/classes out of sync with autotest evolving. > >> I don't see a lot of pitfalls here. At the end of the day, we're not >> talking about all that much code. > > Even so (you see it's not so small), it demands effort that has to be > well justified, as I pointed out. I'm starting to believe it might be > worth the effort. > >> It's not about where they live, it's about the ability to execute them >> in a simple and direct fashion. > > Ok, fair enough, we'll see what we can do, although I wish it were a > simpler problem, see below. > >>> You have a point with regards to having the test cases more stand >>> alone, but >>> it's not as easy as one would think mainly because of the large amount >>> of user >>> configurable options that we have to pass to the tests. >> >> Configuration options are really the devil when it comes to testing. As >> a developer, I want my test suite to just run and tell me if I have >> bugs. I have no interest in configuring it to run a certain way. I don't >> want to think that much about what I'm testing, I just want the tests to >> run and tell me if I screwed something up. > > Still, you need to tell your tests, say, the format of the terminal > prompt for Fedora might be slightly different of the > [insert-distro-here], that the src and destination paths on a file > transfer test are different for Windows and Linux guests, that the > output of a given command when you are running a test with a virtio > card differs from the output when you are using rtl8139, that you want > to build qemu from git repo X, or from brew build Y, using a custom > version of seabios, and so on and so forth. This can be solved in a pretty easy way. We can have (kvm autotest) config files suited for developers, together with a helper script that will run the tests (or test set) given on the command line. Ideally this would live inside the qemu git repo, together with a make target. This config could be targeted to a DSL(-like) guest image that can be downloaded, instead of installed. A ~50...100MB download would be hard to even notice. Maybe could even be copied pristine from backup between each test run. Of course the maintenance of these config files and guest images would be on us.
Am 29.12.2011 19:33, schrieb Anthony Liguori: > I don't think we should focus much on qtest for non-x86 targets. I > mean, if you are interested in it for ARM, fantastic, but I don't think > we would mandate it. I'm actually very interested in having a qtest framework for non-x86 for a) unit testing TCG (execute instruction, check register/memory results) b) unit testing device emulation Your v2 qtest RFC is still missing the libqtest C implementation. We code in C so I'd like to test b) in C, not Python or some random other language that someone happens to like for whatever reason. Andreas
Am 29.12.2011 23:30, schrieb Anthony Liguori: > On 12/29/2011 04:10 PM, Peter Maydell wrote: >> How does your framework deal with non-x86 targets? > > http://git.qemu.org/qemu-jeos.git > > I've already got ppc32 support working. Adding a new arch is just a > matter of adding a kernel config and uClibc config to configs/ > >> Finding and >> installing a working crosscompiler can be painful, especially if >> you wanted to crosscompile userspace rather than just the kernel... > > I spent a couple days researching what to do here and ended up settling on: > > 1) build binutils for desired target > > 2) build GCC using (1) as a cross compiler. This is a limited form of > GCC (no thread support) targeted as uClibc > > 3) build kernel using (2) and installer headers into our new sysroot > > 4) build uClibc using (2) and (3) > > 5) build busybox using (2) and (4) > > The whole process takes about 30 minutes on my laptop using -j1 which > isn't all that bad. It's certainly faster than doing a distro install > with TCG :-) > > qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc, > and busybox plus miscellaneous scripts needed to make a working initramfs. "One ring to rule them all ... and in the darkness bind them"? ;) Seriously, what you describe may work for mainstream targets. But for new targets the trouble starts with 1) already: which binutils? The latest stable may not work for all architectures yet, so a generic submodule approach is doomed to fail. Will uClibc work for all targets? There's glibc, eglibc, newlib, ... Not all targets have a softmmu at this time: unicore32 (Fwiw this also means no testing outside Linux hosts.) There's no requirement that the Linux kernel must have been ported to a QEMU target yet or that it is still in mainline or that what is in mainline is complete enough for our testing. So, I'm fine if you come up with a cool testing framework, with or without Q in the name. Just please don't imply from testing two targets that this is a one-size-fits-all solution for all current and future targets. The qtest approach seems more promising in that regard. > Once we get more ARCHs working, I'll add a cronjob to qemu.org to build > weekly binary packages so that for most people, all you have to do is > grab an RPM/DEB with prebuilt binaries. I have a feeling OBS, Koji, etc. are more suited for this than trying to set up a build service of your own. Let's leave qrpm for April 1st. Regards, Andreas
On 12/29/2011 11:10 PM, Peter Maydell wrote: > > > Even if that turns out to be the case, it's fine. Better to > > have a few good devices than dozens of bad ones. > > This is easier to say on the x86 side of the fence, since > most of the devices you need are already in the codebase and > nobody is going to throw them out again if tests don't get > written for them :-) > > There are lots of things where I'd rather have a "tested by > booting a guest OS" implementation than none at all (like audio > support for the versatile/etc boards, which went in recently) or > TrustZone support (not in yet but may be along later). At least > then we have something that works for most people and something > we can fix bugs in, rather than a gaping hole in capability. > We can have different criteria for different parts of the tree. Undesirable, but tradeoffs have to be made.
On 12/29/2011 07:33 PM, Anthony Liguori wrote: > > FWIW, I expect virtio-scsi to be the first guinea pig here... I believe > Stefan has already started looking at writing some qtest based tests for > virtio-scsi. I'm curious, because a SCSI HBA makes it as hard as it can be to write unit tests. Pretty much everything you do will test the interactions between the SCSI devices and the block layer, more than the HBA itself. This is especially for such a simple HBA; unit testing is much more "interesting" for something like the LSI device. So, to test the HBA you need write either a mock SCSI device, or a mock block device. (I hate mocks). Now that the client is asynchronous, NBD is a pretty effective way to write a mock block device, though not the simplest. Paolo
On Mon, Jan 02, 2012 at 03:07:08PM +0100, Paolo Bonzini wrote: > On 12/29/2011 07:33 PM, Anthony Liguori wrote: > > > >FWIW, I expect virtio-scsi to be the first guinea pig here... I believe > >Stefan has already started looking at writing some qtest based tests for > >virtio-scsi. > > I'm curious, because a SCSI HBA makes it as hard as it can be to > write unit tests. Pretty much everything you do will test the > interactions between the SCSI devices and the block layer, more than > the HBA itself. This is especially for such a simple HBA; unit > testing is much more "interesting" for something like the LSI > device. > > So, to test the HBA you need write either a mock SCSI device, or a > mock block device. (I hate mocks). Now that the client is > asynchronous, NBD is a pretty effective way to write a mock block > device, though not the simplest. I haven't started yet but I am planning to write qtest virtio PCI tests and will definitely be taking a look at virtio-scsi. We don't necessarily need mock devices, just really simply temporary raw files or similar. On the other hand mock devices can be a good solution for programmatically verifying that our intended action was performed on the host device. Stefan
On 01/03/2012 09:19 AM, Stefan Hajnoczi wrote: > I haven't started yet but I am planning to write qtest virtio PCI tests > and will definitely be taking a look at virtio-scsi. > > We don't necessarily need mock devices, just really simply temporary raw > files or similar. On the other hand mock devices can be a good solution > for programmatically verifying that our intended action was performed on > the host device. Mocks, no matter whether they're mock SCSI devices or mock block devices, are needed if you want to test TMFs. Cancellation was sorta broken by the coroutine conversion, but I think it can be fixed. Paolo
On 12/30/2011 09:43 AM, Andreas Färber wrote: > Am 29.12.2011 23:30, schrieb Anthony Liguori: >> On 12/29/2011 04:10 PM, Peter Maydell wrote: >>> How does your framework deal with non-x86 targets? >> >> http://git.qemu.org/qemu-jeos.git >> >> I've already got ppc32 support working. Adding a new arch is just a >> matter of adding a kernel config and uClibc config to configs/ >> >>> Finding and >>> installing a working crosscompiler can be painful, especially if >>> you wanted to crosscompile userspace rather than just the kernel... >> >> I spent a couple days researching what to do here and ended up settling on: >> >> 1) build binutils for desired target >> >> 2) build GCC using (1) as a cross compiler. This is a limited form of >> GCC (no thread support) targeted as uClibc >> >> 3) build kernel using (2) and installer headers into our new sysroot >> >> 4) build uClibc using (2) and (3) >> >> 5) build busybox using (2) and (4) >> >> The whole process takes about 30 minutes on my laptop using -j1 which >> isn't all that bad. It's certainly faster than doing a distro install >> with TCG :-) >> >> qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc, >> and busybox plus miscellaneous scripts needed to make a working initramfs. > > "One ring to rule them all ... and in the darkness bind them"? ;) > > Seriously, what you describe may work for mainstream targets. But for > new targets the trouble starts with 1) already: which binutils? The > latest stable may not work for all architectures yet, so a generic > submodule approach is doomed to fail. Let's not speculate here.. There's already an existence proof that this approach works for most of the targets we support (Aboriginal Linux). > Will uClibc work for all targets? There's glibc, eglibc, newlib, ... > > Not all targets have a softmmu at this time: unicore32 > (Fwiw this also means no testing outside Linux hosts.) > > There's no requirement that the Linux kernel must have been ported to a > QEMU target yet or that it is still in mainline or that what is in > mainline is complete enough for our testing. Yes, but this just becomes the exception to prove the rule. Regards, Anthony Liguori
On 12/29/2011 07:20 PM, Lucas Meneghel Rodrigues wrote: > On 12/29/2011 10:33 PM, Anthony Liguori wrote: >> So I decided to do some snooping. Here are some stats: >> >> anthony@titi:~/git/autotest/client/tests/kvm/tests$ wc -l *.py >> 150 balloon_check.py >> 68 boot_savevm.py >> 190 cdrom.py >> 1875 cgroup.py >> 111 cpu_hotplug.py >> 170 enospc.py >> 71 floppy.py >> 72 getfd.py >> 89 hdparm.py >> 0 __init__.py >> 615 ksm_overcommit.py >> 107 migration_multi_host.py >> 117 migration.py >> 85 migration_with_file_transfer.py >> 43 migration_with_reboot.py >> 138 multi_disk.py >> 62 nic_bonding.py >> 104 nic_hotplug.py >> 60 nmi_watchdog.py >> 203 pci_hotplug.py >> 182 physical_resources_check.py >> 439 qemu_img.py >> 48 qemu_iotests.py >> 407 qmp_basic.py >> 389 qmp_basic_rhel6.py >> 54 set_link.py >> 67 smbios_table.py >> 356 stepmaker.py >> 247 steps.py >> 43 stop_continue.py >> 31 system_reset_bootable.py >> 181 timedrift.py >> 96 timedrift_with_migration.py >> 91 timedrift_with_reboot.py >> 103 timedrift_with_stop.py >> 28 unittest_kvmctl.py >> 121 unittest.py >> 90 usb.py >> 2174 virtio_console.py >> 85 vmstop.py >> 9562 total > > There are more tests under client/virt/tests: > > [lmr@freedom tests]$ wc -l *.py > 25 autotest.py > 100 boot.py > 42 build.py > 32 clock_getres.py > 235 ethtool.py > 84 file_transfer.py > 53 fillup_disk.py > 76 guest_s4.py > 80 guest_test.py > 45 image_copy.py > 0 __init__.py > 136 iofuzz.py > 31 ioquit.py > 40 iozone_windows.py > 127 jumbo.py > 85 kdump.py > 41 linux_s3.py > 84 lvm.py > 60 mac_change.py > 46 module_probe.py > 90 multicast.py > 106 netperf.py > 149 netstress_kill_guest.py > 255 nfs_corrupt.py > 63 nicdriver_unload.py > 39 nic_promisc.py > 73 ping.py > 29 pxe.py > 42 shutdown.py > 147 softlockup.py > 53 stress_boot.py > 164 trans_hugepage_defrag.py > 127 trans_hugepage.py > 113 trans_hugepage_swapping.py > 852 unattended_install.py > 175 vlan.py > 46 watchdog.py > 136 whql_client_install.py > 275 whql_submission.py > 49 yum_update.py > 4405 total Hrm, what's the significance of the two separate directories? At any rate, my point still stands. 13k LOC is still small enough that making large changes to the infrastructure shouldn't be that big of a deal. I think the important observation is that each test is pretty small which means they can be reasonably moved 1-by-1 to a new stand alone infrastructure. > Not to mention the infrastructure libraries: > > [lmr@freedom virt]$ wc -l *.py > 1351 aexpect.py > 705 base_installer.py > 9 common.py > 0 __init__.py > 195 installer.py > 64 installer_unittest.py > 240 kvm_installer.py > 921 kvm_monitor.py > 1635 kvm_vm.py > 324 libvirt_monitor.py > 1271 libvirt_vm.py > 110 passfd.py > 237 ppm_utils.py > 519 rss_client.py > 527 virt_env_process.py > 124 virt_http_server.py > 51 virt_passfd_setup.py > 229 virt_scheduler.py > 1401 virt_step_editor.py > 131 virt_test.py > 315 virt_test_setup.py > 813 virt_test_utils.py > 3556 virt_utils.py > 196 virt_utils_unittest.py > 224 virt_video_maker.py > 1032 virt_vm.py > 16180 total > > There's a bit more outside this dir, not to mention the auxiliary data files. > The support libraries account for roughly for the same number of lines of test > code written, so we can say in very rough terms we have around 33 k of python > code for kvm/libvirt tests. I agree it is more than an order of magnitude > smaller than the qemu code base, but still it demands effort that has to be > justified. > >> anthony@titi:~/git/autotest/client/tests$ git log --format="%an <%ae>" >> kvm | sort -u >> >> Amos Kong <akong@redhat.com> >> Chris Evich <cevich@redhat.com> >> Cleber Rosa <crosa@redhat.com> >> Jerry Tang <jtang@suse.com> >> lmr <lmr@592f7852-d20e-0410-864c-8624ca9c26a4> >> Lucas Meneghel Rodrigues <lmr@redhat.com> >> Lucas Meneghel Rodrigues <lookkas@gmail.com> >> Lukas Doktor <ldoktor@redhat.com> >> mbligh <mbligh@592f7852-d20e-0410-864c-8624ca9c26a4> >> Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com> >> pradeep <psuriset@linux.vnet.ibm.com> >> Qingtang Zhou <qzhou@redhat.com> >> Thomas Jarosch <thomas.jarosch@intra2net.com> >> Yiqiao Pu <ypu@redhat.com> >> >> Which leads me to the following conclusions: >> >> 1) No one outside of autotest developers is actually contributing tests >> to kvm-autotest. > > Autotest was maintained using subversion until recently, all commits from people > outside autotest were commited by autotest developers, the git tree was just a > mirror until recently. Grepping for Signed-off-by: lines will give a much better > idea of the contributors: > > [lmr@freedom autotest.git]$ git log client/virt | grep "Signed-off-by:" | sort -u > Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com> > Signed-off-by: Alon Levy <alevy@redhat.com> > Signed-off-by: Amos Kong <akong@redhat.com> > Signed-off-by: Chen Cao <kcao@redhat.com> > Signed-off-by: Chris Evich <cevich@redhat.com> > Signed-off-by: Cleber Rosa <crosa@redhat.com> > Signed-off-by: Feng Yang <fyang@redhat.com> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> > Signed-off-by: Jason D. Gaston <jason.d.gaston@intel.com> > Signed-off-by: Jason Wang <jasowang@redhat.com> > Signed-off-by: Jiri Zupka <jzupka@redhat.com> > Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com> > Signed-off-by: Lukas Doktor <ldoktor@redhat.com> > Signed-off-by: Martin Jenner <mjenner@redhat.com> > Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com> > Signed-off-by: Pradeep Kumar Surisetty <psuriset@linux.vnet.ibm.com> > Signed-off-by: Qingtang Zhou <qzhou@redhat.com> > Signed-off-by: Quentin Deldycke <quentindeldycke@gmail.com> > Signed-off-by: Ray Chauvin <rchauvin@redhat.com> > Signed-off-by: Satheesh Rajendran <sathnaga@linux.vnet.ibm.com> > Signed-off-by: Suqin Huang <shuang@redhat.com> > Signed-off-by: Wei Yang <weiyang@linux.vnet.ibm.com> > Signed-off-by: Xu He Jie <xuhj@linux.vnet.ibm.com> > Signed-off-by: Yiqiao Pu <ypu@redhat.com> > Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com> > Signed-off-by: Yolkfull Chow <yzhou@redhat.com> > > [lmr@freedom autotest.git]$ git log client/tests/kvm | grep "Signed-off-by:" | > sort -u > Signed-off-by: Alon Levy <alevy@redhat.com> > Signed-off-by: Amos Kong <akong@redhat.com> > Signed-off-by: Avi Kivity <avi@redhat.com> > Signed-off-by: Bear Yang <byang@redhat.com> > Signed-off-by: Cao, Chen <kcao@redhat.com> > Signed-off-by: Chen Cao <kcao@redhat.com> > Signed-off-by: Chris Evich <cevich@redhat.com> > Signed-off-by: Cleber Rosa <crosa@redhat.com> > Signed-off-by: David Huff <dhuff@redhat.com> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com> > Signed-off-by: Feng Yang <fyang@redhat.com> > Signed-off-by: Gerd Hoffmann <kraxel@redhat.com> > Signed-off-by: Jason Wang <jasowang@redhat.com> > Signed-off-by: Jerry Tang <jtang@suse.com> > Signed-off-by: Jes Sorensen <Jes.Sorensen@redhat.com> > Signed-off-by: Jin Bing Guo <guojb@cn.ibm.com> > Signed-off-by: Jiri Zupka <jzupka@redhat.com> > Signed-off-by: Ken Cao <kcao@redhat.com> > Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com > Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com> > Signed-off-by: Lukas Doktor <ldoktor@redhat.com> > Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com> > Signed-off-by: Martin Jenner <mjenner@redhat.com> > Signed-off-by: Michael Goldish <mgoldish@redhat.com> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com> > Signed-off-by: Onkar N Mahajan <onkar.n.mahajan@linux.vnet.ibm.com> > Signed-off-by: Pradeep K Surisetty <psuriset@linux.vnet.ibm.com> > Signed-off-by: Qingtang Zhou <qzhou@redhat.com> > Signed-off-by: Ryan Harper <ryanh@us.ibm.com> > Signed-off-by: Shuxi Shang <sshang@redhat.com> > Signed-off-by: Sudhir Kumar <skumar@linux.vnet.ibm.com> > Signed-off-by: Supriya Kannery <supriyak@in.ibm.com> > Signed-off-by: Suqin Huang <shuang@redhat.com> > Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com> > Signed-off-by: Thomas Jarosch <thomas.jarosch@intra2net.com> > Signed-off-by: Uri Lublin <uril@redhat.com> > Signed-off-by: Yiqiao Pu <ypu@redhat.com> > Signed-off-by: Yogananth Subramanian <anantyog@linux.vnet.ibm.com> > Signed-off-by: Yolkfull Chow <yzhou@redhat.com> > > So, not quite what you wanted to demonstrate. Well it actually still is. There isn't a tremendous overlap between that list and the list of QEMU contributors. >> 2) Most of the tests are relatively small and simple enough that they >> could be trivially ported to a stand alone utility. For instance, >> looking at the pci_hotplug.py, it just executes monitor commands and >> does basic pci enumeration in the guest. > > They do depend on infrastructure code to be simple and small. The infrastructure > code was written using autotest libraries, so although it is possible to make it > stand alone, still there's the problem of having these library/classes out of > sync with autotest evolving. > >> I don't see a lot of pitfalls here. At the end of the day, we're not >> talking about all that much code. > > Even so (you see it's not so small), it demands effort that has to be well > justified, as I pointed out. I'm starting to believe it might be worth the effort. There are two discussions tied into one here so just to summarize: 1) I think we can get more people working on kvm autotest and by extracting and making them stand alone. We can potentially even start merging tests into QEMU getting more developers working on kvm autotest tests and mandating new ones for certain types of features. 2) kvm autotest focuses on producing guest-OS agnostic tests which require infrastructure (Python, ssh, etc.). qemu-test essentially uses Linux as a libOS for writing unit tests. (1) does not change the need for qemu-test IMHO. >> It's not about where they live, it's about the ability to execute them >> in a simple and direct fashion. > > Ok, fair enough, we'll see what we can do, although I wish it were a simpler > problem, see below. > >>> You have a point with regards to having the test cases more stand >>> alone, but >>> it's not as easy as one would think mainly because of the large amount >>> of user >>> configurable options that we have to pass to the tests. >> >> Configuration options are really the devil when it comes to testing. As >> a developer, I want my test suite to just run and tell me if I have >> bugs. I have no interest in configuring it to run a certain way. I don't >> want to think that much about what I'm testing, I just want the tests to >> run and tell me if I screwed something up. > > Still, you need to tell your tests, say, the format of the terminal prompt for > Fedora might be slightly different of the [insert-distro-here], that the src and > destination paths on a file transfer test are different for Windows and Linux > guests, There are various clever things we can do. Essentially, the problem is that we want to deliver a test payload to a guest, right? Why not create a simple PCI device and expose the payload through the bar or something? There's no need to screen scrap a terminal session when we can do something more direct. > that the output of a given command when you are running a test with a > virtio card differs from the output when you are using rtl8139, that you want to > build qemu from git repo X, or from brew build Y, using a custom version of > seabios, and so on and so forth. Building qemu should not be tied into the tests themselves. That's useful for regression testing but that just complicates things for development testing. The building part is somethign that probably should continue to live in autotest. Regards, Anthony Liguori >
Am 03.01.2012 14:42, schrieb Anthony Liguori: > On 12/30/2011 09:43 AM, Andreas Färber wrote: >> Am 29.12.2011 23:30, schrieb Anthony Liguori: >>> 1) build binutils for desired target >>> >>> 2) build GCC using (1) as a cross compiler. This is a limited form of >>> GCC (no thread support) targeted as uClibc >>> >>> 3) build kernel using (2) and installer headers into our new sysroot >>> >>> 4) build uClibc using (2) and (3) >>> >>> 5) build busybox using (2) and (4) >>> >>> The whole process takes about 30 minutes on my laptop using -j1 which >>> isn't all that bad. It's certainly faster than doing a distro install >>> with TCG :-) >>> >>> qemu-jeos.git contains git submodules for binutils, GCC, linux, uClibc, >>> and busybox plus miscellaneous scripts needed to make a working >>> initramfs. >> >> "One ring to rule them all ... and in the darkness bind them"? ;) >> >> Seriously, what you describe may work for mainstream targets. But for >> new targets the trouble starts with 1) already: which binutils? The >> latest stable may not work for all architectures yet, so a generic >> submodule approach is doomed to fail. > > Let's not speculate here.. There's already an existence proof that this > approach works for most of the targets we support (Aboriginal Linux). > >> Will uClibc work for all targets? There's glibc, eglibc, newlib, ... >> >> Not all targets have a softmmu at this time: unicore32 >> (Fwiw this also means no testing outside Linux hosts.) >> >> There's no requirement that the Linux kernel must have been ported to a >> QEMU target yet or that it is still in mainline or that what is in >> mainline is complete enough for our testing. > > Yes, but this just becomes the exception to prove the rule. That depends. The way I understood you (and that I would like to get clarified on the call) is that you wanted to require qemu-test test cases for patches. That would require it to work for everything without exception. If however you just want this as an optional "I am more confident of my changes if they work with qemu-test on x86, ppc, myfavorite", sort of an alternative to downloading test images from qemu.org and running them manually, then of course it doesn't really matter if target xyz or feature foo cannot be tested in the same automated manner. Andreas
diff --git a/tests/virtio-serial.sh b/tests/virtio-serial.sh new file mode 100755 index 0000000..e95ae6e --- /dev/null +++ b/tests/virtio-serial.sh @@ -0,0 +1,52 @@ +#!/bin/sh + +canary="** waiting for... **" + +in_host() { + tmpchr=$tmpdir/chr.log + + # Also test alias + devname=`choose virtio-serial virtio-serial-pci` + + qemu -nographic -enable-kvm -device $devname \ + -device virtserialport,name=org.qemu.test,chardev=chr0 \ + -chardev file,path=$tmpchr,id=chr0 + rc=$? + + if test $rc = 0; then + if ! grep "$canary" $tmpchr >/dev/null; then + echo "Failed to see output from guest!" + rc=1 + fi + fi + + rm -f $tmpchr + + return $rc +} + +in_guest() { + sysfspath=/sys/bus/virtio/devices/virtio0/virtio-ports/vport0p1 + if ! test -e $sysfspath/name; then + echo "Device not visible!" + return 1 + fi + + name=`cat $sysfspath/name` + + if test "$name" != "org.qemu.test"; then + echo "Device has wrong name!" + echo "Expected 'org.qemu.test', got '$name'" + return 2 + fi + + echo "$canary" > /dev/vport0p1 + + return 0 +} + +if test $QEMU_TEST; then + in_host +else + in_guest +fi