Patchwork [0/4] add "make check"

login
register
mail settings
Submitter Gerd Hoffmann
Date Oct. 25, 2011, 1:27 p.m.
Message ID <4EA6B947.5030005@redhat.com>
Download mbox | patch
Permalink /patch/121680/
State New
Headers show

Comments

Gerd Hoffmann - Oct. 25, 2011, 1:27 p.m.
Hi,

>> I was hoping for more, but maybe we just need to start here and grow 
>> organically, I'll queue it again.
> 
> A while ago I played with some simple IDE tests. It basically was a
> small x86 kernel with an empty image that sends IDE commands and prints
> some results, and a script that invokes the guest and checks whether the
> test has passed or failed.

That reminds me that I've started toying with running tests inside a
guest too.  Stopped working on it a while back due to other priorities.
 Attached what I have so far.

> So at first I started with my own multiboot kernel and copied over some
> parts of kvm-unittest's libc. Clearly not the best idea once it's more
> than a couple of lines, so at some point I took the code and integrated
> with my real kvm-unittests repository.
> 
> Now I don't have to duplicate code any more, but at the same time
> there's no chance that a 'make check' in an upstream qemu tree could run
> this. Tests for other devices will have exactly the same problem.
> 
> Any suggestions on how to go forward with this kind of tests? Should
> this go into qemu or into kvm-unittests? Or should kvm-unittests be
> merged into the qemu tree? Or is the approach completely wrong?

I think we should have some framework to run tests inside the guest in
the qemu source tree.  I'm not sure kvm-unittests is the right tool for
the job though.  It is quite low-level and mainly targets the kvm bits
inside the linux kernel.  Testing -- for example -- usb device emulation
would pretty much require writing a usb stack for kvm-unitests ...

cheers,
  Gerd
From 096f68ea08c3c4baf1bbdc549b257a67ecc87e25 Mon Sep 17 00:00:00 2001
From: Gerd Hoffmann <kraxel@redhat.com>
Date: Tue, 13 Sep 2011 17:38:37 +0200
Subject: [PATCH] initramfs test framework

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
---
 initramfs/.gitignore         |    3 +
 initramfs/10-qemu-udev.rules |    5 +
 initramfs/Makefile           |   36 +++++++
 initramfs/README             |   44 ++++++++
 initramfs/init.c             |  225 ++++++++++++++++++++++++++++++++++++++++++
 initramfs/initramfs-boot     |   32 ++++++
 initramfs/initramfs-create   |  111 +++++++++++++++++++++
 initramfs/test-ehci          |    3 +
 initramfs/test-ehci.good     |    8 ++
 initramfs/test-hello.c       |    7 ++
 initramfs/test-hello.good    |    1 +
 initramfs/test-uhci          |    3 +
 initramfs/test-uhci.good     |    3 +
 13 files changed, 481 insertions(+), 0 deletions(-)
 create mode 100644 initramfs/.gitignore
 create mode 100644 initramfs/10-qemu-udev.rules
 create mode 100644 initramfs/Makefile
 create mode 100644 initramfs/README
 create mode 100644 initramfs/init.c
 create mode 100755 initramfs/initramfs-boot
 create mode 100755 initramfs/initramfs-create
 create mode 100755 initramfs/test-ehci
 create mode 100644 initramfs/test-ehci.good
 create mode 100644 initramfs/test-hello.c
 create mode 100644 initramfs/test-hello.good
 create mode 100755 initramfs/test-uhci
 create mode 100644 initramfs/test-uhci.good
Kevin Wolf - Oct. 25, 2011, 2:16 p.m.
Am 25.10.2011 15:27, schrieb Gerd Hoffmann:
>   Hi,
> 
>>> I was hoping for more, but maybe we just need to start here and grow 
>>> organically, I'll queue it again.
>>
>> A while ago I played with some simple IDE tests. It basically was a
>> small x86 kernel with an empty image that sends IDE commands and prints
>> some results, and a script that invokes the guest and checks whether the
>> test has passed or failed.
> 
> That reminds me that I've started toying with running tests inside a
> guest too.  Stopped working on it a while back due to other priorities.
>  Attached what I have so far.
> 
>> So at first I started with my own multiboot kernel and copied over some
>> parts of kvm-unittest's libc. Clearly not the best idea once it's more
>> than a couple of lines, so at some point I took the code and integrated
>> with my real kvm-unittests repository.
>>
>> Now I don't have to duplicate code any more, but at the same time
>> there's no chance that a 'make check' in an upstream qemu tree could run
>> this. Tests for other devices will have exactly the same problem.
>>
>> Any suggestions on how to go forward with this kind of tests? Should
>> this go into qemu or into kvm-unittests? Or should kvm-unittests be
>> merged into the qemu tree? Or is the approach completely wrong?
> 
> I think we should have some framework to run tests inside the guest in
> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
> the job though.  It is quite low-level and mainly targets the kvm bits
> inside the linux kernel.  Testing -- for example -- usb device emulation
> would pretty much require writing a usb stack for kvm-unitests ...

Which is more or less the only way to do a reasonably complete test
(particularly including the error handling paths that a current Linux
might never exercise) at least for the host controllers.

I'm not sure if you can send arbitrary USB packets from Linux (is USB
passthrough really direct passthrough or does the kernel mess with it?),
so maybe for everything above host controllers you could use that. If
you can't, you're back to your own USB stack.

Kevin
Eduardo Habkost - Oct. 25, 2011, 3:03 p.m.
On Tue, Oct 25, 2011 at 03:27:35PM +0200, Gerd Hoffmann wrote:
[...]
> > A while ago I played with some simple IDE tests. It basically was a
> > small x86 kernel with an empty image that sends IDE commands and prints
> > some results, and a script that invokes the guest and checks whether the
> > test has passed or failed.
> 
> That reminds me that I've started toying with running tests inside a
> guest too.  Stopped working on it a while back due to other priorities.
>  Attached what I have so far.
> 
> > So at first I started with my own multiboot kernel and copied over some
> > parts of kvm-unittest's libc. Clearly not the best idea once it's more
> > than a couple of lines, so at some point I took the code and integrated
> > with my real kvm-unittests repository.
> > 
> > Now I don't have to duplicate code any more, but at the same time
> > there's no chance that a 'make check' in an upstream qemu tree could run
> > this. Tests for other devices will have exactly the same problem.
> > 
> > Any suggestions on how to go forward with this kind of tests? Should
> > this go into qemu or into kvm-unittests? Or should kvm-unittests be
> > merged into the qemu tree? Or is the approach completely wrong?
> 
> I think we should have some framework to run tests inside the guest in
> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
> the job though.  It is quite low-level and mainly targets the kvm bits
> inside the linux kernel.  Testing -- for example -- usb device emulation
> would pretty much require writing a usb stack for kvm-unitests ...

We have a framework to run tests inside a fully-installed guest, that's
KVM-Autotest. But maybe it's too much for the kind of tests you need, I
don't know. There are different "levels" of testing, with different
reequirements, and we need to have good tools for all levels.

Just trying to enumerate the kind of tests somebody may need:

A) Simple unit tests for internal qemu C functions
   - 'make check' can run them, using either libcheck or gtest.
B) Functional tests that tests actual virtualization/emulation, but only
   of some specific subsystems, not using a fully-featured qemu process.
   - We don't have anything like that, today, right? I am not sure we
     need it.
C) Functional tests that just need to run a small binary with no OS
   installed in the guest, but running a fully-feature qemu process.
   - The tests in the 'tests' directory do this, right? kvm-unittests
     does this, right?
D) Functional tests that need a minimal OS installed, with, e.g., at
   least a Linux kernel and a shell.
   - This is what Gerd's patch below does, right? Also, KVM-Autotest can
     be used for this.
E) Functional tests that need a full OS installed and configured.
   - Today we use KVM-Autotest for this.


Does the above model look correct/complete, or is there some case I
missed?


> 
> cheers,
>   Gerd

> From 096f68ea08c3c4baf1bbdc549b257a67ecc87e25 Mon Sep 17 00:00:00 2001
> From: Gerd Hoffmann <kraxel@redhat.com>
> Date: Tue, 13 Sep 2011 17:38:37 +0200
> Subject: [PATCH] initramfs test framework
> 
> Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
> ---
>  initramfs/.gitignore         |    3 +
>  initramfs/10-qemu-udev.rules |    5 +
>  initramfs/Makefile           |   36 +++++++
>  initramfs/README             |   44 ++++++++
>  initramfs/init.c             |  225 ++++++++++++++++++++++++++++++++++++++++++
>  initramfs/initramfs-boot     |   32 ++++++
>  initramfs/initramfs-create   |  111 +++++++++++++++++++++
>  initramfs/test-ehci          |    3 +
>  initramfs/test-ehci.good     |    8 ++
>  initramfs/test-hello.c       |    7 ++
>  initramfs/test-hello.good    |    1 +
>  initramfs/test-uhci          |    3 +
>  initramfs/test-uhci.good     |    3 +
>  13 files changed, 481 insertions(+), 0 deletions(-)
>  create mode 100644 initramfs/.gitignore
>  create mode 100644 initramfs/10-qemu-udev.rules
>  create mode 100644 initramfs/Makefile
>  create mode 100644 initramfs/README
>  create mode 100644 initramfs/init.c
>  create mode 100755 initramfs/initramfs-boot
>  create mode 100755 initramfs/initramfs-create
>  create mode 100755 initramfs/test-ehci
>  create mode 100644 initramfs/test-ehci.good
>  create mode 100644 initramfs/test-hello.c
>  create mode 100644 initramfs/test-hello.good
>  create mode 100755 initramfs/test-uhci
>  create mode 100644 initramfs/test-uhci.good
>
Kevin Wolf - Oct. 25, 2011, 3:22 p.m.
Am 25.10.2011 17:03, schrieb Eduardo Habkost:
> On Tue, Oct 25, 2011 at 03:27:35PM +0200, Gerd Hoffmann wrote:
> [...]
>>> A while ago I played with some simple IDE tests. It basically was a
>>> small x86 kernel with an empty image that sends IDE commands and prints
>>> some results, and a script that invokes the guest and checks whether the
>>> test has passed or failed.
>>
>> That reminds me that I've started toying with running tests inside a
>> guest too.  Stopped working on it a while back due to other priorities.
>>  Attached what I have so far.
>>
>>> So at first I started with my own multiboot kernel and copied over some
>>> parts of kvm-unittest's libc. Clearly not the best idea once it's more
>>> than a couple of lines, so at some point I took the code and integrated
>>> with my real kvm-unittests repository.
>>>
>>> Now I don't have to duplicate code any more, but at the same time
>>> there's no chance that a 'make check' in an upstream qemu tree could run
>>> this. Tests for other devices will have exactly the same problem.
>>>
>>> Any suggestions on how to go forward with this kind of tests? Should
>>> this go into qemu or into kvm-unittests? Or should kvm-unittests be
>>> merged into the qemu tree? Or is the approach completely wrong?
>>
>> I think we should have some framework to run tests inside the guest in
>> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
>> the job though.  It is quite low-level and mainly targets the kvm bits
>> inside the linux kernel.  Testing -- for example -- usb device emulation
>> would pretty much require writing a usb stack for kvm-unitests ...
> 
> We have a framework to run tests inside a fully-installed guest, that's
> KVM-Autotest. But maybe it's too much for the kind of tests you need, I
> don't know. There are different "levels" of testing, with different
> reequirements, and we need to have good tools for all levels.

Autotest works for an entirely different use case.

Autotest is the right tool when you can let the tests run over night or
over the weekend. Good to run regularly on the current git tree, the
right thing to do to test release candidates.

It's the wrong tool for testing the patch that I'm currently working on.
The focus here is on being able to run very quick tests. I don't want to
wait for a guest installation to complete, and I don't even necessarily
want to wait for a full-blown Linux guest to boot up.

> Just trying to enumerate the kind of tests somebody may need:
> 
> A) Simple unit tests for internal qemu C functions
>    - 'make check' can run them, using either libcheck or gtest.
> B) Functional tests that tests actual virtualization/emulation, but only
>    of some specific subsystems, not using a fully-featured qemu process.
>    - We don't have anything like that, today, right? I am not sure we
>      need it.

I think qemu-iotests could be considered an instance of B)

> C) Functional tests that just need to run a small binary with no OS
>    installed in the guest, but running a fully-feature qemu process.
>    - The tests in the 'tests' directory do this, right? kvm-unittests
>      does this, right?

Not sure what test/ does, but for kvm-unittests yes. And this is also
what I was talking about.

> D) Functional tests that need a minimal OS installed, with, e.g., at
>    least a Linux kernel and a shell.
>    - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>      be used for this.
> E) Functional tests that need a full OS installed and configured.
>    - Today we use KVM-Autotest for this.
> 
> 
> Does the above model look correct/complete, or is there some case I
> missed?

I think it covers what we need.

Kevin
Gerd Hoffmann - Oct. 25, 2011, 3:54 p.m.
Hi,

> Just trying to enumerate the kind of tests somebody may need:
> 
> A) Simple unit tests for internal qemu C functions
>    - 'make check' can run them, using either libcheck or gtest.

Yes, those tests should work and succeed anywhere where qemu compiles.

> B) Functional tests that tests actual virtualization/emulation, but only
>    of some specific subsystems, not using a fully-featured qemu process.
>    - We don't have anything like that, today, right? I am not sure we
>      need it.

qemu-kvm used to have such tests.  There is no point in not using qemu
here though, thats why they where transformed into (C).

> C) Functional tests that just need to run a small binary with no OS
>    installed in the guest, but running a fully-feature qemu process.
>    - The tests in the 'tests' directory do this, right? kvm-unittests
>      does this, right?

kvm-unittests yes.

tests/ seems to be a somewhat bitrotted random collection of tests.

> D) Functional tests that need a minimal OS installed, with, e.g., at
>    least a Linux kernel and a shell.
>    - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>      be used for this.

What the patch tries is to use the bits which are on the host anyway to
run guest tests (obviously linux-on-linux only), so we can do parts of
(E) with less configuration, less internet bandwith (no iso downloads),
less disk space requirements (no iso + disk storage) and faster (no
guest installs needed).  Didn't try yet to mount the host filesystem
using 9p yet, with that we could possibly take the approach quite far as
we suddenly don't have just a "minimal os" any more but alot more.

Using the host bits is at the same time a possible trouble spot.  There
are lots of different distros with more or less subtile differences out
there.  Each machine has a different set of packages installed.  So
having this "just work" everythere probably isn't that easy ...

> E) Functional tests that need a full OS installed and configured.
>    - Today we use KVM-Autotest for this.

cheers,
  Gerd
Lucas Meneghel Rodrigues - Oct. 25, 2011, 4:30 p.m.
On 10/25/2011 01:03 PM, Eduardo Habkost wrote:
> On Tue, Oct 25, 2011 at 03:27:35PM +0200, Gerd Hoffmann wrote:
> [...]
>>> A while ago I played with some simple IDE tests. It basically was a
>>> small x86 kernel with an empty image that sends IDE commands and prints
>>> some results, and a script that invokes the guest and checks whether the
>>> test has passed or failed.
>>
>> That reminds me that I've started toying with running tests inside a
>> guest too.  Stopped working on it a while back due to other priorities.
>>   Attached what I have so far.
>>
>>> So at first I started with my own multiboot kernel and copied over some
>>> parts of kvm-unittest's libc. Clearly not the best idea once it's more
>>> than a couple of lines, so at some point I took the code and integrated
>>> with my real kvm-unittests repository.
>>>
>>> Now I don't have to duplicate code any more, but at the same time
>>> there's no chance that a 'make check' in an upstream qemu tree could run
>>> this. Tests for other devices will have exactly the same problem.
>>>
>>> Any suggestions on how to go forward with this kind of tests? Should
>>> this go into qemu or into kvm-unittests? Or should kvm-unittests be
>>> merged into the qemu tree? Or is the approach completely wrong?
>>
>> I think we should have some framework to run tests inside the guest in
>> the qemu source tree.  I'm not sure kvm-unittests is the right tool for
>> the job though.  It is quite low-level and mainly targets the kvm bits
>> inside the linux kernel.  Testing -- for example -- usb device emulation
>> would pretty much require writing a usb stack for kvm-unitests ...
>
> We have a framework to run tests inside a fully-installed guest, that's
> KVM-Autotest. But maybe it's too much for the kind of tests you need, I
> don't know. There are different "levels" of testing, with different
> reequirements, and we need to have good tools for all levels.
>
> Just trying to enumerate the kind of tests somebody may need:
>
> A) Simple unit tests for internal qemu C functions
>     - 'make check' can run them, using either libcheck or gtest.
> B) Functional tests that tests actual virtualization/emulation, but only
>     of some specific subsystems, not using a fully-featured qemu process.
>     - We don't have anything like that, today, right? I am not sure we
>       need it.
> C) Functional tests that just need to run a small binary with no OS
>     installed in the guest, but running a fully-feature qemu process.
>     - The tests in the 'tests' directory do this, right? kvm-unittests
>       does this, right?
> D) Functional tests that need a minimal OS installed, with, e.g., at
>     least a Linux kernel and a shell.
>     - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>       be used for this.

Yes, it can. There's instrumentation to get ssh sessions, serial 
sessions, execute monitor commands... Only need to specify the path to a 
pre-installed minimal guest, skip install tests and have the bare 
minimal subset of tests needed. If only a small subset of tests are run 
with a minimal image, a kvm autotest job is quick to run and suitable 
for development use. I've looked at Gerd's initial patch to add an 
initramfs for such lightweight testing, and the init ramdisk generation 
could be implemented in autotest.

However...

Problem today is that KVM autotest is considered to be hard to approach 
and integrate with qemu developer's workflow. I won't argue it's not, so 
I'm admitting failure here. We've been juggling to keep it stable for 
the regular, full blown QE case, with expanding test and targets 
coverage (see libvirt test), and yet trying to make it better documented 
and sufficiently easy to approach to do simpler ad hoc testing. I wish 
it was easier to accomplish the later, though.

So, an in-tree solution for D) like the one Gerd started to work on 
seems better, as it won't generate frustration from folks having to 
learn kvm autotest, which will ultimately lead to them ignoring the effort.

> E) Functional tests that need a full OS installed and configured.
>     - Today we use KVM-Autotest for this.
>
>
> Does the above model look correct/complete, or is there some case I
> missed?
>
>
>>
>> cheers,
>>    Gerd
>
>>  From 096f68ea08c3c4baf1bbdc549b257a67ecc87e25 Mon Sep 17 00:00:00 2001
>> From: Gerd Hoffmann<kraxel@redhat.com>
>> Date: Tue, 13 Sep 2011 17:38:37 +0200
>> Subject: [PATCH] initramfs test framework
>>
>> Signed-off-by: Gerd Hoffmann<kraxel@redhat.com>
>> ---
>>   initramfs/.gitignore         |    3 +
>>   initramfs/10-qemu-udev.rules |    5 +
>>   initramfs/Makefile           |   36 +++++++
>>   initramfs/README             |   44 ++++++++
>>   initramfs/init.c             |  225 ++++++++++++++++++++++++++++++++++++++++++
>>   initramfs/initramfs-boot     |   32 ++++++
>>   initramfs/initramfs-create   |  111 +++++++++++++++++++++
>>   initramfs/test-ehci          |    3 +
>>   initramfs/test-ehci.good     |    8 ++
>>   initramfs/test-hello.c       |    7 ++
>>   initramfs/test-hello.good    |    1 +
>>   initramfs/test-uhci          |    3 +
>>   initramfs/test-uhci.good     |    3 +
>>   13 files changed, 481 insertions(+), 0 deletions(-)
>>   create mode 100644 initramfs/.gitignore
>>   create mode 100644 initramfs/10-qemu-udev.rules
>>   create mode 100644 initramfs/Makefile
>>   create mode 100644 initramfs/README
>>   create mode 100644 initramfs/init.c
>>   create mode 100755 initramfs/initramfs-boot
>>   create mode 100755 initramfs/initramfs-create
>>   create mode 100755 initramfs/test-ehci
>>   create mode 100644 initramfs/test-ehci.good
>>   create mode 100644 initramfs/test-hello.c
>>   create mode 100644 initramfs/test-hello.good
>>   create mode 100755 initramfs/test-uhci
>>   create mode 100644 initramfs/test-uhci.good
>>
>
Anthony Liguori - Oct. 26, 2011, 8:49 p.m.
On 10/25/2011 10:22 AM, Kevin Wolf wrote:
> Am 25.10.2011 17:03, schrieb Eduardo Habkost:
> I think qemu-iotests could be considered an instance of B)
>
>> C) Functional tests that just need to run a small binary with no OS
>>     installed in the guest, but running a fully-feature qemu process.
>>     - The tests in the 'tests' directory do this, right? kvm-unittests
>>       does this, right?
>
> Not sure what test/ does, but for kvm-unittests yes. And this is also
> what I was talking about.

Thinking more about this...

We could add a new '-x-test-server CHR' option.  When this option is added, it 
would do the following:

1) Open CHR character device
2) Use /dev/shm for guest memory
3) Listen for connections on CHR
4) When something connects to CHR
  a) reset device model
  b) send /dev/shm fd over CHR
  c) register CPU physical memory client
     1. upon CPU physical memory changes, send the change info over CHR
  d) instead of doing [kvm_]cpu_exec(), block reading on CHR

So when you launch qemu with -x-test-server, it'll sit there doing nothing 
terribly useful.  But this lets you write a program that connects to CHR, and 
then by mapping {out,in}[bwl] to RPCs over the connection, and accessing RAM via 
mmap()'ing the passed fd using the client mapping table, you can essentially 
write kvm-unittest style tests while still having full access to libc.

And since each test program can reset QEMU after running, you could very nicely 
tie into something like gtest as a unit test framework.  I think it's pretty 
appealing from a debugability perspective too.

It also means that it's possible to have 100% C test cases such that you could 
still build something like ppc64-softmmu and run it against the written test 
cases without having to really understand ppc64 assembly or have a ppc64 build 
environment (to generate native binaries to run under ppc64 TCG).

I think this could work out fairly well as a unit test framework.

Regards,

Anthony Liguori

>
>> D) Functional tests that need a minimal OS installed, with, e.g., at
>>     least a Linux kernel and a shell.
>>     - This is what Gerd's patch below does, right? Also, KVM-Autotest can
>>       be used for this.
>> E) Functional tests that need a full OS installed and configured.
>>     - Today we use KVM-Autotest for this.
>>
>>
>> Does the above model look correct/complete, or is there some case I
>> missed?
>
> I think it covers what we need.
>
> Kevin
>
Kevin Wolf - Oct. 27, 2011, 8:20 a.m.
Am 26.10.2011 22:49, schrieb Anthony Liguori:
> On 10/25/2011 10:22 AM, Kevin Wolf wrote:
>> Am 25.10.2011 17:03, schrieb Eduardo Habkost:
>> I think qemu-iotests could be considered an instance of B)
>>
>>> C) Functional tests that just need to run a small binary with no OS
>>>     installed in the guest, but running a fully-feature qemu process.
>>>     - The tests in the 'tests' directory do this, right? kvm-unittests
>>>       does this, right?
>>
>> Not sure what test/ does, but for kvm-unittests yes. And this is also
>> what I was talking about.
> 
> Thinking more about this...
> 
> We could add a new '-x-test-server CHR' option.  When this option is added, it 
> would do the following:
> 
> 1) Open CHR character device
> 2) Use /dev/shm for guest memory
> 3) Listen for connections on CHR
> 4) When something connects to CHR
>   a) reset device model
>   b) send /dev/shm fd over CHR
>   c) register CPU physical memory client
>      1. upon CPU physical memory changes, send the change info over CHR
>   d) instead of doing [kvm_]cpu_exec(), block reading on CHR
> 
> So when you launch qemu with -x-test-server, it'll sit there doing nothing 
> terribly useful.  But this lets you write a program that connects to CHR, and 
> then by mapping {out,in}[bwl] to RPCs over the connection, and accessing RAM via 
> mmap()'ing the passed fd using the client mapping table, you can essentially 
> write kvm-unittest style tests while still having full access to libc.

IRQs need to go through the connection as well.

Oh, and you would finally have a C user for libqmp. The test cases
definitely need to be able to access the monitor. For example I would
really love to have test cases for the I/O error paths that stop the VM
(or actually it's the resume that must be tested).

> And since each test program can reset QEMU after running, you could very nicely 
> tie into something like gtest as a unit test framework.  I think it's pretty 
> appealing from a debugability perspective too.
> 
> It also means that it's possible to have 100% C test cases such that you could 
> still build something like ppc64-softmmu and run it against the written test 
> cases without having to really understand ppc64 assembly or have a ppc64 build 
> environment (to generate native binaries to run under ppc64 TCG).
> 
> I think this could work out fairly well as a unit test framework.

Sounds great, where are the patches? ;-)

Kevin

Patch

diff --git a/initramfs/.gitignore b/initramfs/.gitignore
new file mode 100644
index 0000000..8ece42c
--- /dev/null
+++ b/initramfs/.gitignore
@@ -0,0 +1,3 @@ 
+initramfs.cpio.gz
+init
+test-hello
diff --git a/initramfs/10-qemu-udev.rules b/initramfs/10-qemu-udev.rules
new file mode 100644
index 0000000..fb5cc0a
--- /dev/null
+++ b/initramfs/10-qemu-udev.rules
@@ -0,0 +1,5 @@ 
+# load modules
+DRIVER!="?*", ENV{MODALIAS}=="?*", RUN+="/sbin/modprobe -b $env{MODALIAS}"
+
+# virtio console
+KERNEL=="vport*", ATTR{name}=="?*", SYMLINK+="virtio-ports/$attr{name}"
diff --git a/initramfs/Makefile b/initramfs/Makefile
new file mode 100644
index 0000000..2db2b76
--- /dev/null
+++ b/initramfs/Makefile
@@ -0,0 +1,36 @@ 
+
+CC	:= gcc
+CFLAGS	:= -O2 -g -Wall
+LDFLAGS	:= -lutil
+
+TOOLS	:= init
+TEST_C	:= test-hello
+TEST_SH	:= test-uhci test-ehci
+
+TARGETS := $(TOOLS) $(TEST_C) initramfs.cpio.gz
+TESTS	:= $(TEST_C) $(TEST_SH)
+
+run-test-uhci : QEMU_ARGS := -usb -device usb-tablet
+run-test-ehci : QEMU_ARGS := -readconfig ../docs/ich9-ehci-uhci.cfg
+
+default all: $(TARGETS)
+
+clean:
+	rm -f $(TARGETS) *.o *~
+	rm -f org.qemu.initramfs.log
+
+init: init.o
+test-hello: test-hello.o
+
+initramfs.cpio.gz: $(TOOLS) $(TESTS) initramfs-create
+	./initramfs-create $@ $(TESTS)
+
+boot shell: $(TARGETS)
+	./initramfs-boot
+
+run-test-%: $(TARGETS)
+	./initramfs-boot /tests/test-$* $(QEMU_ARGS)
+	diff -u org.qemu.initramfs.log test-$*.good
+
+run-tests: $(patsubst %,run-%,$(TESTS))
+
diff --git a/initramfs/README b/initramfs/README
new file mode 100644
index 0000000..a9504ad
--- /dev/null
+++ b/initramfs/README
@@ -0,0 +1,44 @@ 
+
+This is an experimental test framework.
+
+Design goals
+------------
+
+  * Allow running tests within a guest.
+  * Be small enougth that it can easily be included in
+    the qemu source tree.
+  * Don't require setup and/or downloading stuff
+    (i.e. guest images) from the internet.
+  * Be easy to use.
+
+
+How it works
+------------
+
+It creates a linux initramfs from the bits found on the host machine,
+then boots the host kernel with the initramfs just created within
+qemu.  A special init handles the setup (create core devices, mount
+filesystems, start udev), command execution and shutdown.
+
+Obviously requires a linux host.  It also needs udev for device setup
+and module loading and virtio-serial support for logging.
+
+
+Getting started
+---------------
+
+Just type "make boot", a few seconds later you should be greeted by
+the guests bash prompt.  You can look around now.  Exiting the shell
+will shutdown the guest.  You'll find the shell output logged in the
+"org.qemu.initramfs.log" file.
+
+
+Run tests
+---------
+
+Type "make run-test-$name" for a single test or "make run-tests" to
+run all of them.  This will run the test program instead of the shell,
+then compare the actual output with the expected output.
+
+This can probably be refined ...
+
diff --git a/initramfs/init.c b/initramfs/init.c
new file mode 100644
index 0000000..94a8764
--- /dev/null
+++ b/initramfs/init.c
@@ -0,0 +1,225 @@ 
+#include <stdio.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <unistd.h>
+#include <string.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <pty.h>
+
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <sys/wait.h>
+#include <sys/mount.h>
+#include <sys/reboot.h>
+#include <sys/select.h>
+
+#define ARRAY_SIZE(x) (sizeof(x)/sizeof(x[0]))
+
+static const char *logdev = "/dev/virtio-ports/org.qemu.initramfs.log";
+
+static struct {
+    mode_t     mode;
+    const char *path;
+    int        major;
+    int        minor;
+} cdevs[] = {
+    { 0666, "/dev/null",    1,  3 },
+    { 0600, "/dev/console", 5,  1 },
+    { 0666, "/dev/ptmx",    5,  2 },
+    { 0660, "/dev/kmsg",    1, 11 },
+};
+
+static struct {
+    const char *dest;
+    const char *type;
+} fs[] = {
+    { "/proc", "proc" },
+    { "/sys", "sysfs" },
+    { "/sys/kernel/debug", "debugfs" },
+    { "/dev/pts", "devpts" },
+    { "/dev/shm", "tmpfs" },
+};
+
+int run(const char *cmd, ...)
+{
+    va_list args;
+    char *argv[16], *arg;
+    int i, pid;
+
+    va_start(args, cmd);
+    argv[0] = (char*)cmd;
+    for (i = 1; i < ARRAY_SIZE(argv)-1; i++) {
+        arg = va_arg(args, char*);
+        if (arg == NULL) {
+            break;
+        }
+        argv[i] = arg;
+    }
+    argv[i] = NULL;
+    va_end(args);
+
+    pid = fork();
+    if (pid != 0) {
+        /* parent */
+        if (pid < 0) {
+            perror("fork");
+        }
+        return pid;
+    }
+    /* child */
+    execv(cmd, argv);
+    fprintf(stderr, "exec %s: %s\n", cmd, strerror(errno));
+    exit(1);
+}
+
+int forward(int from_fd, int to_fd, int log_fd)
+{
+    int len, pos, rc;
+    char buf[512];
+
+    len = read(from_fd, buf, sizeof(buf));
+    if (len < 0) {
+        return len;
+    }
+    for (pos = 0; pos < len; pos += rc) {
+        rc = write(to_fd, buf + pos, len - pos);
+        if (rc < 0) {
+            return rc;
+        }
+    }
+    if (log_fd != -1) {
+        for (pos = 0; pos < len; pos += rc) {
+            rc = write(log_fd, buf + pos, len - pos);
+            if (rc < 0) {
+                return rc;
+            }
+        }
+    }
+    return len;
+}
+
+static void
+tty_raw(int fd)
+{
+    struct termios tattr;
+
+    tcgetattr(fd, &tattr);
+    tattr.c_lflag &= ~(ICANON|ECHO);
+    tattr.c_cc[VMIN] = 1;
+    tattr.c_cc[VTIME] = 0;
+    tcsetattr(fd, TCSAFLUSH, &tattr);
+}
+
+int main(int argc, char *argv[])
+{
+    const char *command;
+    pid_t cmdpid, pid;
+    int status, i, rc, pty, log;
+    bool cmd_exit, eof_seen;
+
+    /* say hi */
+    fprintf(stderr, "-*- qemu initramfs starting -*-\n");
+
+    /* mount filesystems */
+    for (i = 0; i < ARRAY_SIZE(fs); i++) {
+        rc = mount(fs[i].type, fs[i].dest, fs[i].type, 0, NULL);
+        if (rc < 0) {
+            fprintf(stderr, "mount %s at %s: %s\n",
+                    fs[i].type, fs[i].dest, strerror(errno));
+        }
+    }
+
+    /* create basic chardevs */
+    for (i = 0; i < ARRAY_SIZE(cdevs); i++) {
+        rc = mknod(cdevs[i].path, cdevs[i].mode | S_IFCHR,
+                   makedev(cdevs[i].major, cdevs[i].minor));
+        if (rc < 0 && errno != EEXIST) {
+            fprintf(stderr, "mknod %s : %s\n",
+                    cdevs[i].path, strerror(errno));
+        }
+    }
+
+    /* start udev, let it create devices */
+    fprintf(stderr, "-*- starting udev -*-\n");
+    run("/sbin/depmod", "-a", NULL);
+    run("/sbin/udevd", "--daemon", "--resolve-names=never", NULL);
+    run("/sbin/udevadm", "trigger", "--action=add", NULL);
+
+    /* wait until virtio-serial is up'n'running */
+    for (log = -1, i = 0; log == -1 && i < 32; i++) {
+        sleep(1);
+        run("/sbin/udevadm", "settle", NULL);
+        log = open(logdev, O_WRONLY);
+    }
+    if (log == -1) {
+        fprintf(stderr, "open %s: %s\n", logdev, strerror(errno));
+    }
+
+    /* setup environment */
+    setenv("PATH", "/sbin:/bin", 1);
+
+    /* run our command in a pseuto tty */
+    command = getenv("QEMU_RUN");
+    if (command == NULL) {
+        command = "/bin/bash";
+    }
+    fprintf(stderr, "-*- running %s -*-\n", command);
+
+    tty_raw(0);
+    cmdpid = forkpty(&pty, NULL, NULL, NULL);
+    if (cmdpid == 0) {
+        /* child */
+        execl(command, command, NULL);
+        fprintf(stderr, "exec %s: %s\n", command, strerror(errno));
+        exit(1);
+    }
+
+    /* main loop */
+    cmd_exit = false;
+    eof_seen = false;
+    do {
+        fd_set rd;
+
+        /* reap children and zombies */
+        for (;;) {
+            pid = waitpid(-1, &status, WNOHANG);
+            if (pid <= 0) {
+                break;
+            }
+            if (pid == cmdpid) {
+                cmd_exit = true;
+            }
+        }
+
+        /* forward data between stdio (aka /dev/console) and pseudo tty */
+        if (!eof_seen) {
+            FD_ZERO(&rd);
+            FD_SET(0, &rd);
+            FD_SET(pty, &rd);
+            rc = select(pty+1, &rd, NULL, NULL, NULL);
+            if (rc > 0) {
+                if (FD_ISSET(0, &rd)) {
+                    /* stdin -> pseudo tty */
+                    if (forward(0, pty, -1) <= 0) {
+                        eof_seen = 1;
+                    }
+                }
+                if (FD_ISSET(pty, &rd)) {
+                    /* pseudo tty -> stdout + log */
+                    if (forward(pty, 1, log) <= 0) {
+                        eof_seen = 1;
+                    }
+                }
+            }
+        }
+    } while (!cmd_exit || !eof_seen);
+
+    /* powerdown vm */
+    fprintf(stderr, "-*- qemu initramfs done -*-\n");
+    reboot(RB_POWER_OFF);
+
+    /* keep gcc happy ;) */
+    return 0;
+}
diff --git a/initramfs/initramfs-boot b/initramfs/initramfs-boot
new file mode 100755
index 0000000..4884a6d
--- /dev/null
+++ b/initramfs/initramfs-boot
@@ -0,0 +1,32 @@ 
+#!/bin/sh
+
+base="$(dirname $0)"
+command="$1"
+shift;
+
+case "$(uname -m)" in
+	x86_64)	qemu="x86_64-softmmu/qemu-system-x86_64"
+		;;
+	i?86)	qemu="i386-softmmu/qemu-system-i386"
+		;;
+	*)	echo "unknown arch: $(uname -m)"
+		exit 1
+		;;
+esac
+
+kernel="$(echo /boot/vmlinu*$(uname -r)*)"
+initrd="initramfs.cpio.gz"
+append="console=ttyS0"
+if test "$command" != ""; then
+	append="$append QEMU_RUN=$command"
+fi
+
+exec ../$qemu -nographic -no-reboot \
+	-machine accel=kvm:tcg \
+	-kernel "$kernel" \
+	-initrd "$initrd" \
+	-append "$append" \
+	-chardev file,id=cmdlog,path=org.qemu.initramfs.log \
+	-device virtio-serial \
+	-device virtserialport,name=org.qemu.initramfs.log,chardev=cmdlog \
+	"$@"
diff --git a/initramfs/initramfs-create b/initramfs/initramfs-create
new file mode 100755
index 0000000..4525c11
--- /dev/null
+++ b/initramfs/initramfs-create
@@ -0,0 +1,111 @@ 
+#!/bin/bash
+#
+# Create a simple linux initramfs with the stuff found on the host.
+# Intented to be used for a quick boot test with the host kernel.
+#
+
+base="$(dirname $0)"
+file="${1-initramfs.cpio.gz}"; shift
+tests="$*"
+
+# create work dir
+WORK="${TMPDIR-/tmp}/${0##*/}-$$"
+mkdir "$WORK" || exit 1
+trap 'rm -rf "$WORK"' EXIT
+
+
+##############################################################################
+# helper functions
+
+function add_dirs() {
+	local dest="$WORK/fs"
+	local dir
+	for dir in dev dev/pts dev/shm etc proc sys \
+		   var var/tmp /var/log tmp
+	do
+		mkdir -p "$dest/$dir"
+	done
+}
+
+function add_binary() {
+	local dest="$WORK/fs$1"; shift
+	local item
+
+       	mkdir -p "$dest"
+	for item in $*; do
+		cp -L "$(which $item)" "$dest" || exit 1
+	done
+}
+
+function add_symlink() {
+	local target="$1"
+	local name="$2"
+	ln -s "$target" "$WORK/fs$name"
+}
+
+function add_data() {
+	local dest="$WORK/fs"
+	local item
+
+	for item in $*; do
+	       	mkdir -p $(dirname "$dest/$item")
+		cp -L "$item" "$dest/$item" || exit 1
+	done
+}
+
+function add_data_to_dir() {
+	local dest="$WORK/fs$1"; shift
+	local item
+
+       	mkdir -p "$dest"
+	for item in $*; do
+		cp -L "$item" "$dest" || exit 1
+	done
+}
+
+function add_libs() {
+	local item dest
+	for item in $(ldd $WORK/fs/init $WORK/fs/bin/* $WORK/fs/sbin/* \
+			$WORK/fs/tests/* $WORK/fs/lib/udev/*id); do
+		test -f "$item" || continue
+		test -f "$WORK/fs$item" && continue
+		dest=$(dirname "$WORK/fs$item")
+		mkdir -p "$dest"
+		cp -L "$item" "$dest"
+	done
+}
+
+function add_modules() {
+	local item
+
+	echo -n > "$WORK/modules"
+	for item in $*; do
+		modprobe --show-depends $item 2>/dev/null \
+			| awk '{ print $2 }' >> "$WORK/modules"
+	done
+	add_data $(sort "$WORK/modules" | uniq)
+}
+
+
+##############################################################################
+# main
+
+add_dirs
+add_binary / $base/init
+for t in $tests; do
+	add_binary /tests $base/$t;
+done
+add_binary /bin bash ls cat more dmesg ps uname find sort grep uniq
+add_binary /sbin lspci lsusb mount umount udevd udevadm blkid
+add_binary /sbin depmod insmod lsmod modinfo modprobe rmmod ip
+add_symlink bash /bin/sh
+add_data /usr/share/hwdata/*.ids
+add_data /lib/udev/*id
+add_data /lib/udev/rules.d/60-persistent-storage.rules
+add_data_to_dir /lib/udev/rules.d $base/10-qemu-udev.rules
+add_libs
+add_modules virtio_pci virtio_blk virtio_net virtio_console \
+	virtio_balloon virtio-rng 9pnet_virtio 9p \
+	ata_piix ahci sd_mod sr_mod sg e1000 8139cp
+
+(cd $WORK/fs; find -print | cpio -o -R 0:0 -H newc) | gzip > "$file"
diff --git a/initramfs/test-ehci b/initramfs/test-ehci
new file mode 100755
index 0000000..bd4bb5c
--- /dev/null
+++ b/initramfs/test-ehci
@@ -0,0 +1,3 @@ 
+#!/bin/sh
+/sbin/lspci -s1d
+/sbin/lsusb | sort
diff --git a/initramfs/test-ehci.good b/initramfs/test-ehci.good
new file mode 100644
index 0000000..dd9c6b8
--- /dev/null
+++ b/initramfs/test-ehci.good
@@ -0,0 +1,8 @@ 
+00:1d.0 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)

+00:1d.1 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)

+00:1d.2 USB Controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)

+00:1d.7 USB Controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)

+Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

+Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

+Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

+Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

diff --git a/initramfs/test-hello.c b/initramfs/test-hello.c
new file mode 100644
index 0000000..e5ce2d2
--- /dev/null
+++ b/initramfs/test-hello.c
@@ -0,0 +1,7 @@ 
+#include <stdio.h>
+
+int main(int argc, char *argv[])
+{
+    printf("Hello world!\n");
+    return 0;
+}
diff --git a/initramfs/test-hello.good b/initramfs/test-hello.good
new file mode 100644
index 0000000..dfd6895
--- /dev/null
+++ b/initramfs/test-hello.good
@@ -0,0 +1 @@ 
+Hello world!

diff --git a/initramfs/test-uhci b/initramfs/test-uhci
new file mode 100755
index 0000000..0af70e0
--- /dev/null
+++ b/initramfs/test-uhci
@@ -0,0 +1,3 @@ 
+#!/bin/sh
+/sbin/lspci -s1.2
+/sbin/lsusb | sort
\ No newline at end of file
diff --git a/initramfs/test-uhci.good b/initramfs/test-uhci.good
new file mode 100644
index 0000000..c87271e
--- /dev/null
+++ b/initramfs/test-uhci.good
@@ -0,0 +1,3 @@ 
+00:01.2 USB Controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)

+Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

+Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd