| Message ID | 20180622004435.10291-5-f4bug@amsat.org |
|---|---|
| State | New |
| Headers | show |
| Series | Avocado: start multi-arch tests, add a Travis job | expand |
Philippe Mathieu-Daudé <f4bug@amsat.org> writes: > Similar to the BootLinuxConsoleX86_64 test: > boot a Linux kernel on a Malta board and verify the serial is working. > > This test can be run using: > > $ avocado run -t endian:big tests/acceptance > > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> > --- > tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++ > 1 file changed, 38 insertions(+) > > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py > index 17dc8d58c1..72cf5e943c 100644 > --- a/tests/acceptance/boot_linux_console.py > +++ b/tests/acceptance/boot_linux_console.py > @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test): > break > if 'Kernel panic - not syncing' in msg: > self.fail("Kernel panic reached") > + > + > +class BootLinuxConsoleMips(Test): > + """ > + Boots a mips Linux kernel and checks that the console is operational > + and the kernel command line is properly passed from QEMU to the kernel > + > + :avocado: enable > + :avocado: tags=endian:big > + :avocado: tags=arch:mips > + :avocado: tags=board:malta > + """ > + > + arch = "mips" > + timeout = 60 > + > + def test(self): > + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' > + 'vmlinux-3.2.0-4-4kc-malta') > + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' > + kernel_path = self.fetch_asset(kernel_url, > asset_hash=kernel_hash) I'm uncomfortable using "random" binaries of websites as the source of our test kernels. I can see the justification for distro kernels as they at least have the infrastructure to rebuild from source if you really want to, but even then the distros don't cover a lot of the architectures. I had experimented with using docker based builds for building test fixtures (see tests/docker/dockerbuilds): https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2 As these tests are fairly simple boot tests that just need kernels maybe we could look at tooling up the generation of these images in a repeatable way - similar to the way vmtest builds VMs. > + > + self.vm.set_machine('malta') > + self.vm.set_console() > + kernel_command_line = 'console=ttyS0 printk.time=0' > + self.vm.add_args('-m', "64", > + '-kernel', kernel_path, > + '-append', kernel_command_line) > + self.vm.launch() > + console = self.vm.console_socket.makefile() > + console_logger = logging.getLogger('console') > + while True: > + msg = console.readline() > + console_logger.debug(msg.strip()) > + if 'Kernel command line: %s' % kernel_command_line in msg: > + break > + if 'Kernel panic - not syncing' in msg: > + self.fail("Kernel panic reached") Of course for bonus points a simple rootfs with hackbench or some such in it would be nice. But I appreciate this makes the building job a lot more complex than just a kernel. -- Alex Bennée
On 06/28/2018 01:23 PM, Alex Bennée wrote: > > Philippe Mathieu-Daudé <f4bug@amsat.org> writes: > >> Similar to the BootLinuxConsoleX86_64 test: >> boot a Linux kernel on a Malta board and verify the serial is working. >> >> This test can be run using: >> >> $ avocado run -t endian:big tests/acceptance >> >> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >> --- >> tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++ >> 1 file changed, 38 insertions(+) >> >> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py >> index 17dc8d58c1..72cf5e943c 100644 >> --- a/tests/acceptance/boot_linux_console.py >> +++ b/tests/acceptance/boot_linux_console.py >> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test): >> break >> if 'Kernel panic - not syncing' in msg: >> self.fail("Kernel panic reached") >> + >> + >> +class BootLinuxConsoleMips(Test): >> + """ >> + Boots a mips Linux kernel and checks that the console is operational >> + and the kernel command line is properly passed from QEMU to the kernel >> + >> + :avocado: enable >> + :avocado: tags=endian:big >> + :avocado: tags=arch:mips >> + :avocado: tags=board:malta >> + """ >> + >> + arch = "mips" >> + timeout = 60 >> + >> + def test(self): >> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' >> + 'vmlinux-3.2.0-4-4kc-malta') >> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' >> + kernel_path = self.fetch_asset(kernel_url, >> asset_hash=kernel_hash) > > I'm uncomfortable using "random" binaries of websites as the source of > our test kernels. I can see the justification for distro kernels as they > at least have the infrastructure to rebuild from source if you really > want to, but even then the distros don't cover a lot of the > architectures. And now I notice I made an mistake here :) I guess remember the Avocado team started using the SHA-1 hash as default and I suggested them to be able to use other hashes for this particular case, since Aurelien provided the MD5 hashes signed by his GPG key, which is signed/trusted by Peter and used to merge mips32 pulls. That would verify the QEMU community circle of trust right? I don't think Avocado should parse the FTP/HTTP signed indexes, but a manual verification when merging this series should suffice. > > I had experimented with using docker based builds for building test > fixtures (see tests/docker/dockerbuilds): > > https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2 > > As these tests are fairly simple boot tests that just need kernels maybe > we could look at tooling up the generation of these images in a > repeatable way - similar to the way vmtest builds VMs. Yes, I have another acceptance branch where I cross-build an old mipssim kernel to test the board, using the following: http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html But preparing a Docker cross image, fetching the Linux kernel source, building it, takes a lot of time/storage I'd rather avoid; at least with Aurelien kernels, since they are known to work since years. >> + >> + self.vm.set_machine('malta') >> + self.vm.set_console() >> + kernel_command_line = 'console=ttyS0 printk.time=0' >> + self.vm.add_args('-m', "64", >> + '-kernel', kernel_path, >> + '-append', kernel_command_line) >> + self.vm.launch() >> + console = self.vm.console_socket.makefile() >> + console_logger = logging.getLogger('console') >> + while True: >> + msg = console.readline() >> + console_logger.debug(msg.strip()) >> + if 'Kernel command line: %s' % kernel_command_line in msg: >> + break >> + if 'Kernel panic - not syncing' in msg: >> + self.fail("Kernel panic reached") > > Of course for bonus points a simple rootfs with hackbench or some such > in it would be nice. But I appreciate this makes the building job a lot > more complex than just a kernel. My idea is to use the rootfs for larger tests, and tag the "Kernel panic" tests as "quick", so we can have a "make acceptance-speed" or similar. We can already test than many devices were initialized correctly quickly looking at this console output. Regards, Phil.
Philippe Mathieu-Daudé <f4bug@amsat.org> writes: > On 06/28/2018 01:23 PM, Alex Bennée wrote: >> >> Philippe Mathieu-Daudé <f4bug@amsat.org> writes: >> >>> Similar to the BootLinuxConsoleX86_64 test: >>> boot a Linux kernel on a Malta board and verify the serial is working. >>> >>> This test can be run using: >>> >>> $ avocado run -t endian:big tests/acceptance >>> >>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >>> --- >>> tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++ >>> 1 file changed, 38 insertions(+) >>> >>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py >>> index 17dc8d58c1..72cf5e943c 100644 >>> --- a/tests/acceptance/boot_linux_console.py >>> +++ b/tests/acceptance/boot_linux_console.py >>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test): >>> break >>> if 'Kernel panic - not syncing' in msg: >>> self.fail("Kernel panic reached") >>> + >>> + >>> +class BootLinuxConsoleMips(Test): >>> + """ >>> + Boots a mips Linux kernel and checks that the console is operational >>> + and the kernel command line is properly passed from QEMU to the kernel >>> + >>> + :avocado: enable >>> + :avocado: tags=endian:big >>> + :avocado: tags=arch:mips >>> + :avocado: tags=board:malta >>> + """ >>> + >>> + arch = "mips" >>> + timeout = 60 >>> + >>> + def test(self): >>> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' >>> + 'vmlinux-3.2.0-4-4kc-malta') >>> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' >>> + kernel_path = self.fetch_asset(kernel_url, >>> asset_hash=kernel_hash) >> >> I'm uncomfortable using "random" binaries of websites as the source of >> our test kernels. I can see the justification for distro kernels as they >> at least have the infrastructure to rebuild from source if you really >> want to, but even then the distros don't cover a lot of the >> architectures. > > And now I notice I made an mistake here :) I guess remember the Avocado > team started using the SHA-1 hash as default and I suggested them to be > able to use other hashes for this particular case, since Aurelien > provided the MD5 hashes signed by his GPG key, which is signed/trusted > by Peter and used to merge mips32 pulls. > > That would verify the QEMU community circle of trust right? It's not so much the chain of trust and more repeatability of building the test cases. I trust Aurel32's binaries are good test cases for MIPS but at the moment I have no idea how to rebuild them which is a bit of an issue given it is covered by the GPL. > > I don't think Avocado should parse the FTP/HTTP signed indexes, but a > manual verification when merging this series should suffice. > >> >> I had experimented with using docker based builds for building test >> fixtures (see tests/docker/dockerbuilds): >> >> https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2 >> >> As these tests are fairly simple boot tests that just need kernels maybe >> we could look at tooling up the generation of these images in a >> repeatable way - similar to the way vmtest builds VMs. > > Yes, I have another acceptance branch where I cross-build an old mipssim > kernel to test the board, using the following: > http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html > > But preparing a Docker cross image, fetching the Linux kernel source, > building it, takes a lot of time/storage I'd rather avoid; at least with > Aurelien kernels, since they are known to work since years. Well you can throw away the docker image as long as we have a mechanism to dump out the final binary. I have no interest in forcing people to keep checked out code using up space, just that they can re-build if they need to. > >>> + >>> + self.vm.set_machine('malta') >>> + self.vm.set_console() >>> + kernel_command_line = 'console=ttyS0 printk.time=0' >>> + self.vm.add_args('-m', "64", >>> + '-kernel', kernel_path, >>> + '-append', kernel_command_line) >>> + self.vm.launch() >>> + console = self.vm.console_socket.makefile() >>> + console_logger = logging.getLogger('console') >>> + while True: >>> + msg = console.readline() >>> + console_logger.debug(msg.strip()) >>> + if 'Kernel command line: %s' % kernel_command_line in msg: >>> + break >>> + if 'Kernel panic - not syncing' in msg: >>> + self.fail("Kernel panic reached") >> >> Of course for bonus points a simple rootfs with hackbench or some such >> in it would be nice. But I appreciate this makes the building job a lot >> more complex than just a kernel. > > My idea is to use the rootfs for larger tests, and tag the "Kernel > panic" tests as "quick", so we can have a "make acceptance-speed" or > similar. > > We can already test than many devices were initialized correctly quickly > looking at this console output. Yeah the kernel boot up is a pretty good smoke test (it's all most of kernelci manages as well). However it certainly doesn't fully exercise the translator. I've lost track of the number of bugs we found after successfully booting an kernel because we were doing exhaustive instruction testing. > > Regards, > > Phil. -- Alex Bennée
On 06/28/2018 03:36 PM, Alex Bennée wrote: > Philippe Mathieu-Daudé <f4bug@amsat.org> writes: >> On 06/28/2018 01:23 PM, Alex Bennée wrote: >>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes: >>> >>>> Similar to the BootLinuxConsoleX86_64 test: >>>> boot a Linux kernel on a Malta board and verify the serial is working. >>>> >>>> This test can be run using: >>>> >>>> $ avocado run -t endian:big tests/acceptance >>>> >>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >>>> --- >>>> tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++ >>>> 1 file changed, 38 insertions(+) >>>> >>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py >>>> index 17dc8d58c1..72cf5e943c 100644 >>>> --- a/tests/acceptance/boot_linux_console.py >>>> +++ b/tests/acceptance/boot_linux_console.py >>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test): >>>> break >>>> if 'Kernel panic - not syncing' in msg: >>>> self.fail("Kernel panic reached") >>>> + >>>> + >>>> +class BootLinuxConsoleMips(Test): >>>> + """ >>>> + Boots a mips Linux kernel and checks that the console is operational >>>> + and the kernel command line is properly passed from QEMU to the kernel >>>> + >>>> + :avocado: enable >>>> + :avocado: tags=endian:big >>>> + :avocado: tags=arch:mips >>>> + :avocado: tags=board:malta >>>> + """ >>>> + >>>> + arch = "mips" >>>> + timeout = 60 >>>> + >>>> + def test(self): >>>> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' >>>> + 'vmlinux-3.2.0-4-4kc-malta') >>>> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' >>>> + kernel_path = self.fetch_asset(kernel_url, >>>> asset_hash=kernel_hash) >>> >>> I'm uncomfortable using "random" binaries of websites as the source of >>> our test kernels. I can see the justification for distro kernels as they >>> at least have the infrastructure to rebuild from source if you really >>> want to, but even then the distros don't cover a lot of the >>> architectures. >> >> And now I notice I made an mistake here :) I guess remember the Avocado >> team started using the SHA-1 hash as default and I suggested them to be >> able to use other hashes for this particular case, since Aurelien >> provided the MD5 hashes signed by his GPG key, which is signed/trusted >> by Peter and used to merge mips32 pulls. >> >> That would verify the QEMU community circle of trust right? > > It's not so much the chain of trust and more repeatability of building > the test cases. I trust Aurel32's binaries are good test cases for MIPS > but at the moment I have no idea how to rebuild them which is a bit of > an issue given it is covered by the GPL. OK! I didn't even think about that since this is a "Linux" kernel and a "Debian" rootfs provided by a Debian developer, hosted on a Debian server, so I'm pretty sure this is GPL compliant, but it makes sens. I'll find a way to have reproducible test binaries for acceptance testing. >> I don't think Avocado should parse the FTP/HTTP signed indexes, but a >> manual verification when merging this series should suffice. >> >>> >>> I had experimented with using docker based builds for building test >>> fixtures (see tests/docker/dockerbuilds): >>> >>> https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2 >>> >>> As these tests are fairly simple boot tests that just need kernels maybe >>> we could look at tooling up the generation of these images in a >>> repeatable way - similar to the way vmtest builds VMs. >> >> Yes, I have another acceptance branch where I cross-build an old mipssim >> kernel to test the board, using the following: >> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html >> >> But preparing a Docker cross image, fetching the Linux kernel source, >> building it, takes a lot of time/storage I'd rather avoid; at least with >> Aurelien kernels, since they are known to work since years. > > Well you can throw away the docker image as long as we have a mechanism > to dump out the final binary. I have no interest in forcing people to > keep checked out code using up space, just that they can re-build if > they need to. The Avocado guys said they have plenty of storage for assets. A cool feature would be to have some config built by some CI builder that then sign and push reproducible config + generated binary to the assets storage automatically. >>>> + >>>> + self.vm.set_machine('malta') >>>> + self.vm.set_console() >>>> + kernel_command_line = 'console=ttyS0 printk.time=0' >>>> + self.vm.add_args('-m', "64", >>>> + '-kernel', kernel_path, >>>> + '-append', kernel_command_line) >>>> + self.vm.launch() >>>> + console = self.vm.console_socket.makefile() >>>> + console_logger = logging.getLogger('console') >>>> + while True: >>>> + msg = console.readline() >>>> + console_logger.debug(msg.strip()) >>>> + if 'Kernel command line: %s' % kernel_command_line in msg: >>>> + break >>>> + if 'Kernel panic - not syncing' in msg: >>>> + self.fail("Kernel panic reached") >>> >>> Of course for bonus points a simple rootfs with hackbench or some such >>> in it would be nice. But I appreciate this makes the building job a lot >>> more complex than just a kernel. >> >> My idea is to use the rootfs for larger tests, and tag the "Kernel >> panic" tests as "quick", so we can have a "make acceptance-speed" or >> similar. >> >> We can already test than many devices were initialized correctly quickly >> looking at this console output. > > Yeah the kernel boot up is a pretty good smoke test (it's all most of > kernelci manages as well). However it certainly doesn't fully exercise > the translator. I've lost track of the number of bugs we found after > successfully booting an kernel because we were doing exhaustive > instruction testing. This is not indeed a translator test, but simple enough to test than devices are instantiated, mapped at the correct address, detected by Linux, probe the device enough to initialize it. I actually wrote this test after breaking a I/O device why a previous Super I/O refactor, this test would have failed. You could say a simple "info mtree" parser would notice it too... but acceptance testing is closer to real-world imho, and would be also useful to catch changes in the guest code (as in "QEMU expects things that way, but the Linux kernel updated to something new"). Regards, Phil.
Hi Alex, Cleber. On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote: > On 06/28/2018 03:36 PM, Alex Bennée wrote: >> Philippe Mathieu-Daudé <f4bug@amsat.org> writes: >>> On 06/28/2018 01:23 PM, Alex Bennée wrote: >>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes: >>>> >>>>> Similar to the BootLinuxConsoleX86_64 test: >>>>> boot a Linux kernel on a Malta board and verify the serial is working. >>>>> >>>>> This test can be run using: >>>>> >>>>> $ avocado run -t endian:big tests/acceptance >>>>> >>>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> >>>>> --- >>>>> tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++ >>>>> 1 file changed, 38 insertions(+) >>>>> >>>>> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py >>>>> index 17dc8d58c1..72cf5e943c 100644 >>>>> --- a/tests/acceptance/boot_linux_console.py >>>>> +++ b/tests/acceptance/boot_linux_console.py >>>>> @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test): >>>>> break >>>>> if 'Kernel panic - not syncing' in msg: >>>>> self.fail("Kernel panic reached") >>>>> + >>>>> + >>>>> +class BootLinuxConsoleMips(Test): >>>>> + """ >>>>> + Boots a mips Linux kernel and checks that the console is operational >>>>> + and the kernel command line is properly passed from QEMU to the kernel >>>>> + >>>>> + :avocado: enable >>>>> + :avocado: tags=endian:big >>>>> + :avocado: tags=arch:mips >>>>> + :avocado: tags=board:malta >>>>> + """ >>>>> + >>>>> + arch = "mips" >>>>> + timeout = 60 >>>>> + >>>>> + def test(self): >>>>> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' >>>>> + 'vmlinux-3.2.0-4-4kc-malta') >>>>> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' >>>>> + kernel_path = self.fetch_asset(kernel_url, >>>>> asset_hash=kernel_hash) >>>> >>>> I'm uncomfortable using "random" binaries of websites as the source of >>>> our test kernels. I can see the justification for distro kernels as they >>>> at least have the infrastructure to rebuild from source if you really >>>> want to, but even then the distros don't cover a lot of the >>>> architectures. Alex: I could find all the Linux kernel I'm interested to console-test with Avocado on the http://snapshot.debian.org/ archive website. For example Aurelien's one (more up-to-date) is available here: http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48 I also added a SH-4 test for the SM501 series of Zoltan BALATON using the kernel extracted from this distrib built kernel: http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30 The Debian distribution also provide the source package and the kernels can be simply rebuilt using make-kpkg or (make bindeb-pkg with more recent kernels). Would it be enough to satisfy the GPL requirements to provided that info in the header and use these handy pre-compiled kernels? Cleber: As you added tarballs and zip support, I'd also need a "dpkg" support in your fetch_asset() archive management... Cleber: And another future request is to be able to extract files from ISO images with the guestfish tool :P To be able to add a test for the Apple Machintosh Quadra 800 machine: http://lists.nongnu.org/archive/html/qemu-devel/2018-06/msg08139.html >>> And now I notice I made an mistake here :) I guess remember the Avocado >>> team started using the SHA-1 hash as default and I suggested them to be >>> able to use other hashes for this particular case, since Aurelien >>> provided the MD5 hashes signed by his GPG key, which is signed/trusted >>> by Peter and used to merge mips32 pulls. >>> >>> That would verify the QEMU community circle of trust right? >> >> It's not so much the chain of trust and more repeatability of building >> the test cases. I trust Aurel32's binaries are good test cases for MIPS >> but at the moment I have no idea how to rebuild them which is a bit of >> an issue given it is covered by the GPL. > > OK! I didn't even think about that since this is a "Linux" kernel and a > "Debian" rootfs provided by a Debian developer, hosted on a Debian > server, so I'm pretty sure this is GPL compliant, but it makes sens. > > I'll find a way to have reproducible test binaries for acceptance testing. > >>> I don't think Avocado should parse the FTP/HTTP signed indexes, but a >>> manual verification when merging this series should suffice. >>> >>>> >>>> I had experimented with using docker based builds for building test >>>> fixtures (see tests/docker/dockerbuilds): >>>> >>>> https://github.com/stsquad/qemu/tree/docker/linux-user-and-ltp-builds-v2 >>>> >>>> As these tests are fairly simple boot tests that just need kernels maybe >>>> we could look at tooling up the generation of these images in a >>>> repeatable way - similar to the way vmtest builds VMs. >>> >>> Yes, I have another acceptance branch where I cross-build an old mipssim >>> kernel to test the board, using the following: >>> http://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html >>> >>> But preparing a Docker cross image, fetching the Linux kernel source, >>> building it, takes a lot of time/storage I'd rather avoid; at least with >>> Aurelien kernels, since they are known to work since years. >> >> Well you can throw away the docker image as long as we have a mechanism >> to dump out the final binary. I have no interest in forcing people to >> keep checked out code using up space, just that they can re-build if >> they need to. > > The Avocado guys said they have plenty of storage for assets. > A cool feature would be to have some config built by some CI builder > that then sign and push reproducible config + generated binary to the > assets storage automatically. > >>>>> + >>>>> + self.vm.set_machine('malta') >>>>> + self.vm.set_console() >>>>> + kernel_command_line = 'console=ttyS0 printk.time=0' >>>>> + self.vm.add_args('-m', "64", >>>>> + '-kernel', kernel_path, >>>>> + '-append', kernel_command_line) >>>>> + self.vm.launch() >>>>> + console = self.vm.console_socket.makefile() >>>>> + console_logger = logging.getLogger('console') >>>>> + while True: >>>>> + msg = console.readline() >>>>> + console_logger.debug(msg.strip()) >>>>> + if 'Kernel command line: %s' % kernel_command_line in msg: >>>>> + break >>>>> + if 'Kernel panic - not syncing' in msg: >>>>> + self.fail("Kernel panic reached") >>>> >>>> Of course for bonus points a simple rootfs with hackbench or some such >>>> in it would be nice. But I appreciate this makes the building job a lot >>>> more complex than just a kernel. >>> >>> My idea is to use the rootfs for larger tests, and tag the "Kernel >>> panic" tests as "quick", so we can have a "make acceptance-speed" or >>> similar. >>> >>> We can already test than many devices were initialized correctly quickly >>> looking at this console output. >> >> Yeah the kernel boot up is a pretty good smoke test (it's all most of >> kernelci manages as well). However it certainly doesn't fully exercise >> the translator. I've lost track of the number of bugs we found after >> successfully booting an kernel because we were doing exhaustive >> instruction testing. > > This is not indeed a translator test, but simple enough to test than > devices are instantiated, mapped at the correct address, detected by > Linux, probe the device enough to initialize it. > > I actually wrote this test after breaking a I/O device why a previous > Super I/O refactor, this test would have failed. > > You could say a simple "info mtree" parser would notice it too... but > acceptance testing is closer to real-world imho, and would be also > useful to catch changes in the guest code (as in "QEMU expects things > that way, but the Linux kernel updated to something new"). > > Regards, > > Phil. >
On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote: > On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote: > > On 06/28/2018 03:36 PM, Alex Bennée wrote: > >> Philippe Mathieu-Daudé <f4bug@amsat.org> writes: > >>> On 06/28/2018 01:23 PM, Alex Bennée wrote: [...] > >>>>> + def test(self): > >>>>> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' > >>>>> + 'vmlinux-3.2.0-4-4kc-malta') > >>>>> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' > >>>>> + kernel_path = self.fetch_asset(kernel_url, > >>>>> asset_hash=kernel_hash) > >>>> > >>>> I'm uncomfortable using "random" binaries of websites as the source of > >>>> our test kernels. I can see the justification for distro kernels as they > >>>> at least have the infrastructure to rebuild from source if you really > >>>> want to, but even then the distros don't cover a lot of the > >>>> architectures. > > Alex: I could find all the Linux kernel I'm interested to console-test > with Avocado on the http://snapshot.debian.org/ archive website. > > For example Aurelien's one (more up-to-date) is available here: > http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48 > > I also added a SH-4 test for the SM501 series of Zoltan BALATON using > the kernel extracted from this distrib built kernel: > http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30 > > The Debian distribution also provide the source package and the kernels > can be simply rebuilt using make-kpkg or (make bindeb-pkg with more > recent kernels). > > Would it be enough to satisfy the GPL requirements to provided that info > in the header and use these handy pre-compiled kernels? (IANAL, etc.) Personally, I would try to avoid dealing with the "written offer" option of the GPL, and just publish the sources in the same medium (HTTP download repository, git repository, etc) as the binary. There was a related discussion about distribution of firmware binaries at <https://www.mail-archive.com/qemu-devel@nongnu.org/msg475674.html>. It looks like we never applied the docs/firmware.txt patch, though.
On 07/04/2018 05:47 PM, Eduardo Habkost wrote: > On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote: >> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote: >>> On 06/28/2018 03:36 PM, Alex Bennée wrote: >>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes: >>>>> On 06/28/2018 01:23 PM, Alex Bennée wrote: > [...] >>>>>>> + def test(self): >>>>>>> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' >>>>>>> + 'vmlinux-3.2.0-4-4kc-malta') >>>>>>> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' >>>>>>> + kernel_path = self.fetch_asset(kernel_url, >>>>>>> asset_hash=kernel_hash) >>>>>> >>>>>> I'm uncomfortable using "random" binaries of websites as the source of >>>>>> our test kernels. I can see the justification for distro kernels as they >>>>>> at least have the infrastructure to rebuild from source if you really >>>>>> want to, but even then the distros don't cover a lot of the >>>>>> architectures. >> >> Alex: I could find all the Linux kernel I'm interested to console-test >> with Avocado on the http://snapshot.debian.org/ archive website. >> >> For example Aurelien's one (more up-to-date) is available here: >> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48 >> >> I also added a SH-4 test for the SM501 series of Zoltan BALATON using >> the kernel extracted from this distrib built kernel: >> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30 >> >> The Debian distribution also provide the source package and the kernels >> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more >> recent kernels). >> >> Would it be enough to satisfy the GPL requirements to provided that info >> in the header and use these handy pre-compiled kernels? > > (IANAL, etc.) > > Personally, I would try to avoid dealing with the "written offer" > option of the GPL, and just publish the sources in the same > medium (HTTP download repository, git repository, etc) as the > binary. About Debian packaged kernel: the sources are in the same directory: http://snapshot.debian.org/package/linux-2.6/2.6.32-30/ linux-2.6_2.6.32.orig.tar.gz linux-image-2.6.32-5-sh7751r_2.6.32-30_sh4.deb linux-support-2.6.32-5_2.6.32-30_all.deb The other way, trying to compile a 2013 kernel is painful because the oldest GCC I succeed installing complain about many errors. The next absurd option I have is install a 2013 userland distrib and use qemu-user to cross^H^H^H^H(?? I dunno how to name it)-compile this kernel.... "backward-compile"? This might make sens with the coming quantum computers. > There was a related discussion about distribution of firmware > binaries at <https://www.mail-archive.com/qemu-devel@nongnu.org/msg475674.html>. > It looks like we never applied the docs/firmware.txt patch, > though. This case is not about firmware we need to run QEMU, but Linux kernel to run acceptance tests using QEMU. But same legal stuffs, you know...
On Wed, Jul 04, 2018 at 06:44:02PM -0300, Philippe Mathieu-Daudé wrote: > On 07/04/2018 05:47 PM, Eduardo Habkost wrote: > > On Wed, Jul 04, 2018 at 04:56:44PM -0300, Philippe Mathieu-Daudé wrote: > >> On 06/28/2018 07:45 PM, Philippe Mathieu-Daudé wrote: > >>> On 06/28/2018 03:36 PM, Alex Bennée wrote: > >>>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes: > >>>>> On 06/28/2018 01:23 PM, Alex Bennée wrote: > > [...] > >>>>>>> + def test(self): > >>>>>>> + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' > >>>>>>> + 'vmlinux-3.2.0-4-4kc-malta') > >>>>>>> + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' > >>>>>>> + kernel_path = self.fetch_asset(kernel_url, > >>>>>>> asset_hash=kernel_hash) > >>>>>> > >>>>>> I'm uncomfortable using "random" binaries of websites as the source of > >>>>>> our test kernels. I can see the justification for distro kernels as they > >>>>>> at least have the infrastructure to rebuild from source if you really > >>>>>> want to, but even then the distros don't cover a lot of the > >>>>>> architectures. > >> > >> Alex: I could find all the Linux kernel I'm interested to console-test > >> with Avocado on the http://snapshot.debian.org/ archive website. > >> > >> For example Aurelien's one (more up-to-date) is available here: > >> http://snapshot.debian.org/package/linux-2.6/2.6.32-48/#linux-image-2.6.32-5-4kc-malta_2.6.32-48 > >> > >> I also added a SH-4 test for the SM501 series of Zoltan BALATON using > >> the kernel extracted from this distrib built kernel: > >> http://snapshot.debian.org/package/linux-2.6/2.6.32-30/#linux-image-2.6.32-5-sh7751r_2.6.32-30 > >> > >> The Debian distribution also provide the source package and the kernels > >> can be simply rebuilt using make-kpkg or (make bindeb-pkg with more > >> recent kernels). > >> > >> Would it be enough to satisfy the GPL requirements to provided that info > >> in the header and use these handy pre-compiled kernels? > > > > (IANAL, etc.) > > > > Personally, I would try to avoid dealing with the "written offer" > > option of the GPL, and just publish the sources in the same > > medium (HTTP download repository, git repository, etc) as the > > binary. > > About Debian packaged kernel: the sources are in the same directory: > > http://snapshot.debian.org/package/linux-2.6/2.6.32-30/ > > linux-2.6_2.6.32.orig.tar.gz > linux-image-2.6.32-5-sh7751r_2.6.32-30_sh4.deb > linux-support-2.6.32-5_2.6.32-30_all.deb Oh, I thought you were talking about publishing the binaries elsewhere. If we're not the ones publishing the binaries (we're just downloading it), then this is somebody else's problem. :) Would it be OK to make test code download directly from snapshot.debian.org, though? Their home page have instructions to use apt repositories there, so I guess it's reasonable use of the service.
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py index 17dc8d58c1..72cf5e943c 100644 --- a/tests/acceptance/boot_linux_console.py +++ b/tests/acceptance/boot_linux_console.py @@ -46,3 +46,41 @@ class BootLinuxConsoleX86_64(Test): break if 'Kernel panic - not syncing' in msg: self.fail("Kernel panic reached") + + +class BootLinuxConsoleMips(Test): + """ + Boots a mips Linux kernel and checks that the console is operational + and the kernel command line is properly passed from QEMU to the kernel + + :avocado: enable + :avocado: tags=endian:big + :avocado: tags=arch:mips + :avocado: tags=board:malta + """ + + arch = "mips" + timeout = 60 + + def test(self): + kernel_url = ('http://people.debian.org/~aurel32/qemu/mips/' + 'vmlinux-3.2.0-4-4kc-malta') + kernel_hash = '592e384a4edc16dade52a6cd5c785c637bcbc9ad' + kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + + self.vm.set_machine('malta') + self.vm.set_console() + kernel_command_line = 'console=ttyS0 printk.time=0' + self.vm.add_args('-m', "64", + '-kernel', kernel_path, + '-append', kernel_command_line) + self.vm.launch() + console = self.vm.console_socket.makefile() + console_logger = logging.getLogger('console') + while True: + msg = console.readline() + console_logger.debug(msg.strip()) + if 'Kernel command line: %s' % kernel_command_line in msg: + break + if 'Kernel panic - not syncing' in msg: + self.fail("Kernel panic reached")
Similar to the BootLinuxConsoleX86_64 test: boot a Linux kernel on a Malta board and verify the serial is working. This test can be run using: $ avocado run -t endian:big tests/acceptance Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org> --- tests/acceptance/boot_linux_console.py | 38 ++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)