| Message ID | 20190221005753.27955-10-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | Acceptance Tests: target architecture support | expand |
On Wed, 20 Feb 2019 19:57:42 -0500 Cleber Rosa <crosa@redhat.com> wrote: > To the stock Fedora 29 kernel, from the Fedora 28. New tests will be > added using the 29 kernel, so for consistency, let's also update it > here. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > Reviewed-by: Caio Carrara <ccarrara@redhat.com> > --- > tests/acceptance/boot_linux_console.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py > index 89df7f6e4f..35b31162d4 100644 > --- a/tests/acceptance/boot_linux_console.py > +++ b/tests/acceptance/boot_linux_console.py > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test): > :avocado: tags=arch:x86_64 > :avocado: tags=machine:pc > """ > - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/' > + kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/' > 'Everything/x86_64/os/images/pxeboot/vmlinuz') > - kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a' > + kernel_hash = '23bebd2680757891cf7adedb033532163a792495' > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) > > self.vm.set_machine('pc') This looks reasonable; but an obvious follow-up question is how we handle updates to new distro levels in general. I don't think we'll need to run acceptance tests on old qemus, though.
On 2/21/19 1:57 AM, Cleber Rosa wrote: > To the stock Fedora 29 kernel, from the Fedora 28. New tests will be [Update] to ... ? > added using the 29 kernel, so for consistency, let's also update it > here. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > Reviewed-by: Caio Carrara <ccarrara@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> > --- > tests/acceptance/boot_linux_console.py | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py > index 89df7f6e4f..35b31162d4 100644 > --- a/tests/acceptance/boot_linux_console.py > +++ b/tests/acceptance/boot_linux_console.py > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test): > :avocado: tags=arch:x86_64 > :avocado: tags=machine:pc > """ > - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/' > + kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/' > 'Everything/x86_64/os/images/pxeboot/vmlinuz') > - kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a' > + kernel_hash = '23bebd2680757891cf7adedb033532163a792495' > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) > > self.vm.set_machine('pc') >
On Fri, Mar 01, 2019 at 11:41:44AM +0100, Cornelia Huck wrote: > On Wed, 20 Feb 2019 19:57:42 -0500 > Cleber Rosa <crosa@redhat.com> wrote: > > > To the stock Fedora 29 kernel, from the Fedora 28. New tests will be > > added using the 29 kernel, so for consistency, let's also update it > > here. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > Reviewed-by: Caio Carrara <ccarrara@redhat.com> > > --- > > tests/acceptance/boot_linux_console.py | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py > > index 89df7f6e4f..35b31162d4 100644 > > --- a/tests/acceptance/boot_linux_console.py > > +++ b/tests/acceptance/boot_linux_console.py > > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test): > > :avocado: tags=arch:x86_64 > > :avocado: tags=machine:pc > > """ > > - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/' > > + kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/' > > 'Everything/x86_64/os/images/pxeboot/vmlinuz') > > - kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a' > > + kernel_hash = '23bebd2680757891cf7adedb033532163a792495' > > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) > > > > self.vm.set_machine('pc') > > This looks reasonable; but an obvious follow-up question is how we > handle updates to new distro levels in general. > > I don't think we'll need to run acceptance tests on old qemus, though. No, I don't think we have to bother with that. Backport(er)s would be expected to handle this (if they ever come to existence). You do have a point about consistenly updating those images, though. I won't pretend to have an answer for that just yet... but it's possible that these "kernel_urls" can be turned into parameters that will eventually be expand based on the user's selected distro and version (similar to the Avocado vmimage library[1]). - Cleber. [1] https://avocado-framework.readthedocs.io/en/69.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
On Sat, Mar 02, 2019 at 10:42:13PM +0100, Philippe Mathieu-Daudé wrote: > On 2/21/19 1:57 AM, Cleber Rosa wrote: > > To the stock Fedora 29 kernel, from the Fedora 28. New tests will be > > [Update] to ... ? > Right... I used this line as a continuation of the commit title, but probably shouldn't have done that. > > added using the 29 kernel, so for consistency, let's also update it > > here. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > Reviewed-by: Caio Carrara <ccarrara@redhat.com> > > Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> > Thanks! - Cleber. > > --- > > tests/acceptance/boot_linux_console.py | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py > > index 89df7f6e4f..35b31162d4 100644 > > --- a/tests/acceptance/boot_linux_console.py > > +++ b/tests/acceptance/boot_linux_console.py > > @@ -28,9 +28,9 @@ class BootLinuxConsole(Test): > > :avocado: tags=arch:x86_64 > > :avocado: tags=machine:pc > > """ > > - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/' > > + kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/' > > 'Everything/x86_64/os/images/pxeboot/vmlinuz') > > - kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a' > > + kernel_hash = '23bebd2680757891cf7adedb033532163a792495' > > kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) > > > > self.vm.set_machine('pc') > >
Hi, > possible that these "kernel_urls" can be turned into parameters > that will eventually be expand based on the user's selected distro > and version (similar to the Avocado vmimage library[1]). Speaking of distros ... I'd like automate some of the graphics testing. For starters boot guest, run test utility to display something, take a screendump, check whenever it actually shows what I expect it to show. Maybe later run test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/. Ideally it would allow to not only test qemu, but to also test the kernel drivers (i.e. allows to boot and test with some custom development kernel). For that I need a bit more than just a kernel. Suggestions how to do that best with autotest? Fetch distro image, install/compile what I need, cache the image somehow, then use it to run the tests? Would the vmimage library allow to do that? From a brief look it seems to only support fetching the images, not customizing them ... Create an appliance using supermin? Create an special initrd using dracut? Something else? thanks, Gerd
On Tue, Mar 12, 2019 at 10:35:15AM +0100, Gerd Hoffmann wrote: > Hi, > > > possible that these "kernel_urls" can be turned into parameters > > that will eventually be expand based on the user's selected distro > > and version (similar to the Avocado vmimage library[1]). > > Speaking of distros ... > > I'd like automate some of the graphics testing. For starters boot > guest, run test utility to display something, take a screendump, check > whenever it actually shows what I expect it to show. Maybe later run > test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/. > > Ideally it would allow to not only test qemu, but to also test the > kernel drivers (i.e. allows to boot and test with some custom > development kernel). > > For that I need a bit more than just a kernel. Suggestions how to do > that best with autotest? > > Fetch distro image, install/compile what I need, cache the image > somehow, then use it to run the tests? Would the vmimage library allow > to do that? From a brief look it seems to only support fetching the > images, not customizing them ... > Do you feel like most of the customization would be done constantly? I'm trying to grasp if it would make sense to have an "almost ready" image, and adding a final level of customization (adding freshly built drivers?) on top of that. If that sounds like a possibility, than it should be possible to host your images somewhere[1] and let the vmimage library get and cache them (you can expand the list of providers). Then, when it comes to the customization part, do you expect your running environments to have a "reasonable" level of high level tools such as SSH and Python? The reason I ask is, as much as it sounds like overkill, tools like ansible (that depend on both of these in the managed system) can take a considerable amount of complexity out of test. In my experience it's only when tools like ansible are used as programming languages that everything starts to crumble and fall apart. In the context of "acceptance" tests, an "ansible" dependency can be expressed as a requirement on "tests/requirements.txt" so the bootstrap should not be a problem. Then, I'd really like to add some level of integration between the: * the avocado.utils.vmimage library * the avocado.utils.cloudinit library * the avocado.utils.ssh library * the QEMUMachine instances we spawn * running an ansible playbook on the previous Brainstorming about it: def setUp(self): with self.get_vm() as vm: vm.add_drive(utils.vmimage.get("mydistro")) vm.add_drive(utils.cloudinit.iso()) vm.start() utils.ansible.run(vm, "graphic_drivers.yml") # failure to get here can CANCEL the test def test(self): with utils.ssh.SSHSession(self.get_vm("default_vm")) as ssh: ... > Create an appliance using supermin? > > Create an special initrd using dracut? > If your systems are not expected to contain higher level tools, than, I'm afraid those are good options indeed. > Something else? > > thanks, > Gerd > PS: I have strong feelings about what a test should do, and that should not include any kind of setup (including downloading files it'll use). This is somehow loosely expressed in work I intend to do at a given point[2]. [1] - There's lot of unused disk space and bandwidth available in the "avocado-project.org" server. [2] - https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring
Hi, > > I'd like automate some of the graphics testing. For starters boot > > guest, run test utility to display something, take a screendump, check > > whenever it actually shows what I expect it to show. Maybe later run > > test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/. > > > > Ideally it would allow to not only test qemu, but to also test the > > kernel drivers (i.e. allows to boot and test with some custom > > development kernel). > > > > For that I need a bit more than just a kernel. Suggestions how to do > > that best with autotest? > > > > Fetch distro image, install/compile what I need, cache the image > > somehow, then use it to run the tests? Would the vmimage library allow > > to do that? From a brief look it seems to only support fetching the > > images, not customizing them ... > > Do you feel like most of the customization would be done constantly? For qemu testing no. For linux kernel testing yes. > I'm trying to grasp if it would make sense to have an "almost ready" > image, and adding a final level of customization (adding freshly built > drivers?) on top of that. > > If that sounds like a possibility, than it should be possible to host > your images somewhere[1] and let the vmimage library get and cache them > (you can expand the list of providers). Hmm. Put kernel testing aside for now. I don't think it is that an good idea to host a bunch of custom images just because some test tools are needed for some test. The cloud images are typically pretty stripped down, so when using them for testing the need to install additional is probably not rare. Be it my graphics test tools, or usbutils for usb testing, or something else network/block testing. I think it would make sense to support creating and caching such custom images, i.e. download standard fedora image, install test tools (fetch rpms or compile stuff), then put that image into a local cache and use it for test runs. > Then, when it comes to the customization part, do you expect your > running environments to have a "reasonable" level of high level tools > such as SSH and Python? Wouldn't hurt, but isn't required. At least initially. When maybe running a whole gfx test suite inside the guest some day it'll be a different story. That'll probably out of scope for acceptance tests, but using avocado might still be useful for that. > PS: I have strong feelings about what a test should do, and that should not > include any kind of setup (including downloading files it'll use). This > is somehow loosely expressed in work I intend to do at a given point[2]. Yep. Setup and test run should be separate, so you setup once and don't fetch stuff from the Internet for each test run. cheers, Gerd
diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py index 89df7f6e4f..35b31162d4 100644 --- a/tests/acceptance/boot_linux_console.py +++ b/tests/acceptance/boot_linux_console.py @@ -28,9 +28,9 @@ class BootLinuxConsole(Test): :avocado: tags=arch:x86_64 :avocado: tags=machine:pc """ - kernel_url = ('https://mirrors.kernel.org/fedora/releases/28/' + kernel_url = ('https://mirrors.kernel.org/fedora/releases/29/' 'Everything/x86_64/os/images/pxeboot/vmlinuz') - kernel_hash = '238e083e114c48200f80d889f7e32eeb2793e02a' + kernel_hash = '23bebd2680757891cf7adedb033532163a792495' kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) self.vm.set_machine('pc')