| Message ID | 20190117185628.21862-15-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | Acceptance Tests: target architecture support | expand |
On Thu, Jan 17, 2019 at 01:56:24PM -0500, Cleber Rosa wrote: > Just like the previous tests, boots a Linux kernel on a ppc64 target > using the pseries machine. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> Reviewed-by: Caio Carrara <ccarrara@redhat.com> > --- > .travis.yml | 2 +- > scripts/qemu.py | 1 - > tests/acceptance/boot_linux_console.py | 19 +++++++++++++++++++ > 3 files changed, 20 insertions(+), 2 deletions(-) > {...} >
Cleber Rosa <crosa@redhat.com> writes: > Just like the previous tests, boots a Linux kernel on a ppc64 target > using the pseries machine. So running this on my rather slow SynQuacer I get: (04/16) /home/alex/lsrc/qemu.git/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_ppc64_pseries: INTERRUPTED: Test reported status but did not finish\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n{'name': '04-/home/alex/lsrc/qemu.git/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_ppc64_pseries', 'logdir': '/home/alex/lsrc/qemu.git/te... (60.93 s) which I'm guessing is a timeout occurring. I wonder if that means we should: a) set timeouts longer for when running on TCG or b) split tests into TCG and KVM tests and select KVM tests on appropriate HW The qemu.py code has (slightly flawed) logic for detecting KVM and passing --enable-kvm. Maybe we should be doing that here? > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > .travis.yml | 2 +- > scripts/qemu.py | 1 - > tests/acceptance/boot_linux_console.py | 19 +++++++++++++++++++ > 3 files changed, 20 insertions(+), 2 deletions(-) > > diff --git a/.travis.yml b/.travis.yml > index 28648f7a61..54100eea5a 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -187,7 +187,7 @@ matrix: > > # Acceptance (Functional) tests > - env: > - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu" > + - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu" > - TEST_CMD="make check-acceptance" > addons: > apt: > diff --git a/scripts/qemu.py b/scripts/qemu.py > index ef84b0f843..1531e28fc1 100644 > --- a/scripts/qemu.py > +++ b/scripts/qemu.py > @@ -37,7 +37,6 @@ CONSOLE_DEV_TYPES = { > r'^clipper$': 'isa-serial', > r'^(pc.*|q35.*|isapc)$': 'isa-serial', > r'^(40p|powernv|prep)$': 'isa-serial', > - r'^pseries.*': 'spapr-vty', > r'^s390-ccw-virtio.*': 'sclpconsole', > } > > diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py > index 20b845fce1..f3ccd23a7a 100644 > --- a/tests/acceptance/boot_linux_console.py > +++ b/tests/acceptance/boot_linux_console.py > @@ -119,3 +119,22 @@ class BootLinuxConsole(Test): > self.vm.launch() > console_pattern = 'Kernel command line: %s' % kernel_command_line > self.wait_for_console_pattern(console_pattern) > + > + def test_ppc64_pseries(self): > + """ > + :avocado: tags=arch:ppc64 > + :avocado: tags=machine:pseries > + """ > + kernel_url = ('http://mirrors.rit.edu/fedora/fedora-secondary/' > + 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz') > + kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77' > + kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) > + > + self.vm.set_machine('pseries') > + self.vm.set_console() > + kernel_command_line = 'console=hvc0' > + self.vm.add_args('-kernel', kernel_path, > + '-append', kernel_command_line) > + self.vm.launch() > + console_pattern = 'Kernel command line: %s' % kernel_command_line > + self.wait_for_console_pattern(console_pattern) -- Alex Bennée
On 1/22/19 11:07 AM, Alex Bennée wrote: > > Cleber Rosa <crosa@redhat.com> writes: > >> Just like the previous tests, boots a Linux kernel on a ppc64 target >> using the pseries machine. > > So running this on my rather slow SynQuacer I get: > > (04/16) /home/alex/lsrc/qemu.git/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_ppc64_pseries: INTERRUPTED: Test reported status but did not finish\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n{'name': '04-/home/alex/lsrc/qemu.git/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_ppc64_pseries', 'logdir': '/home/alex/lsrc/qemu.git/te... (60.93 s) > > which I'm guessing is a timeout occurring. > Yes, that's what's happening. It's hard to pinpoint, and control, the sluggishness points in such a test running on a different environment. For this one execution, I do trust your assessment, and it's most certainly caused by your "slow SynQuacer", spending too much running emulation code. But, I'd like to mention that there are other possibilities. One is that you're hitting a "asset fetcher bug" that I recently fixed in Avocado[1] (fix to be available on 68.0, to be released next Monday, Feb 4th). Even with that bug fixed, I feel like it's unfair to test code to spend its time waiting to download a file when it's not testing *the file download itself*. Because of that, there are plans to add an (optional) job pre-processing step that will make sure the needed assets are in the cache ahead of time[2][3]. > I wonder if that means we should: > > a) set timeouts longer for when running on TCG > or > b) split tests into TCG and KVM tests and select KVM tests on appropriate HW > I wonder the same, and I believe this falls into a similar scenario we've seen with the setup of console devices in the QEMUMachine class. I've started by setting the device types defined at the framework level, and then reverted to the machine's default devices (using '-serial'), because the "default" behavior of QEMU is usually what a test writer wants when not setting something else explicitly. > The qemu.py code has (slightly flawed) logic for detecting KVM and > passing --enable-kvm. Maybe we should be doing that here? > I'm not sure. IMO, the common question is: should we magically (at a framework level) configure tests based on probed host environment characteristics? I feel like we should attempt to minimize that for the sake of tests being more obvious and more easily reproducible. And because of that, I'd go, *initially*, with an approach more similar to your option "b". Having said that, we don't want to rewrite most tests just to be able to test with either KVM or TCG, if the tests are not explicitly testing KVM or TCG. At this point, using KVM or TCG is test/framework *configuration*, and in Avocado we hope to solve this by having the executed tests easily identifiable and reproducible (a test ID will contain a information about the options passed, and a replay of the job will apply the same configuration). For now, I think the best approach is to increase the timeout, because I think it's much worse to have to deal with false negatives (longer execution times that don't really mean a failure), than having a test possibly taking some more time to finish. And sorry for extremely the long answer! - Cleber. [1] - https://github.com/avocado-framework/avocado/pull/2996 [2] - https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring [3] - https://trello.com/c/CKP7YS6G/1481-on-cache-check-for-asset-fetcher
On 1/31/19 3:37 AM, Cleber Rosa wrote: > On 1/22/19 11:07 AM, Alex Bennée wrote: >> Cleber Rosa <crosa@redhat.com> writes: >> >>> Just like the previous tests, boots a Linux kernel on a ppc64 target >>> using the pseries machine. >> >> So running this on my rather slow SynQuacer I get: >> >> (04/16) /home/alex/lsrc/qemu.git/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_ppc64_pseries: INTERRUPTED: Test reported status but did not finish\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n{'name': '04-/home/alex/lsrc/qemu.git/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_ppc64_pseries', 'logdir': '/home/alex/lsrc/qemu.git/te... (60.93 s) >> >> which I'm guessing is a timeout occurring. >> > > Yes, that's what's happening. It's hard to pinpoint, and control, the > sluggishness points in such a test running on a different environment. > For this one execution, I do trust your assessment, and it's most > certainly caused by your "slow SynQuacer", spending too much running > emulation code. > > But, I'd like to mention that there are other possibilities. One is > that you're hitting a "asset fetcher bug" that I recently fixed in > Avocado[1] (fix to be available on 68.0, to be released next Monday, Feb > 4th). > > Even with that bug fixed, I feel like it's unfair to test code to spend > its time waiting to download a file when it's not testing *the file > download itself*. Because of that, there are plans to add an (optional) > job pre-processing step that will make sure the needed assets are in the > cache ahead of time[2][3]. > >> I wonder if that means we should: >> >> a) set timeouts longer for when running on TCG I hit the same problem with VM tests, and suggested a poor "increase timeout" patch [1]. Then Peter sent a different patch [2] which happens to inadvertently Dictionary resolve my problem, since the longer a VM took to boot on the Cavium ThunderX I have access is 288 seconds, which is closely below the 300 seconds limit =) I understood nobody seemed to really care about testing the x86 TCG backend this way, so I didn't worry much. [1] http://lists.nongnu.org/archive/html/qemu-devel/2018-07/msg03416.html [2] http://lists.nongnu.org/archive/html/qemu-devel/2018-08/msg04977.html >> or >> b) split tests into TCG and KVM tests and select KVM tests on appropriate HW >> > > I wonder the same, and I believe this falls into a similar scenario > we've seen with the setup of console devices in the QEMUMachine class. > I've started by setting the device types defined at the framework level, > and then reverted to the machine's default devices (using '-serial'), > because the "default" behavior of QEMU is usually what a test writer > wants when not setting something else explicitly. > >> The qemu.py code has (slightly flawed) logic for detecting KVM and >> passing --enable-kvm. Maybe we should be doing that here? >> > > I'm not sure. IMO, the common question is: should we magically (at a > framework level) configure tests based on probed host environment > characteristics? I feel like we should attempt to minimize that for the > sake of tests being more obvious and more easily reproducible. I agree we shouldn't randomly test different features, but rather explicitly add 2 tests (TCG/KVM), and if it is not possible to run a test, mark it as SKIPPED. An user with KVM available would then have to run --filter-out=tcg, or build QEMU with --disable-tcg. > And because of that, I'd go, *initially*, with an approach more similar > to your option "b". > > Having said that, we don't want to rewrite most tests just to be able to > test with either KVM or TCG, if the tests are not explicitly testing KVM > or TCG. At this point, using KVM or TCG is test/framework > *configuration*, and in Avocado we hope to solve this by having the > executed tests easily identifiable and reproducible (a test ID will > contain a information about the options passed, and a replay of the job > will apply the same configuration). > > For now, I think the best approach is to increase the timeout, because I > think it's much worse to have to deal with false negatives (longer > execution times that don't really mean a failure), than having a test > possibly taking some more time to finish. > > And sorry for extremely the long answer! > - Cleber. > > [1] - https://github.com/avocado-framework/avocado/pull/2996 > [2] - > https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring > [3] - https://trello.com/c/CKP7YS6G/1481-on-cache-check-for-asset-fetcher >
diff --git a/.travis.yml b/.travis.yml index 28648f7a61..54100eea5a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -187,7 +187,7 @@ matrix: # Acceptance (Functional) tests - env: - - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu" + - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,ppc64-softmmu" - TEST_CMD="make check-acceptance" addons: apt: diff --git a/scripts/qemu.py b/scripts/qemu.py index ef84b0f843..1531e28fc1 100644 --- a/scripts/qemu.py +++ b/scripts/qemu.py @@ -37,7 +37,6 @@ CONSOLE_DEV_TYPES = { r'^clipper$': 'isa-serial', r'^(pc.*|q35.*|isapc)$': 'isa-serial', r'^(40p|powernv|prep)$': 'isa-serial', - r'^pseries.*': 'spapr-vty', r'^s390-ccw-virtio.*': 'sclpconsole', } diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py index 20b845fce1..f3ccd23a7a 100644 --- a/tests/acceptance/boot_linux_console.py +++ b/tests/acceptance/boot_linux_console.py @@ -119,3 +119,22 @@ class BootLinuxConsole(Test): self.vm.launch() console_pattern = 'Kernel command line: %s' % kernel_command_line self.wait_for_console_pattern(console_pattern) + + def test_ppc64_pseries(self): + """ + :avocado: tags=arch:ppc64 + :avocado: tags=machine:pseries + """ + kernel_url = ('http://mirrors.rit.edu/fedora/fedora-secondary/' + 'releases/29/Everything/ppc64le/os/ppc/ppc64/vmlinuz') + kernel_hash = '3fe04abfc852b66653b8c3c897a59a689270bc77' + kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash) + + self.vm.set_machine('pseries') + self.vm.set_console() + kernel_command_line = 'console=hvc0' + self.vm.add_args('-kernel', kernel_path, + '-append', kernel_command_line) + self.vm.launch() + console_pattern = 'Kernel command line: %s' % kernel_command_line + self.wait_for_console_pattern(console_pattern)
Just like the previous tests, boots a Linux kernel on a ppc64 target using the pseries machine. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- .travis.yml | 2 +- scripts/qemu.py | 1 - tests/acceptance/boot_linux_console.py | 19 +++++++++++++++++++ 3 files changed, 20 insertions(+), 2 deletions(-)