diff mbox series

[v4,20/30] tests/avocado: disable sh4 rd2 tests on Gitlab

Message ID 20221027183637.2772968-21-alex.bennee@linaro.org
State New
Headers show
Series testing/next pre-PR | expand

Commit Message

Alex Bennée Oct. 27, 2022, 6:36 p.m. UTC 
Running repeated invocations on a number of test boxes show a fairly
high error rate:

  $ retry.py -n 100 -c -- ./tests/venv/bin/avocado run tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d
  retry.py called with ['./tests/venv/bin/avocado', 'run', 'tests/avocado/boot_linux_console.py:BootLinuxConsole.test_sh4_r2d']
  Results:
  Run, Ret, Pass/Fail, Time, Total Pass, Total Run
  ...
  Results summary:
  0: 94 times (94.00%), avg time 2.254 (0.00 varience/0.04 deviation)
  1: 3 times (3.00%), avg time 1.837 (0.02 varience/0.14 deviation)
  8: 3 times (3.00%), avg time 91.288 (0.02 varience/0.15 deviation)

Examining the logs they fall into various categories of un-handled
unaligned access by user space and unexpected FPU usage by the kernel
which ultimately lead to the failure to reach the login prompt. This
could be bugs in the translator that only get hit occasionally or just
a flaky kernel - its hard to tell. To avoid these failures gating CI
lets skip on GitLab.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Thomas Huth <thuth@redhat.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Magnus Damm <magnus.damm@gmail.com>

---
v2
  - add import for skipIf
---
 tests/avocado/boot_linux_console.py | 5 +++++
 1 file changed, 5 insertions(+)
diff mbox series

Patch

diff --git a/tests/avocado/boot_linux_console.py b/tests/avocado/boot_linux_console.py
index eed4b49e6e..4c9d551f47 100644
--- a/tests/avocado/boot_linux_console.py
+++ b/tests/avocado/boot_linux_console.py
@@ -15,6 +15,7 @@ 
 
 from avocado import skip
 from avocado import skipUnless
+from avocado import skipIf
 from avocado_qemu import QemuSystemTest
 from avocado_qemu import exec_command
 from avocado_qemu import exec_command_and_wait_for_pattern
@@ -1175,6 +1176,10 @@  def test_ppc_mac99(self):
         self.vm.add_args('-M', 'graphics=off')
         self.do_test_advcal_2018('15', tar_hash, 'invaders.elf')
 
+    # This test has a 6-10% failure rate on various hosts that look
+    # like issues with a buggy kernel. As a result we don't want it
+    # gating releases on Gitlab.
+    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
     def test_sh4_r2d(self):
         """
         :avocado: tags=arch:sh4