| Message ID | 20231116115354.228678-3-npiggin@gmail.com |
|---|---|
| State | New |
| Headers | show |
| Series | [1/3] tests/avocado: reverse_debugging drain console to prevent hang | expand |
On 16/11/2023 12.53, Nicholas Piggin wrote: > Let's try enable reverse_debugging.py in gitlab CI. > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> > --- > Maybe we could try this again at some point? The bug might have been > noticed sooner. > > They only take a couple of seconds to run so should not take too much > overhead. But my gitlab CI pipeline doesn't run the avocado tests for > some reason, so I can't see if it's still causing problems. Go to Settings -> CI/CD -> Variables in your gitlab project, and create a QEMU_CI_AVOCADO_TESTING variable there, then the jobs should show up next time. It's mentioned in the documentation in docs/devel/ci-jobs.rst.inc ... but I have to admit that it's easy to miss. Thomas
On 16/11/2023 12.53, Nicholas Piggin wrote: > Let's try enable reverse_debugging.py in gitlab CI. > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> > --- > Maybe we could try this again at some point? The bug might have been > noticed sooner. > > They only take a couple of seconds to run so should not take too much > overhead. But my gitlab CI pipeline doesn't run the avocado tests for > some reason, so I can't see if it's still causing problems. > > Thanks, > Nick > --- > > tests/avocado/reverse_debugging.py | 7 ------- > 1 file changed, 7 deletions(-) FYI, I gave it a try, and it survived my CI run: https://gitlab.com/thuth/qemu/-/jobs/5552213972#L403 So I went ahead and put it (together with the first patch) in my current pull request, let's see how it goes... (for the second patch, I hope our chardev maintainer Marc-André could comment on that first) Thomas
On Fri Nov 17, 2023 at 4:11 AM AEST, Thomas Huth wrote: > On 16/11/2023 12.53, Nicholas Piggin wrote: > > Let's try enable reverse_debugging.py in gitlab CI. > > > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> > > --- > > Maybe we could try this again at some point? The bug might have been > > noticed sooner. > > > > They only take a couple of seconds to run so should not take too much > > overhead. But my gitlab CI pipeline doesn't run the avocado tests for > > some reason, so I can't see if it's still causing problems. > > > > Thanks, > > Nick > > --- > > > > tests/avocado/reverse_debugging.py | 7 ------- > > 1 file changed, 7 deletions(-) > > FYI, I gave it a try, and it survived my CI run: > > https://gitlab.com/thuth/qemu/-/jobs/5552213972#L403 > > So I went ahead and put it (together with the first patch) in my current > pull request, let's see how it goes... Great, thank you. > (for the second patch, I hope our chardev maintainer Marc-André could > comment on that first) Yeah that would be good, no rush for that one. Maybe it's not desirable for other reasons too. Thanks, Nick
On 17/11/2023 08.35, Nicholas Piggin wrote: > On Fri Nov 17, 2023 at 4:11 AM AEST, Thomas Huth wrote: >> On 16/11/2023 12.53, Nicholas Piggin wrote: >>> Let's try enable reverse_debugging.py in gitlab CI. >>> >>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> >>> --- >>> Maybe we could try this again at some point? The bug might have been >>> noticed sooner. >>> >>> They only take a couple of seconds to run so should not take too much >>> overhead. But my gitlab CI pipeline doesn't run the avocado tests for >>> some reason, so I can't see if it's still causing problems. >>> >>> Thanks, >>> Nick >>> --- >>> >>> tests/avocado/reverse_debugging.py | 7 ------- >>> 1 file changed, 7 deletions(-) >> >> FYI, I gave it a try, and it survived my CI run: >> >> https://gitlab.com/thuth/qemu/-/jobs/5552213972#L403 >> >> So I went ahead and put it (together with the first patch) in my current >> pull request, let's see how it goes... > > Great, thank you. ... and here it's failing again (current master branch): https://gitlab.com/thuth/qemu/-/jobs/5582657378#L404 According to the debug.log in the artifacts, it's failing here: 08:28:32 DEBUG| [ 0.230392217,5] OPAL v7.0 starting... 08:28:32 DEBUG| [ 0.230674939,7] initial console log level: memory 7, driver 5 08:28:32 DEBUG| [ 0.231048494,6] CPU: P9 generation processor (max 4 threads/core) 08:28:32 DEBUG| [ 08:28:32 DEBUG| [ 0.231412547,7] CPU: Boot CPU PIR is 0x0000 PVR is 0x004e1202 08:28:32 DEBUG| [ 08:28:32 ERROR| 08:28:32 ERROR| Reproduced traceback from: /builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/core/test.py:770 08:28:32 ERROR| Traceback (most recent call last): 08:28:32 ERROR| File "/builds/thuth/qemu/build/tests/avocado/reverse_debugging.py", line 262, in test_ppc64_powernv 08:28:32 ERROR| self.reverse_debugging() 08:28:32 ERROR| File "/builds/thuth/qemu/build/tests/avocado/reverse_debugging.py", line 178, in reverse_debugging 08:28:32 ERROR| g.cmd(b'c') 08:28:32 ERROR| File "/builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/utils/gdb.py", line 783, in cmd 08:28:32 ERROR| response_payload = self.decode(result) 08:28:32 ERROR| File "/builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/utils/gdb.py", line 738, in decode 08:28:32 ERROR| raise InvalidPacketError 08:28:32 ERROR| avocado.utils.gdb.InvalidPacketError 08:28:32 ERROR| 08:28:32 DEBUG| Local variables: 08:28:32 DEBUG| -> self <class 'reverse_debugging.ReverseDebugging_ppc64'>: 79-tests/avocado/reverse_debugging.py:ReverseDebugging_ppc64.test_ppc64_powernv 08:28:32 DEBUG| Shutting down VM appliance; timeout=30 08:28:32 DEBUG| Attempting graceful termination 08:28:32 DEBUG| Closing console socket 08:28:32 DEBUG| Politely asking QEMU to terminate So unless someone has a clue how to fix that, I guess it's likely best to revert this enablement patch again... Thomas
On Tue, Nov 21, 2023 at 09:56:24AM +0100, Thomas Huth wrote: > On 17/11/2023 08.35, Nicholas Piggin wrote: > > On Fri Nov 17, 2023 at 4:11 AM AEST, Thomas Huth wrote: > > > On 16/11/2023 12.53, Nicholas Piggin wrote: > > > > Let's try enable reverse_debugging.py in gitlab CI. > > > > > > > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com> > > > > --- > > > > Maybe we could try this again at some point? The bug might have been > > > > noticed sooner. > > > > > > > > They only take a couple of seconds to run so should not take too much > > > > overhead. But my gitlab CI pipeline doesn't run the avocado tests for > > > > some reason, so I can't see if it's still causing problems. > > > > > > > > Thanks, > > > > Nick > > > > --- > > > > > > > > tests/avocado/reverse_debugging.py | 7 ------- > > > > 1 file changed, 7 deletions(-) > > > > > > FYI, I gave it a try, and it survived my CI run: > > > > > > https://gitlab.com/thuth/qemu/-/jobs/5552213972#L403 > > > > > > So I went ahead and put it (together with the first patch) in my current > > > pull request, let's see how it goes... > > > > Great, thank you. > > ... and here it's failing again (current master branch): > > https://gitlab.com/thuth/qemu/-/jobs/5582657378#L404 > > According to the debug.log in the artifacts, it's failing here: > > 08:28:32 DEBUG| [ 0.230392217,5] OPAL v7.0 starting... > > 08:28:32 DEBUG| [ 0.230674939,7] initial console log level: memory 7, driver 5 > > 08:28:32 DEBUG| [ 0.231048494,6] CPU: P9 generation processor (max 4 threads/core) > > 08:28:32 DEBUG| [ > 08:28:32 DEBUG| [ 0.231412547,7] CPU: Boot CPU PIR is 0x0000 PVR is 0x004e1202 > > 08:28:32 DEBUG| [ > 08:28:32 ERROR| > 08:28:32 ERROR| Reproduced traceback from: /builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/core/test.py:770 > 08:28:32 ERROR| Traceback (most recent call last): > 08:28:32 ERROR| File "/builds/thuth/qemu/build/tests/avocado/reverse_debugging.py", line 262, in test_ppc64_powernv > 08:28:32 ERROR| self.reverse_debugging() > 08:28:32 ERROR| File "/builds/thuth/qemu/build/tests/avocado/reverse_debugging.py", line 178, in reverse_debugging > 08:28:32 ERROR| g.cmd(b'c') > 08:28:32 ERROR| File "/builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/utils/gdb.py", line 783, in cmd > 08:28:32 ERROR| response_payload = self.decode(result) > 08:28:32 ERROR| File "/builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/utils/gdb.py", line 738, in decode > 08:28:32 ERROR| raise InvalidPacketError > 08:28:32 ERROR| avocado.utils.gdb.InvalidPacketError > 08:28:32 ERROR| > 08:28:32 DEBUG| Local variables: > 08:28:32 DEBUG| -> self <class 'reverse_debugging.ReverseDebugging_ppc64'>: 79-tests/avocado/reverse_debugging.py:ReverseDebugging_ppc64.test_ppc64_powernv > 08:28:32 DEBUG| Shutting down VM appliance; timeout=30 > 08:28:32 DEBUG| Attempting graceful termination > 08:28:32 DEBUG| Closing console socket > 08:28:32 DEBUG| Politely asking QEMU to terminate > > So unless someone has a clue how to fix that, I guess it's > likely best to revert this enablement patch again... A little further in the log we see 08:28:32 DEBUG| Politely asking QEMU to terminate 08:28:32 DEBUG| --> { "execute": "quit" } 08:28:32 DEBUG| <-- { "timestamp": { "seconds": 1700555312, "microseconds": 86122 }, "event": "RESUME" } 08:28:32 ERROR| Task.Reader: BrokenPipeError: [Errno 32] Broken pipe With seeing a bad packet from GDB and seeing Broken pipe from QMP, my impression is that the QEMU process is no longer present, most likely it has SEGV'd I reckon. IOW, I think we might well have a genuine bug here, not merely an unreliable test suite. None the less, unless someone can guess what the problem is, we'll need to disable the test to get reliable CI. A bug should be opened though with the CI logs. With regards, Daniel
On 21/11/2023 10.14, Daniel P. Berrangé wrote: > On Tue, Nov 21, 2023 at 09:56:24AM +0100, Thomas Huth wrote: >> On 17/11/2023 08.35, Nicholas Piggin wrote: >>> On Fri Nov 17, 2023 at 4:11 AM AEST, Thomas Huth wrote: >>>> On 16/11/2023 12.53, Nicholas Piggin wrote: >>>>> Let's try enable reverse_debugging.py in gitlab CI. >>>>> >>>>> Signed-off-by: Nicholas Piggin <npiggin@gmail.com> >>>>> --- >>>>> Maybe we could try this again at some point? The bug might have been >>>>> noticed sooner. >>>>> >>>>> They only take a couple of seconds to run so should not take too much >>>>> overhead. But my gitlab CI pipeline doesn't run the avocado tests for >>>>> some reason, so I can't see if it's still causing problems. ... >>>> FYI, I gave it a try, and it survived my CI run: >>>> >>>> https://gitlab.com/thuth/qemu/-/jobs/5552213972#L403 >>>> >>>> So I went ahead and put it (together with the first patch) in my current >>>> pull request, let's see how it goes... >>> >>> Great, thank you. >> >> ... and here it's failing again (current master branch): >> >> https://gitlab.com/thuth/qemu/-/jobs/5582657378#L404 >> >> According to the debug.log in the artifacts, it's failing here: ... > 08:28:32 ERROR| Task.Reader: BrokenPipeError: [Errno 32] Broken pipe > > With seeing a bad packet from GDB and seeing Broken pipe from QMP, > my impression is that the QEMU process is no longer present, most > likely it has SEGV'd I reckon. > > IOW, I think we might well have a genuine bug here, not merely an > unreliable test suite. > > None the less, unless someone can guess what the problem is, we'll > need to disable the test to get reliable CI. I'll sent a patch to revert the commit. > A bug should be opened though with the CI logs. Done: https://gitlab.com/qemu-project/qemu/-/issues/1992 Thomas
diff --git a/tests/avocado/reverse_debugging.py b/tests/avocado/reverse_debugging.py index 128d85bc0e..b1410e7a69 100644 --- a/tests/avocado/reverse_debugging.py +++ b/tests/avocado/reverse_debugging.py @@ -205,8 +205,6 @@ def get_pc(self, g): return self.get_reg_le(g, self.REG_PC) \ + self.get_reg_le(g, self.REG_CS) * 0x10 - # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_x86_64_pc(self): """ :avocado: tags=arch:x86_64 @@ -222,8 +220,6 @@ class ReverseDebugging_AArch64(ReverseDebugging): REG_PC = 32 - # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_aarch64_virt(self): """ :avocado: tags=arch:aarch64 @@ -246,8 +242,6 @@ class ReverseDebugging_ppc64(ReverseDebugging): REG_PC = 0x40 - # unidentified gitlab timeout problem - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_ppc64_pseries(self): """ :avocado: tags=arch:ppc64 @@ -259,7 +253,6 @@ def test_ppc64_pseries(self): self.endian_is_le = False self.reverse_debugging() - @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_ppc64_powernv(self): """ :avocado: tags=arch:ppc64
Let's try enable reverse_debugging.py in gitlab CI. Signed-off-by: Nicholas Piggin <npiggin@gmail.com> --- Maybe we could try this again at some point? The bug might have been noticed sooner. They only take a couple of seconds to run so should not take too much overhead. But my gitlab CI pipeline doesn't run the avocado tests for some reason, so I can't see if it's still causing problems. Thanks, Nick --- tests/avocado/reverse_debugging.py | 7 ------- 1 file changed, 7 deletions(-)