diff mbox series

[2/2] support/scripts/boot-qemu-image.py: increase the timeout by 10

Message ID 20220115143344.12319-2-romain.naour@gmail.com
State Accepted
Headers show
Series [1/2] board/qemu/arm-vexpress-tz: use enable=on | expand

Commit Message

Romain Naour Jan. 15, 2022, 2:33 p.m. UTC
As for the Buildroot testsuite, multiply every emulator timeout by 10
to avoid sporadic failures in elastic runners.

qemu_arm_vexpress_tz_defconfig tested locally with sucess.

Fixes:
https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046

Signed-off-by: Romain Naour <romain.naour@gmail.com>
---
 support/scripts/boot-qemu-image.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

Comments

Yann E. MORIN Jan. 15, 2022, 4:05 p.m. UTC | #1
Romain, All,

On 2022-01-15 15:33 +0100, Romain Naour spake thusly:
> As for the Buildroot testsuite, multiply every emulator timeout by 10
> to avoid sporadic failures in elastic runners.
> 
> qemu_arm_vexpress_tz_defconfig tested locally with sucess.
> 
> Fixes:
> https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046

Are you sure increasing the timeout will fix this failure? The
build-run.log contains another error:

    loaded file zImage from 40400000 to 408CAABF, 004CAAC0 bytes
    loaded file rootfs.cpio.uboot from 44000000 to 4438FA4E, 0038FA4F bytes
    Kernel image @ 0x40400000 [ 0x000000 - 0x4caac0 ]
    ## Loading init Ramdisk from Legacy Image at 44000000 ...
       Image Name:   
       Created:      2022-01-13  16:06:53 UTC
       Image Type:   ARM Linux RAMDisk Image (uncompressed)
       Data Size:    3734031 Bytes = 3.6 MiB
       Load Address: 00000000
       Entry Point:  00000000
       Verifying Checksum ... OK
    ## Flattened Device Tree blob at 40000000
       Booting using the fdt blob at 0x40000000
    ERROR: reserving fdt memory region failed (addr=7fe00000 size=200000)
       Using Device Tree in place at 40000000, end 400051f7

So, to me, this is not a timeout issue, but an actual boot issue.

Regards,
Yann E. MORIN.

> Signed-off-by: Romain Naour <romain.naour@gmail.com>
> ---
>  support/scripts/boot-qemu-image.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/support/scripts/boot-qemu-image.py b/support/scripts/boot-qemu-image.py
> index fba1533bb7..ba50ea69ee 100755
> --- a/support/scripts/boot-qemu-image.py
> +++ b/support/scripts/boot-qemu-image.py
> @@ -25,7 +25,7 @@ def main():
>      qemu_start = os.path.join(os.getcwd(), 'output/images/start-qemu.sh')
>  
>      child = pexpect.spawn(qemu_start, ['serial-only'],
> -                          timeout=5, encoding='utf-8',
> +                          timeout=50, encoding='utf-8',
>                            env={"QEMU_AUDIO_DRV": "none"})
>  
>      # We want only stdout into the log to avoid double echo
> @@ -36,7 +36,7 @@ def main():
>      time.sleep(1)
>  
>      try:
> -        child.expect(["buildroot login:"], timeout=60)
> +        child.expect(["buildroot login:"], timeout=600)
>      except pexpect.EOF as e:
>          # Some emulations require a fork of qemu-system, which may be
>          # missing on the system, and is not provided by Buildroot.
> @@ -58,7 +58,7 @@ def main():
>      child.sendline("root\r")
>  
>      try:
> -        child.expect(["# "], timeout=60)
> +        child.expect(["# "], timeout=600)
>      except pexpect.EOF:
>          print("Cannot connect to shell")
>          sys.exit(1)
> @@ -69,7 +69,7 @@ def main():
>      child.sendline("poweroff\r")
>  
>      try:
> -        child.expect(["System halted"], timeout=60)
> +        child.expect(["System halted"], timeout=600)
>          child.expect(pexpect.EOF)
>      except pexpect.EOF:
>          pass
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
Romain Naour Jan. 15, 2022, 7:28 p.m. UTC | #2
Hello Yann,

Le 15/01/2022 à 17:05, Yann E. MORIN a écrit :
> Romain, All,
> 
> On 2022-01-15 15:33 +0100, Romain Naour spake thusly:
>> As for the Buildroot testsuite, multiply every emulator timeout by 10
>> to avoid sporadic failures in elastic runners.
>>
>> qemu_arm_vexpress_tz_defconfig tested locally with sucess.
>>
>> Fixes:
>> https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046
> 
> Are you sure increasing the timeout will fix this failure? The
> build-run.log contains another error:
> 
>     loaded file zImage from 40400000 to 408CAABF, 004CAAC0 bytes
>     loaded file rootfs.cpio.uboot from 44000000 to 4438FA4E, 0038FA4F bytes
>     Kernel image @ 0x40400000 [ 0x000000 - 0x4caac0 ]
>     ## Loading init Ramdisk from Legacy Image at 44000000 ...
>        Image Name:   
>        Created:      2022-01-13  16:06:53 UTC
>        Image Type:   ARM Linux RAMDisk Image (uncompressed)
>        Data Size:    3734031 Bytes = 3.6 MiB
>        Load Address: 00000000
>        Entry Point:  00000000
>        Verifying Checksum ... OK
>     ## Flattened Device Tree blob at 40000000
>        Booting using the fdt blob at 0x40000000
>     ERROR: reserving fdt memory region failed (addr=7fe00000 size=200000)
>        Using Device Tree in place at 40000000, end 400051f7
> 
> So, to me, this is not a timeout issue, but an actual boot issue.

The system stop in the middle of optee startup

optee: dynamic shared memory is enabled

Also, I got the same error message while testing locally.

Best regards,
Romain


> 
> Regards,
> Yann E. MORIN.
> 
>> Signed-off-by: Romain Naour <romain.naour@gmail.com>
>> ---
>>  support/scripts/boot-qemu-image.py | 8 ++++----
>>  1 file changed, 4 insertions(+), 4 deletions(-)
>>
>> diff --git a/support/scripts/boot-qemu-image.py b/support/scripts/boot-qemu-image.py
>> index fba1533bb7..ba50ea69ee 100755
>> --- a/support/scripts/boot-qemu-image.py
>> +++ b/support/scripts/boot-qemu-image.py
>> @@ -25,7 +25,7 @@ def main():
>>      qemu_start = os.path.join(os.getcwd(), 'output/images/start-qemu.sh')
>>  
>>      child = pexpect.spawn(qemu_start, ['serial-only'],
>> -                          timeout=5, encoding='utf-8',
>> +                          timeout=50, encoding='utf-8',
>>                            env={"QEMU_AUDIO_DRV": "none"})
>>  
>>      # We want only stdout into the log to avoid double echo
>> @@ -36,7 +36,7 @@ def main():
>>      time.sleep(1)
>>  
>>      try:
>> -        child.expect(["buildroot login:"], timeout=60)
>> +        child.expect(["buildroot login:"], timeout=600)
>>      except pexpect.EOF as e:
>>          # Some emulations require a fork of qemu-system, which may be
>>          # missing on the system, and is not provided by Buildroot.
>> @@ -58,7 +58,7 @@ def main():
>>      child.sendline("root\r")
>>  
>>      try:
>> -        child.expect(["# "], timeout=60)
>> +        child.expect(["# "], timeout=600)
>>      except pexpect.EOF:
>>          print("Cannot connect to shell")
>>          sys.exit(1)
>> @@ -69,7 +69,7 @@ def main():
>>      child.sendline("poweroff\r")
>>  
>>      try:
>> -        child.expect(["System halted"], timeout=60)
>> +        child.expect(["System halted"], timeout=600)
>>          child.expect(pexpect.EOF)
>>      except pexpect.EOF:
>>          pass
>> -- 
>> 2.31.1
>>
>> _______________________________________________
>> buildroot mailing list
>> buildroot@buildroot.org
>> https://lists.buildroot.org/mailman/listinfo/buildroot
>
Yann E. MORIN Jan. 15, 2022, 8:23 p.m. UTC | #3
Romain, All,

On 2022-01-15 20:28 +0100, Romain Naour spake thusly:
> Le 15/01/2022 à 17:05, Yann E. MORIN a écrit :
> > On 2022-01-15 15:33 +0100, Romain Naour spake thusly:
> >> As for the Buildroot testsuite, multiply every emulator timeout by 10
> >> to avoid sporadic failures in elastic runners.
> >>
> >> qemu_arm_vexpress_tz_defconfig tested locally with sucess.
> >>
> >> Fixes:
> >> https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046
> > Are you sure increasing the timeout will fix this failure? The
> > build-run.log contains another error:
[--SNIP--]
> >     ERROR: reserving fdt memory region failed (addr=7fe00000 size=200000)
> >        Using Device Tree in place at 40000000, end 400051f7
> > So, to me, this is not a timeout issue, but an actual boot issue.
> 
> The system stop in the middle of optee startup
> optee: dynamic shared memory is enabled

I was very confused by the above. Indeed, the log file I downladed
earler really stopped at the error message I pasted. I am sure of that,
as I still had it in my ~/tmp directory, so I could really double-check.

But now I downloaded it again, and indeed, there are much more traces,
up to that optee issue you point ed at.

This is a really weird situation, that the download I had was only
partial...

> Also, I got the same error message while testing locally.

But testing locally should not have the same impact on boot time, would
it? The boot times out on the eleastic runners because they are too slow
(and many congtainers contend for the CPU there). But locally, it is
really surprising that you hit the exact same timeout issue, and the
exact same point in the boot sequence...

Anyway, can you resend your patch, as I marked it rejected in patchwork,
please?

Regards,
Yann E. MORIN.
Romain Naour Jan. 16, 2022, 12:37 p.m. UTC | #4
Hello Yann, All,

Le 15/01/2022 à 21:23, Yann E. MORIN a écrit :
> Romain, All,
> 
> On 2022-01-15 20:28 +0100, Romain Naour spake thusly:
>> Le 15/01/2022 à 17:05, Yann E. MORIN a écrit :
>>> On 2022-01-15 15:33 +0100, Romain Naour spake thusly:
>>>> As for the Buildroot testsuite, multiply every emulator timeout by 10
>>>> to avoid sporadic failures in elastic runners.
>>>>
>>>> qemu_arm_vexpress_tz_defconfig tested locally with sucess.
>>>>
>>>> Fixes:
>>>> https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046
>>> Are you sure increasing the timeout will fix this failure? The
>>> build-run.log contains another error:
> [--SNIP--]
>>>     ERROR: reserving fdt memory region failed (addr=7fe00000 size=200000)
>>>        Using Device Tree in place at 40000000, end 400051f7
>>> So, to me, this is not a timeout issue, but an actual boot issue.
>>
>> The system stop in the middle of optee startup
>> optee: dynamic shared memory is enabled
> 
> I was very confused by the above. Indeed, the log file I downladed
> earler really stopped at the error message I pasted. I am sure of that,
> as I still had it in my ~/tmp directory, so I could really double-check.
> 
> But now I downloaded it again, and indeed, there are much more traces,
> up to that optee issue you point ed at.
> 
> This is a really weird situation, that the download I had was only
> partial...

weird ineed.

> 
>> Also, I got the same error message while testing locally.
> 
> But testing locally should not have the same impact on boot time, would
> it? The boot times out on the eleastic runners because they are too slow
> (and many congtainers contend for the CPU there). But locally, it is
> really surprising that you hit the exact same timeout issue, and the
> exact same point in the boot sequence...

I'm not able to reproduce the timeout locally but by looking at the timeout
multiplier used for the Buildroot testsuite I would say that a sporadic failures.

I restarted the job in gitlab and this time it succeeded:

https://gitlab.com/buildroot.org/buildroot/-/jobs/1979108650

Best regards,
Romain

> 
> Anyway, can you resend your patch, as I marked it rejected in patchwork,
> please?
> 
> Regards,
> Yann E. MORIN.
>
Thomas Petazzoni July 27, 2022, 1:57 p.m. UTC | #5
On Sat, 15 Jan 2022 15:33:44 +0100
Romain Naour <romain.naour@gmail.com> wrote:

> As for the Buildroot testsuite, multiply every emulator timeout by 10
> to avoid sporadic failures in elastic runners.
> 
> qemu_arm_vexpress_tz_defconfig tested locally with sucess.
> 
> Fixes:
> https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046
> 
> Signed-off-by: Romain Naour <romain.naour@gmail.com>
> ---
>  support/scripts/boot-qemu-image.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Applied to master, thanks.

Thomas
Peter Korsgaard Aug. 30, 2022, 4:04 p.m. UTC | #6
>>>>> "Thomas" == Thomas Petazzoni via buildroot <buildroot@buildroot.org> writes:

 > On Sat, 15 Jan 2022 15:33:44 +0100
 > Romain Naour <romain.naour@gmail.com> wrote:

 >> As for the Buildroot testsuite, multiply every emulator timeout by 10
 >> to avoid sporadic failures in elastic runners.
 >> 
 >> qemu_arm_vexpress_tz_defconfig tested locally with sucess.
 >> 
 >> Fixes:
 >> https://gitlab.com/buildroot.org/buildroot/-/jobs/1970084046
 >> 
 >> Signed-off-by: Romain Naour <romain.naour@gmail.com>
 >> ---
 >> support/scripts/boot-qemu-image.py | 8 ++++----
 >> 1 file changed, 4 insertions(+), 4 deletions(-)

 > Applied to master, thanks.

Committed to 2022.05.x and 2022.02.x, thanks.
diff mbox series

Patch

diff --git a/support/scripts/boot-qemu-image.py b/support/scripts/boot-qemu-image.py
index fba1533bb7..ba50ea69ee 100755
--- a/support/scripts/boot-qemu-image.py
+++ b/support/scripts/boot-qemu-image.py
@@ -25,7 +25,7 @@  def main():
     qemu_start = os.path.join(os.getcwd(), 'output/images/start-qemu.sh')
 
     child = pexpect.spawn(qemu_start, ['serial-only'],
-                          timeout=5, encoding='utf-8',
+                          timeout=50, encoding='utf-8',
                           env={"QEMU_AUDIO_DRV": "none"})
 
     # We want only stdout into the log to avoid double echo
@@ -36,7 +36,7 @@  def main():
     time.sleep(1)
 
     try:
-        child.expect(["buildroot login:"], timeout=60)
+        child.expect(["buildroot login:"], timeout=600)
     except pexpect.EOF as e:
         # Some emulations require a fork of qemu-system, which may be
         # missing on the system, and is not provided by Buildroot.
@@ -58,7 +58,7 @@  def main():
     child.sendline("root\r")
 
     try:
-        child.expect(["# "], timeout=60)
+        child.expect(["# "], timeout=600)
     except pexpect.EOF:
         print("Cannot connect to shell")
         sys.exit(1)
@@ -69,7 +69,7 @@  def main():
     child.sendline("poweroff\r")
 
     try:
-        child.expect(["System halted"], timeout=60)
+        child.expect(["System halted"], timeout=600)
         child.expect(pexpect.EOF)
     except pexpect.EOF:
         pass