diff mbox series

[v5,4/5] support/scripts/boot-qemu-image.py: boot Qemu images with Qemu-system.

Message ID 20200217205030.8157-5-romain.naour@smile.fr
State Accepted
Headers show
Series gitlab Qemu runtime testing | expand

Commit Message

Romain Naour Feb. 17, 2020, 8:50 p.m. UTC 
From: Jugurtha BELKALEM <jugurtha.belkalem@smile.fr>

This script is intended to be used by gitlab CI to test
at runtime Qemu images generated by Buildroot's Qemu defconfigs.

This allows to troubleshoot different issues that may be
associated with defective builds by lanching a qemu machine,
sending root password, waiting for login shell and then perform
a shutdown.

This script is inspired by toolchain builder [1] and the Buildroot
testing infrastructure.

Usage: boot-qemu-image.py <qemu_arch_defconfig>

The gitlab CI will call this script for each defconfig build but
only Qemu defconfig will be runtime tested, all others defconfig are
ignored. The script retrieve the qemu command line from start-qemu.sh
script. The start-qemu.sh script is no executed directly since we
want to handle the serial output with pexpect to detect the
shell, enter the password and powerdown the system.
If the script is missing, error out with a message in the log.

The script check is the corresponding Qemu binary (qemu-system) is
available, either built by Buildroot or available from the host.
Some Qemu defconfig must be used with a specific Qemu version (fork)
that is not always available. The script doesn't error out when
qemu-system is missing but a message is added in the log.

Finally, the script start Qemu like it's done for the Buildroot
testing infrastructure (using pexpect).

Note:
We noticed some timeout issues with pexpect when the Qemu machine is
powered off. That's because Qemu process doesn't stop even if the
system is halted (after "System halted"). So the script doesn't error
out when such timeout occure. The behaviour depends on the architecture
emulated by Qemu.

[1] https://github.com/bootlin/toolchains-builder/blob/master/build.sh

Signed-off-by: Jugurtha BELKALEM <jugurtha.belkalem@smile.fr>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
---
v5: No Change
---
 support/scripts/boot-qemu-image.py | 105 +++++++++++++++++++++++++++++
 1 file changed, 105 insertions(+)
 create mode 100755 support/scripts/boot-qemu-image.py

Comments

Yann E. MORIN April 13, 2020, 7:15 a.m. UTC | #1 
Romain, Jugurtha, All,

On 2020-02-17 21:50 +0100, Romain Naour spake thusly:
> From: Jugurtha BELKALEM <jugurtha.belkalem@smile.fr>
> This script is intended to be used by gitlab CI to test
> at runtime Qemu images generated by Buildroot's Qemu defconfigs.
[--SNIP--]
> diff --git a/support/scripts/boot-qemu-image.py b/support/scripts/boot-qemu-image.py
> new file mode 100755
> index 0000000000..abaf88d446
> --- /dev/null
> +++ b/support/scripts/boot-qemu-image.py
> @@ -0,0 +1,105 @@
> +#!/usr/bin/env python3
> +
> +# This script expect to run from the Buildroot top directory.
> +
> +import pexpect
> +import sys
> +import os
> +import re
> +import shlex
> +import shutil
> +import subprocess
> +
> +argc = len(sys.argv)
> +if not (argc == 2):
> +    print("Error: incorrect number of arguments")
> +    print("""Usage: boot-qemu-image.py <qemu_arch_defconfig>""")
> +    sys.exit(1)
> +
> +defconfig_filename = sys.argv[1]
> +
> +# Ignore non Qemu defconfig
> +if defconfig_filename.startswith('qemu_') is False:
> +    sys.exit(0)
> +
> +qemu_start_script_filepath = os.path.join(os.getcwd(), 'output/images/start-qemu.sh')
> +
> +# Ignore if start-qemu.sh is missing, we can't test.
> +if not os.path.exists(qemu_start_script_filepath):
> +    print("Error: " + qemu_start_script_filepath) + " file is missing."
> +    sys.exit(1)
> +
> +qemu_cmd = ""
> +
> +with open(qemu_start_script_filepath, 'r') as script_file:
> +    for line in script_file:
> +        if re.search("qemu-system", line):
> +            qemu_cmd = line
> +            break
> +
> +if not qemu_cmd:
> +    print("Error: No QEMU command line found in " + qemu_start_script_filepath)
> +    sys.exit(1)

Why do you extract the command line, and can't directly run the script?

> +# Replace bashism
> +qemu_cmd = line.replace("${IMAGE_DIR}", "output/images")

This would avoid this dance...

> +# pexpect needs a list, convert a sting to a list and escape quoted substring.
> +qemu_cmd = shlex.split(qemu_cmd)

... as well as this one.

> +# Use host Qemu if provided by Buildroot.
> +os.environ["PATH"] = os.getcwd() + "/output/host/bin" + os.pathsep + os.environ["PATH"]

And this one can be avoided too if the script is called with PATH
properly set to include ${HOST_DIR}/bin. This should be easy, as this
script is only expected to run from the gitlab-ci pipelines.

> +# Ignore when no Qemu emulation is available
> +if not shutil.which(qemu_cmd[0]):
> +    print("No " + qemu_cmd[0] + " binary available, THIS DEFCONFIG CAN NOT BE TESTED!")
> +    sys.exit(0)
> +
> +
> +def main():
> +    global child

Mixing code at the top-level and in a main() is ugly. Mocve everything
to a main (and helper functions if youlike), but don't mix the two.

> +    try:
> +        child.expect(["buildroot login:", pexpect.TIMEOUT], timeout=60)
> +    except pexpect.EOF:
> +        print("Connection problem, exiting.")
> +        sys.exit(1)
> +    except pexpect.TIMEOUT:
> +        print("System did not boot in time, exiting.")
> +        sys.exit(1)
> +
> +    child.sendline("root\r")
> +
> +    try:
> +        child.expect(["# ", pexpect.TIMEOUT], timeout=60)
> +    except pexpect.EOF:
> +        print("Cannot connect to shell")
> +        sys.exit(1)
> +    except pexpect.TIMEOUT:
> +        print("Timeout while waiting for shell")
> +        sys.exit(1)
> +
> +    child.sendline("poweroff\r")
> +
> +    try:
> +        child.expect(["System halted", pexpect.TIMEOUT], timeout=60)
> +        child.expect(pexpect.EOF)
> +    except pexpect.EOF:
> +        sys.exit(0)
> +    except pexpect.TIMEOUT:
> +        print("Cannot halt machine")
> +        # Qemu may not exit properly after "System halted", ignore.
> +        sys.exit(0)
> +
> +
> +# Log the Qemu version
> +subprocess.call([qemu_cmd[0], "--version"])
> +
> +# Log the Qemu command line
> +print(qemu_cmd)

You would not have to do that if you just called the start-qemu.sh script.

Regards,
Yann E. MORIN.

> +child = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], timeout=5, encoding='utf-8',
> +                      env={"QEMU_AUDIO_DRV": "none", 'PATH': os.environ["PATH"]})
> +# We want only stdout into the log to avoid double echo
> +child.logfile = sys.stdout
> +main()
> -- 
> 2.24.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
Romain Naour April 13, 2020, 9:38 p.m. UTC | #2 
Hi Yann, All,

Le 13/04/2020 à 09:15, Yann E. MORIN a écrit :
> Romain, Jugurtha, All,
> 
> On 2020-02-17 21:50 +0100, Romain Naour spake thusly:
>> From: Jugurtha BELKALEM <jugurtha.belkalem@smile.fr>
>> This script is intended to be used by gitlab CI to test
>> at runtime Qemu images generated by Buildroot's Qemu defconfigs.
> [--SNIP--]
>> diff --git a/support/scripts/boot-qemu-image.py b/support/scripts/boot-qemu-image.py
>> new file mode 100755
>> index 0000000000..abaf88d446
>> --- /dev/null
>> +++ b/support/scripts/boot-qemu-image.py
>> @@ -0,0 +1,105 @@
>> +#!/usr/bin/env python3
>> +
>> +# This script expect to run from the Buildroot top directory.
>> +
>> +import pexpect
>> +import sys
>> +import os
>> +import re
>> +import shlex
>> +import shutil
>> +import subprocess
>> +
>> +argc = len(sys.argv)
>> +if not (argc == 2):
>> +    print("Error: incorrect number of arguments")
>> +    print("""Usage: boot-qemu-image.py <qemu_arch_defconfig>""")
>> +    sys.exit(1)
>> +
>> +defconfig_filename = sys.argv[1]
>> +
>> +# Ignore non Qemu defconfig
>> +if defconfig_filename.startswith('qemu_') is False:
>> +    sys.exit(0)
>> +
>> +qemu_start_script_filepath = os.path.join(os.getcwd(), 'output/images/start-qemu.sh')
>> +
>> +# Ignore if start-qemu.sh is missing, we can't test.
>> +if not os.path.exists(qemu_start_script_filepath):
>> +    print("Error: " + qemu_start_script_filepath) + " file is missing."
>> +    sys.exit(1)
>> +
>> +qemu_cmd = ""
>> +
>> +with open(qemu_start_script_filepath, 'r') as script_file:
>> +    for line in script_file:
>> +        if re.search("qemu-system", line):
>> +            qemu_cmd = line
>> +            break
>> +
>> +if not qemu_cmd:
>> +    print("Error: No QEMU command line found in " + qemu_start_script_filepath)
>> +    sys.exit(1)
> 
> Why do you extract the command line, and can't directly run the script?
> 
>> +# Replace bashism
>> +qemu_cmd = line.replace("${IMAGE_DIR}", "output/images")
> 
> This would avoid this dance...
> 
>> +# pexpect needs a list, convert a sting to a list and escape quoted substring.
>> +qemu_cmd = shlex.split(qemu_cmd)
> 
> ... as well as this one.
> 
>> +# Use host Qemu if provided by Buildroot.
>> +os.environ["PATH"] = os.getcwd() + "/output/host/bin" + os.pathsep + os.environ["PATH"]
> 
> And this one can be avoided too if the script is called with PATH
> properly set to include ${HOST_DIR}/bin. This should be easy, as this
> script is only expected to run from the gitlab-ci pipelines.
> 
>> +# Ignore when no Qemu emulation is available
>> +if not shutil.which(qemu_cmd[0]):
>> +    print("No " + qemu_cmd[0] + " binary available, THIS DEFCONFIG CAN NOT BE TESTED!")
>> +    sys.exit(0)

Extracting the qemu command line in the boot-qemu-image.py allowed to test if at
least one qemu-systemd-<arch> binary is present. Starting start-qemu.sh directly
crash the python script.

https://gitlab.com/kubu93/buildroot/-/jobs/509053135/artifacts/file/runtime-test.log

csky and or1k defconfig doesn't select any host-qemu and there is none installed
in the Buildroot Docker image used by gitlab CI.

It seems we need to catch the exception while calling pexpect.spawn(qemu_start...)

https://git.buildroot.net/buildroot/tree/support/scripts/boot-qemu-image.py#n22

Or just add this test in start-qemu.sh

Best regards,
Romain
diff mbox series

Patch

diff --git a/support/scripts/boot-qemu-image.py b/support/scripts/boot-qemu-image.py
new file mode 100755
index 0000000000..abaf88d446
--- /dev/null
+++ b/support/scripts/boot-qemu-image.py
@@ -0,0 +1,105 @@ 
+#!/usr/bin/env python3
+
+# This script expect to run from the Buildroot top directory.
+
+import pexpect
+import sys
+import os
+import re
+import shlex
+import shutil
+import subprocess
+
+argc = len(sys.argv)
+if not (argc == 2):
+    print("Error: incorrect number of arguments")
+    print("""Usage: boot-qemu-image.py <qemu_arch_defconfig>""")
+    sys.exit(1)
+
+defconfig_filename = sys.argv[1]
+
+# Ignore non Qemu defconfig
+if defconfig_filename.startswith('qemu_') is False:
+    sys.exit(0)
+
+qemu_start_script_filepath = os.path.join(os.getcwd(), 'output/images/start-qemu.sh')
+
+# Ignore if start-qemu.sh is missing, we can't test.
+if not os.path.exists(qemu_start_script_filepath):
+    print("Error: " + qemu_start_script_filepath) + " file is missing."
+    sys.exit(1)
+
+qemu_cmd = ""
+
+with open(qemu_start_script_filepath, 'r') as script_file:
+    for line in script_file:
+        if re.search("qemu-system", line):
+            qemu_cmd = line
+            break
+
+if not qemu_cmd:
+    print("Error: No QEMU command line found in " + qemu_start_script_filepath)
+    sys.exit(1)
+
+# Replace bashism
+qemu_cmd = line.replace("${IMAGE_DIR}", "output/images")
+
+# pexpect needs a list, convert a sting to a list and escape quoted substring.
+qemu_cmd = shlex.split(qemu_cmd)
+
+# Use host Qemu if provided by Buildroot.
+os.environ["PATH"] = os.getcwd() + "/output/host/bin" + os.pathsep + os.environ["PATH"]
+
+# Ignore when no Qemu emulation is available
+if not shutil.which(qemu_cmd[0]):
+    print("No " + qemu_cmd[0] + " binary available, THIS DEFCONFIG CAN NOT BE TESTED!")
+    sys.exit(0)
+
+
+def main():
+    global child
+
+    try:
+        child.expect(["buildroot login:", pexpect.TIMEOUT], timeout=60)
+    except pexpect.EOF:
+        print("Connection problem, exiting.")
+        sys.exit(1)
+    except pexpect.TIMEOUT:
+        print("System did not boot in time, exiting.")
+        sys.exit(1)
+
+    child.sendline("root\r")
+
+    try:
+        child.expect(["# ", pexpect.TIMEOUT], timeout=60)
+    except pexpect.EOF:
+        print("Cannot connect to shell")
+        sys.exit(1)
+    except pexpect.TIMEOUT:
+        print("Timeout while waiting for shell")
+        sys.exit(1)
+
+    child.sendline("poweroff\r")
+
+    try:
+        child.expect(["System halted", pexpect.TIMEOUT], timeout=60)
+        child.expect(pexpect.EOF)
+    except pexpect.EOF:
+        sys.exit(0)
+    except pexpect.TIMEOUT:
+        print("Cannot halt machine")
+        # Qemu may not exit properly after "System halted", ignore.
+        sys.exit(0)
+
+
+# Log the Qemu version
+subprocess.call([qemu_cmd[0], "--version"])
+
+# Log the Qemu command line
+print(qemu_cmd)
+
+child = pexpect.spawn(qemu_cmd[0], qemu_cmd[1:], timeout=5, encoding='utf-8',
+                      env={"QEMU_AUDIO_DRV": "none", 'PATH': os.environ["PATH"]})
+# We want only stdout into the log to avoid double echo
+child.logfile = sys.stdout
+main()