| Message ID | 1490042214-6762-2-git-send-email-thomas.petazzoni@free-electrons.com |
|---|---|
| State | Accepted |
| Headers | show |
On Mon, Mar 20, 2017 at 9:36 PM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > This commit adds the core of a new testing infrastructure that allows to > perform runtime testing of Buildroot generated systems. This > infrastructure uses the Python unittest logic as its foundation. > > This core infrastructure commit includes the following aspects: > > - A base test class, called BRTest, defined in > support/testing/infra/basetest.py. This base test class inherited > from the Python provided unittest.TestCase, and must be subclassed by > all Buildroot test cases. > > Its main purpose is to provide the Python unittest setUp() and > tearDown() methods. In our case, setUp() takes care of building the > Buildroot system described in the test case, and instantiate the > Emulator object in case runtime testing is needed. The tearDown() > method simply cleans things up (stop the emulator, remove the output > directory). > > - A Builder class, defined in support/testing/infra/builder.py, simply > responsible for building the Buildroot system in each test case. > > - An Emulator class, defined in support/testing/infra/emulator.py, > responsible for running the generated system under Qemu, allowing > each test case to run arbitrary commands inside the emulated system. > > - A run-tests script, which is the entry point to start the tests. > > Even though I wrote the original version of this small infrastructure, a > huge amount of rework and improvement has been done by Maxime > Hadjinlian, and squashed into this patch. So many thanks to Maxime for > cleaning up and improving my Python code! > > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> > --- > support/testing/conf/unittest.cfg | 6 ++ > support/testing/infra/__init__.py | 89 +++++++++++++++++++++++++ > support/testing/infra/basetest.py | 66 +++++++++++++++++++ > support/testing/infra/builder.py | 49 ++++++++++++++ > support/testing/infra/emulator.py | 135 ++++++++++++++++++++++++++++++++++++++ > support/testing/run-tests | 83 +++++++++++++++++++++++ > support/testing/tests/__init__.py | 0 > 7 files changed, 428 insertions(+) > create mode 100644 support/testing/conf/unittest.cfg > create mode 100644 support/testing/infra/__init__.py > create mode 100644 support/testing/infra/basetest.py > create mode 100644 support/testing/infra/builder.py > create mode 100644 support/testing/infra/emulator.py > create mode 100755 support/testing/run-tests > create mode 100644 support/testing/tests/__init__.py > [..] > diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py > new file mode 100644 > index 0000000..7c476cb > --- /dev/null > +++ b/support/testing/infra/emulator.py > @@ -0,0 +1,135 @@ > +import socket > +import subprocess > +import telnetlib > + > +import infra > +import infra.basetest > + > +# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss > +# with the qemu machine. > +class Emulator(object): > + > + def __init__(self, builddir, downloaddir, logtofile): > + self.qemu = None > + self.__tn = None > + self.downloaddir = downloaddir > + self.log = "" > + self.log_file = "{}-run.log".format(builddir) > + if logtofile is None: > + self.log_file = None > + > + # Start Qemu to boot the system > + # > + # arch: Qemu architecture to use > + # > + # kernel: path to the kernel image, or the special string > + # 'builtin'. 'builtin' means a pre-built kernel image will be > + # downloaded from ARTEFACTS_URL and suitable options are There is a stale reference to ARTEFACTS_URL here (ARTIFACTS_URL) [..] > + # Wait for the login prompt to appear, and then login as root with > + # the provided password, or no password if not specified. > + def login(self, password=None): > + self.__read_until("buildroot login:", 10) > + if "buildroot login:" not in self.log: If we want people to use this testing infrastructure on their own defconfigs too, then encoding the hostname in this string is a bit restrictive. What about searching for 'login:' only ? Do you think it is too generic? [..] /Thomas
Thomas, There are still 2 things that I think can be solved together. On Mon, Mar 20, 2017 at 05:36 PM, Thomas Petazzoni wrote: [snip] > +++ b/support/testing/conf/unittest.cfg > @@ -0,0 +1,6 @@ > +[unittest] > +plugins = nose2.plugins.mp > + > +[multiprocess] > +processes = 1 1) If I increase the value of processes in unittest.cfg to unleash the power of nose2, emulator fails sporadically because telnet port is fixed. [snip] > +++ b/support/testing/infra/emulator.py [snip] > + qemu_cmd = ["qemu-system-{}".format(qemu_arch), > + "-serial", "telnet::1234,server", > + "-display", "none"] 2) If I do this (not restricted to these testcases) ... for i in $(seq 100); do \ support/testing/run-tests -d dl1 -o output -k tests.package || break ; done ... while performing other resource-consuming tasks on my PC, such as starting an unrelated build that uses all the cores, I eventually get this: ERROR: test_run (tests.package.test_dropbear.TestDropbear) ---------------------------------------------------------------------- Traceback (most recent call last): File "/tmp/testing-v3/support/testing/tests/package/test_dropbear.py", line 22, in test_run self.emulator.login("testpwd") File "/tmp/testing-v3/support/testing/infra/emulator.py", line 101, in login self.__read_until("buildroot login:", 10) File "/tmp/testing-v3/support/testing/infra/emulator.py", line 89, in __read_until data = self.__tn.read_until(waitstr, timeout) File "/usr/lib/python2.7/telnetlib.py", line 294, in read_until return self._read_until_with_poll(match, timeout) File "/usr/lib/python2.7/telnetlib.py", line 329, in _read_until_with_poll self.fill_rawq() File "/usr/lib/python2.7/telnetlib.py", line 576, in fill_rawq buf = self.sock.recv(50) error: [Errno 104] Connection reset by peer Something like this patch could fix those: +++ b/support/testing/infra/emulator.py @@ -1,3 +1,3 @@ +import pexpect import socket -import subprocess import telnetlib @@ -7,2 +7,5 @@ import infra.basetest +TELNET_PORT_INITIAL = 1234 +TELNET_PORT_LAST = TELNET_PORT_INITIAL + 20 + # TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss @@ -13,2 +16,3 @@ class Emulator(object): self.qemu = None + self.port = None self.__tn = None @@ -43,3 +47,2 @@ class Emulator(object): qemu_cmd = ["qemu-system-{}".format(qemu_arch), - "-serial", "telnet::1234,server", "-display", "none"] @@ -75,4 +78,15 @@ class Emulator(object): with infra.smart_open(self.log_file) as lfh: - lfh.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd)) - self.qemu = subprocess.Popen(qemu_cmd, stdout=lfh, stderr=lfh) + for port in range(TELNET_PORT_INITIAL, TELNET_PORT_LAST): + telnet_arg = ["-serial", "telnet::{},server".format(port)] + cmd = qemu_cmd + telnet_arg + lfh.write("> starting qemu with '{}'\n".format(" ".join(cmd))) + self.qemu = pexpect.spawn(cmd[0], cmd[1:], logfile=lfh) + ret = self.qemu.expect(["waiting for connection", + "Address already in use"]) + if ret == 0: + self.port = port + break + self.qemu.read() + if not self.port: + raise SystemError("Could not find a free port to run emulator") @@ -81,3 +95,3 @@ class Emulator(object): try: - self.__tn = telnetlib.Telnet("localhost", 1234) + self.__tn = telnetlib.Telnet("localhost", self.port) if self.__tn: @@ -100,6 +114,8 @@ class Emulator(object): def login(self, password=None): - self.__read_until("buildroot login:", 10) + # The login prompt can take some time to appear when running multiple + # instances in parallel, so set the timeout to a large value + self.__read_until("buildroot login:", 30) if "buildroot login:" not in self.log: with infra.smart_open(self.log_file) as lfh: - lfh.write("==> System does not boot") + lfh.write("==> System does not boot\n") raise SystemError("System does not boot") @@ -133,3 +149,2 @@ class Emulator(object): return - self.qemu.terminate() - self.qemu.kill() + self.qemu.terminate(force=True) Regards, Ricardo
Hello, On Thu, 23 Mar 2017 00:02:30 -0300, Ricardo Martincoski wrote: > 1) If I increase the value of processes in unittest.cfg to unleash the power of > nose2, emulator fails sporadically because telnet port is fixed. Yes, this is a known problem, and there's a TODO about this in emulator.py: # TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss # with the qemu machine. > 2) If I do this (not restricted to these testcases) ... > > for i in $(seq 100); do \ > support/testing/run-tests -d dl1 -o output -k tests.package || break ; done > > ... while performing other resource-consuming tasks on my PC, such as starting > an unrelated build that uses all the cores, I eventually get this: > > ERROR: test_run (tests.package.test_dropbear.TestDropbear) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/tmp/testing-v3/support/testing/tests/package/test_dropbear.py", line 22, in test_run > self.emulator.login("testpwd") > File "/tmp/testing-v3/support/testing/infra/emulator.py", line 101, in login > self.__read_until("buildroot login:", 10) > File "/tmp/testing-v3/support/testing/infra/emulator.py", line 89, in __read_until > data = self.__tn.read_until(waitstr, timeout) > File "/usr/lib/python2.7/telnetlib.py", line 294, in read_until > return self._read_until_with_poll(match, timeout) > File "/usr/lib/python2.7/telnetlib.py", line 329, in _read_until_with_poll > self.fill_rawq() > File "/usr/lib/python2.7/telnetlib.py", line 576, in fill_rawq > buf = self.sock.recv(50) > error: [Errno 104] Connection reset by peer Hum, do you know why this one happens? I definitely understand that will a qemu is running, another qemu cannot bind its telnet server on the same TCP port, but I don't see why it would cause a "Connection reset by peer". Or is the logic of the test bogus, and if it doesn't manage to start qemu, the test still continues, and it in fact connects to an already running qemu instance from a different test, which then gets killed in the middle ? > Something like this patch could fix those: It would probably be nicer to interact through stdio and pexpect, but in the mean time, this does the job. I'll integrate that into my series. Thanks! Thomas
Hello, On Thu, Mar 23, 2017 at 05:07 AM, Thomas Petazzoni wrote: >> error: [Errno 104] Connection reset by peer > > Hum, do you know why this one happens? I definitely understand that > will a qemu is running, another qemu cannot bind its telnet server on > the same TCP port, but I don't see why it would cause a "Connection > reset by peer". It is not clear to me why this one happens. > Or is the logic of the test bogus, and if it doesn't manage to start > qemu, the test still continues, and it in fact connects to an already > running qemu instance from a different test, which then gets killed in > the middle ? I redirected the output from Popen to stdout and stderr and when the error occurs only the messages from the first qemu appear before the Traceback: [output/TestPythonBase/2017-03-23 20:52:29] Starting QEMU waiting for connection on: disconnected:telnet::1234,server pulseaudio: set_sink_input_volume() failed pulseaudio: Reason: Invalid argument pulseaudio: set_sink_input_mute() failed pulseaudio: Reason: Invalid argument [output/TestPythonBase/2017-03-23 20:52:37] Cleaning up .[output/TestDropbear/2017-03-23 20:52:37] Starting [output/TestDropbear/2017-03-23 20:52:37] Cleaning up E So I suppose it could be a bogus logic. Anyway I think waiting for "waiting for connection" before connecting should be more reliable to know the qemu started than just trying to connect using telnet. Regards, Ricardo
Hi, only a minor nit that I noticed while reviewing patch 5. On 20/03/2017 21:36, Thomas Petazzoni wrote: [...] > diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py > new file mode 100644 > index 0000000..c3f645c > --- /dev/null > +++ b/support/testing/infra/__init__.py [...] > +def get_elf_prog_interpreter(builddir, prefix, fpath): > + cmd = ["host/usr/bin/{}-readelf".format(prefix), > + "-l", os.path.join("target", fpath)] > + out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"}) > + regexp = re.compile("^ *\[Requesting program interpreter: (.*)\]$") > + for line in out.splitlines(): > + m = regexp.match(line) > + if not m: > + continue > + return m.group(1) > + return None This should be documented, just like get_elf_arch_tag() above. Proposal (unchecked): """ Runs the cross readelf on 'fpath' to extract the program interpreter name and returns it. Example: >>> get_elf_prog_interpreter('output', 'arm-none-linux-gnueabi-', 'bin/busybox') /lib/ld-uClibc.so.0 >>> """
Hi, this the review I had written a few days ago. Since the patch has just been applied, I've reworded it as a list of proposed improvements and request for clarifications. On 20/03/2017 21:36, Thomas Petazzoni wrote: > This commit adds the core of a new testing infrastructure that allows to > perform runtime testing of Buildroot generated systems. This > infrastructure uses the Python unittest logic as its foundation. Good we have this on master now! Things that I already commented about and for which I plan to send patches: - remove smart_open - document code - better reporting [...] > diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py > new file mode 100644 > index 0000000..c3f645c > --- /dev/null > +++ b/support/testing/infra/__init__.py > @@ -0,0 +1,89 @@ > +import contextlib > +import os > +import re > +import sys > +import tempfile > +import subprocess > +from urllib2 import urlopen, HTTPError, URLError > + > +ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/" Since you renamed artefacts to artifacts, it's probably good to rename in the URL too (and update the server). Sure it's a minor nit, but since it's an URL that should stay available for long, better having it fixed as soon as possible. [...] > diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py > new file mode 100644 > index 0000000..7c476cb > --- /dev/null > +++ b/support/testing/infra/emulator.py > @@ -0,0 +1,135 @@ > +import socket > +import subprocess > +import telnetlib > + > +import infra > +import infra.basetest > + > +# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss > +# with the qemu machine. > +class Emulator(object): > + > + def __init__(self, builddir, downloaddir, logtofile): > + self.qemu = None > + self.__tn = None > + self.downloaddir = downloaddir > + self.log = "" > + self.log_file = "{}-run.log".format(builddir) > + if logtofile is None: > + self.log_file = None > + > + # Start Qemu to boot the system > + # > + # arch: Qemu architecture to use > + # > + # kernel: path to the kernel image, or the special string > + # 'builtin'. 'builtin' means a pre-built kernel image will be > + # downloaded from ARTEFACTS_URL and suitable options are > + # automatically passed to qemu and added to the kernel cmdline. So > + # far only armv5, armv7 and i386 builtin kernels are available. > + # If None, then no kernel is used, and we assume a bootable device > + # will be specified. > + # > + # kernel_cmdline: array of kernel arguments to pass to Qemu -append option > + # > + # options: array of command line options to pass to Qemu > + # > + def boot(self, arch, kernel=None, kernel_cmdline=None, options=None): > + if arch in ["armv7", "armv5"]: > + qemu_arch = "arm" > + else: > + qemu_arch = arch > + > + qemu_cmd = ["qemu-system-{}".format(qemu_arch), > + "-serial", "telnet::1234,server", > + "-display", "none"] > + > + if options: > + qemu_cmd += options > + > + if kernel_cmdline is None: > + kernel_cmdline = [] > + > + if kernel: > + if kernel == "builtin": > + if arch in ["armv7", "armv5"]: > + kernel_cmdline.append("console=ttyAMA0") > + > + if arch == "armv7": > + kernel = infra.download(self.downloaddir, > + "kernel-vexpress") > + dtb = infra.download(self.downloaddir, > + "vexpress-v2p-ca9.dtb") > + qemu_cmd += ["-dtb", dtb] > + qemu_cmd += ["-M", "vexpress-a9"] > + elif arch == "armv5": > + kernel = infra.download(self.downloaddir, > + "kernel-versatile") > + qemu_cmd += ["-M", "versatilepb"] > + > + qemu_cmd += ["-kernel", kernel] I'm OK with the "builtin" logic, but I really dislike it being hard-coded in Emulator.boot(). It's acceptable for the moment since we have only very few builtin kernels. Should we add more in the future, I think we should at least have a sort of "database" of builtin kernels, perhaps in the form of an associative array. > diff --git a/support/testing/run-tests b/support/testing/run-tests > new file mode 100755 > index 0000000..339bb66 > --- /dev/null > +++ b/support/testing/run-tests > @@ -0,0 +1,83 @@ > +#!/usr/bin/env python2 > +import argparse > +import sys > +import os > +import nose2 > + > +from infra.basetest import BRTest > + > +def main(): > + parser = argparse.ArgumentParser(description='Run Buildroot tests') > + parser.add_argument('testname', nargs='*', > + help='list of test cases to execute') > + parser.add_argument('--list', '-l', action='store_true', > + help='list of available test cases') > + parser.add_argument('--all', '-a', action='store_true', > + help='execute all test cases') > + parser.add_argument('--stdout', '-s', action='store_true', > + help='log everything to stdout') > + parser.add_argument('--output', '-o', > + help='output directory') > + parser.add_argument('--download', '-d', > + help='download directory') > + parser.add_argument('--keep', '-k', > + help='keep build directories', > + action='store_true') Stylish note: I would put the one-letter form before the long form. This is not only my personal taste, but also what the manpages usually do, and what Python does with the automatically-added -h/--help parameter: $ ./support/testing/run-tests [...] optional arguments: -h, --help show this help message and exit --list, -l list of available test cases I'll send a patch.
Hello, On Sun, 7 May 2017 23:07:31 +0200, Luca Ceresoli wrote: > Things that I already commented about and for which I plan to send patches: > - remove smart_open > - document code > - better reporting Great! > > +ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/" > > Since you renamed artefacts to artifacts, it's probably good to rename > in the URL too (and update the server). Sure it's a minor nit, but since > it's an URL that should stay available for long, better having it fixed > as soon as possible. Correct, I'll fix that up. > > + qemu_cmd += ["-kernel", kernel] > > I'm OK with the "builtin" logic, but I really dislike it being > hard-coded in Emulator.boot(). It's acceptable for the moment since we > have only very few builtin kernels. Should we add more in the future, I > think we should at least have a sort of "database" of builtin kernels, > perhaps in the form of an associative array. Fully agreed. What we have now is OK for a few kernel/qemu setups, but clearly does not scale, and will have to be improved. > Stylish note: I would put the one-letter form before the long form. This > is not only my personal taste, but also what the manpages usually do, > and what Python does with the automatically-added -h/--help parameter: Indeed. Patch welcome! :-) Thanks! Thomas
diff --git a/support/testing/conf/unittest.cfg b/support/testing/conf/unittest.cfg new file mode 100644 index 0000000..6eaa234 --- /dev/null +++ b/support/testing/conf/unittest.cfg @@ -0,0 +1,6 @@ +[unittest] +plugins = nose2.plugins.mp + +[multiprocess] +processes = 1 +always-on = True diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py new file mode 100644 index 0000000..c3f645c --- /dev/null +++ b/support/testing/infra/__init__.py @@ -0,0 +1,89 @@ +import contextlib +import os +import re +import sys +import tempfile +import subprocess +from urllib2 import urlopen, HTTPError, URLError + +ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/" + +@contextlib.contextmanager +def smart_open(filename=None): + """ + Return a file-like object that can be written to using the 'with' + keyword, as in the example: + with infra.smart_open("test.log") as outfile: + outfile.write("Hello, world!\n") + """ + if filename and filename != '-': + fhandle = open(filename, 'a+') + else: + fhandle = sys.stdout + + try: + yield fhandle + finally: + if fhandle is not sys.stdout: + fhandle.close() + +def filepath(relpath): + return os.path.join(os.getcwd(), "support/testing", relpath) + +def download(dldir, filename): + finalpath = os.path.join(dldir, filename) + if os.path.exists(finalpath): + return finalpath + + if not os.path.exists(dldir): + os.makedirs(dldir) + + tmpfile = tempfile.mktemp(dir=dldir) + print "Downloading to {}".format(tmpfile) + + try: + url_fh = urlopen(os.path.join(ARTIFACTS_URL, filename)) + with open(tmpfile, "w+") as tmpfile_fh: + tmpfile_fh.write(url_fh.read()) + except (HTTPError, URLError), err: + os.unlink(tmpfile) + raise err + + print "Renaming from %s to %s" % (tmpfile, finalpath) + os.rename(tmpfile, finalpath) + return finalpath + +def get_elf_arch_tag(builddir, prefix, fpath, tag): + """ + Runs the cross readelf on 'fpath', then extracts the value of tag 'tag'. + Example: + >>> get_elf_arch_tag('output', 'arm-none-linux-gnueabi-', + 'bin/busybox', 'Tag_CPU_arch') + v5TEJ + >>> + """ + cmd = ["host/usr/bin/{}-readelf".format(prefix), + "-A", os.path.join("target", fpath)] + out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"}) + regexp = re.compile("^ {}: (.*)$".format(tag)) + for line in out.splitlines(): + m = regexp.match(line) + if not m: + continue + return m.group(1) + return None + +def get_file_arch(builddir, prefix, fpath): + return get_elf_arch_tag(builddir, prefix, fpath, "Tag_CPU_arch") + +def get_elf_prog_interpreter(builddir, prefix, fpath): + cmd = ["host/usr/bin/{}-readelf".format(prefix), + "-l", os.path.join("target", fpath)] + out = subprocess.check_output(cmd, cwd=builddir, env={"LANG": "C"}) + regexp = re.compile("^ *\[Requesting program interpreter: (.*)\]$") + for line in out.splitlines(): + m = regexp.match(line) + if not m: + continue + return m.group(1) + return None diff --git a/support/testing/infra/basetest.py b/support/testing/infra/basetest.py new file mode 100644 index 0000000..eb9da90 --- /dev/null +++ b/support/testing/infra/basetest.py @@ -0,0 +1,66 @@ +import unittest +import os +import datetime + +from infra.builder import Builder +from infra.emulator import Emulator + +BASIC_TOOLCHAIN_CONFIG = \ +""" +BR2_arm=y +BR2_TOOLCHAIN_EXTERNAL=y +BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y +BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y +BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-full-2015.05-1190-g4a48479.tar.bz2" +BR2_TOOLCHAIN_EXTERNAL_GCC_4_7=y +BR2_TOOLCHAIN_EXTERNAL_HEADERS_3_10=y +BR2_TOOLCHAIN_EXTERNAL_LOCALE=y +# BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set +BR2_TOOLCHAIN_EXTERNAL_INET_RPC=y +BR2_TOOLCHAIN_EXTERNAL_CXX=y +""" + +MINIMAL_CONFIG = \ +""" +BR2_INIT_NONE=y +BR2_SYSTEM_BIN_SH_NONE=y +# BR2_PACKAGE_BUSYBOX is not set +# BR2_TARGET_ROOTFS_TAR is not set +""" + +class BRTest(unittest.TestCase): + config = None + downloaddir = None + outputdir = None + logtofile = True + keepbuilds = False + + def show_msg(self, msg): + print "[%s/%s/%s] %s" % (os.path.basename(self.__class__.outputdir), + self.testname, + datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), + msg) + def setUp(self): + self.testname = self.__class__.__name__ + self.builddir = os.path.join(self.__class__.outputdir, self.testname) + self.runlog = self.builddir + "-run.log" + self.emulator = None + self.show_msg("Starting") + self.b = Builder(self.__class__.config, self.builddir, self.logtofile) + + if not self.keepbuilds: + self.b.delete() + + if not self.b.is_finished(): + self.show_msg("Building") + self.b.build() + self.show_msg("Building done") + + self.emulator = Emulator(self.builddir, self.downloaddir, self.logtofile) + + def tearDown(self): + self.show_msg("Cleaning up") + if self.emulator: + self.emulator.stop() + if self.b and not self.keepbuilds: + self.b.delete() diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py new file mode 100644 index 0000000..105da01 --- /dev/null +++ b/support/testing/infra/builder.py @@ -0,0 +1,49 @@ +import os +import shutil +import subprocess + +import infra + +class Builder(object): + def __init__(self, config, builddir, logtofile): + self.config = config + self.builddir = builddir + self.logtofile = logtofile + + def build(self): + if not os.path.isdir(self.builddir): + os.makedirs(self.builddir) + + log = "{}-build.log".format(self.builddir) + if not self.logtofile: + log = None + + config_file = os.path.join(self.builddir, ".config") + with open(config_file, "w+") as cf: + cf.write(self.config) + + cmd = ["make", + "O={}".format(self.builddir), + "olddefconfig"] + with infra.smart_open(log) as log_fh: + ret = subprocess.call(cmd, stdout=log_fh, stderr=log_fh) + if ret != 0: + raise SystemError("Cannot olddefconfig") + + cmd = ["make", "-C", self.builddir] + with infra.smart_open(log) as log_fh: + ret = subprocess.call(cmd, stdout=log_fh, stderr=log_fh) + if ret != 0: + raise SystemError("Build failed") + + open(self.stamp_path(), 'a').close() + + def stamp_path(self): + return os.path.join(self.builddir, "build-done") + + def is_finished(self): + return os.path.exists(self.stamp_path()) + + def delete(self): + if os.path.exists(self.builddir): + shutil.rmtree(self.builddir) diff --git a/support/testing/infra/emulator.py b/support/testing/infra/emulator.py new file mode 100644 index 0000000..7c476cb --- /dev/null +++ b/support/testing/infra/emulator.py @@ -0,0 +1,135 @@ +import socket +import subprocess +import telnetlib + +import infra +import infra.basetest + +# TODO: Most of the telnet stuff need to be replaced by stdio/pexpect to discuss +# with the qemu machine. +class Emulator(object): + + def __init__(self, builddir, downloaddir, logtofile): + self.qemu = None + self.__tn = None + self.downloaddir = downloaddir + self.log = "" + self.log_file = "{}-run.log".format(builddir) + if logtofile is None: + self.log_file = None + + # Start Qemu to boot the system + # + # arch: Qemu architecture to use + # + # kernel: path to the kernel image, or the special string + # 'builtin'. 'builtin' means a pre-built kernel image will be + # downloaded from ARTEFACTS_URL and suitable options are + # automatically passed to qemu and added to the kernel cmdline. So + # far only armv5, armv7 and i386 builtin kernels are available. + # If None, then no kernel is used, and we assume a bootable device + # will be specified. + # + # kernel_cmdline: array of kernel arguments to pass to Qemu -append option + # + # options: array of command line options to pass to Qemu + # + def boot(self, arch, kernel=None, kernel_cmdline=None, options=None): + if arch in ["armv7", "armv5"]: + qemu_arch = "arm" + else: + qemu_arch = arch + + qemu_cmd = ["qemu-system-{}".format(qemu_arch), + "-serial", "telnet::1234,server", + "-display", "none"] + + if options: + qemu_cmd += options + + if kernel_cmdline is None: + kernel_cmdline = [] + + if kernel: + if kernel == "builtin": + if arch in ["armv7", "armv5"]: + kernel_cmdline.append("console=ttyAMA0") + + if arch == "armv7": + kernel = infra.download(self.downloaddir, + "kernel-vexpress") + dtb = infra.download(self.downloaddir, + "vexpress-v2p-ca9.dtb") + qemu_cmd += ["-dtb", dtb] + qemu_cmd += ["-M", "vexpress-a9"] + elif arch == "armv5": + kernel = infra.download(self.downloaddir, + "kernel-versatile") + qemu_cmd += ["-M", "versatilepb"] + + qemu_cmd += ["-kernel", kernel] + + if kernel_cmdline: + qemu_cmd += ["-append", " ".join(kernel_cmdline)] + + with infra.smart_open(self.log_file) as lfh: + lfh.write("> starting qemu with '%s'\n" % " ".join(qemu_cmd)) + self.qemu = subprocess.Popen(qemu_cmd, stdout=lfh, stderr=lfh) + + # Wait for the telnet port to appear and connect to it. + while True: + try: + self.__tn = telnetlib.Telnet("localhost", 1234) + if self.__tn: + break + except socket.error: + continue + + def __read_until(self, waitstr, timeout=5): + data = self.__tn.read_until(waitstr, timeout) + self.log += data + with infra.smart_open(self.log_file) as lfh: + lfh.write(data) + return data + + def __write(self, wstr): + self.__tn.write(wstr) + + # Wait for the login prompt to appear, and then login as root with + # the provided password, or no password if not specified. + def login(self, password=None): + self.__read_until("buildroot login:", 10) + if "buildroot login:" not in self.log: + with infra.smart_open(self.log_file) as lfh: + lfh.write("==> System does not boot") + raise SystemError("System does not boot") + + self.__write("root\n") + if password: + self.__read_until("Password:") + self.__write(password + "\n") + self.__read_until("# ") + if "# " not in self.log: + raise SystemError("Cannot login") + self.run("dmesg -n 1") + + # Run the given 'cmd' on the target + # return a tuple (output, exit_code) + def run(self, cmd): + self.__write(cmd + "\n") + output = self.__read_until("# ") + output = output.strip().splitlines() + output = output[1:len(output)-1] + + self.__write("echo $?\n") + exit_code = self.__read_until("# ") + exit_code = exit_code.strip().splitlines()[1] + exit_code = int(exit_code) + + return output, exit_code + + def stop(self): + if self.qemu is None: + return + self.qemu.terminate() + self.qemu.kill() diff --git a/support/testing/run-tests b/support/testing/run-tests new file mode 100755 index 0000000..339bb66 --- /dev/null +++ b/support/testing/run-tests @@ -0,0 +1,83 @@ +#!/usr/bin/env python2 +import argparse +import sys +import os +import nose2 + +from infra.basetest import BRTest + +def main(): + parser = argparse.ArgumentParser(description='Run Buildroot tests') + parser.add_argument('testname', nargs='*', + help='list of test cases to execute') + parser.add_argument('--list', '-l', action='store_true', + help='list of available test cases') + parser.add_argument('--all', '-a', action='store_true', + help='execute all test cases') + parser.add_argument('--stdout', '-s', action='store_true', + help='log everything to stdout') + parser.add_argument('--output', '-o', + help='output directory') + parser.add_argument('--download', '-d', + help='download directory') + parser.add_argument('--keep', '-k', + help='keep build directories', + action='store_true') + + args = parser.parse_args() + + script_path = os.path.realpath(__file__) + test_dir = os.path.dirname(script_path) + + if args.stdout: + BRTest.logtofile = False + + if args.list: + print "List of tests" + nose2.discover(argv=[script_path, + "-s", test_dir, + "-v", + "--collect-only"], + plugins=["nose2.plugins.collect"]) + return 0 + + if args.download is None: + args.download = os.getenv("BR2_DL_DIR") + if args.download is None: + print "Missing download directory, please use -d/--download" + print "" + parser.print_help() + return 1 + + BRTest.downloaddir = os.path.abspath(args.download) + + if args.output is None: + print "Missing output directory, please use -o/--output" + print "" + parser.print_help() + return 1 + + if not os.path.exists(args.output): + os.mkdir(args.output) + + BRTest.outputdir = os.path.abspath(args.output) + + if args.all is False and len(args.testname) == 0: + print "No test selected" + print "" + parser.print_help() + return 1 + + BRTest.keepbuilds = args.keep + + nose2_args = ["-v", + "-s", "support/testing", + "-c", "support/testing/conf/unittest.cfg"] + + if len(args.testname) != 0: + nose2_args += args.testname + + nose2.discover(argv=nose2_args) + +if __name__ == "__main__": + sys.exit(main())
This commit adds the core of a new testing infrastructure that allows to perform runtime testing of Buildroot generated systems. This infrastructure uses the Python unittest logic as its foundation. This core infrastructure commit includes the following aspects: - A base test class, called BRTest, defined in support/testing/infra/basetest.py. This base test class inherited from the Python provided unittest.TestCase, and must be subclassed by all Buildroot test cases. Its main purpose is to provide the Python unittest setUp() and tearDown() methods. In our case, setUp() takes care of building the Buildroot system described in the test case, and instantiate the Emulator object in case runtime testing is needed. The tearDown() method simply cleans things up (stop the emulator, remove the output directory). - A Builder class, defined in support/testing/infra/builder.py, simply responsible for building the Buildroot system in each test case. - An Emulator class, defined in support/testing/infra/emulator.py, responsible for running the generated system under Qemu, allowing each test case to run arbitrary commands inside the emulated system. - A run-tests script, which is the entry point to start the tests. Even though I wrote the original version of this small infrastructure, a huge amount of rework and improvement has been done by Maxime Hadjinlian, and squashed into this patch. So many thanks to Maxime for cleaning up and improving my Python code! Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com> --- support/testing/conf/unittest.cfg | 6 ++ support/testing/infra/__init__.py | 89 +++++++++++++++++++++++++ support/testing/infra/basetest.py | 66 +++++++++++++++++++ support/testing/infra/builder.py | 49 ++++++++++++++ support/testing/infra/emulator.py | 135 ++++++++++++++++++++++++++++++++++++++ support/testing/run-tests | 83 +++++++++++++++++++++++ support/testing/tests/__init__.py | 0 7 files changed, 428 insertions(+) create mode 100644 support/testing/conf/unittest.cfg create mode 100644 support/testing/infra/__init__.py create mode 100644 support/testing/infra/basetest.py create mode 100644 support/testing/infra/builder.py create mode 100644 support/testing/infra/emulator.py create mode 100755 support/testing/run-tests create mode 100644 support/testing/tests/__init__.py diff --git a/support/testing/tests/__init__.py b/support/testing/tests/__init__.py new file mode 100644 index 0000000..e69de29