[v2,5/5] support/testing: add toolchain tests

This commit adds an initial toolchain test case, testing the ARM
CodeSourcery toolchain, just checking that the proper sysroot is used,
and that a minimal Linux system boots fine under Qemu.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 support/testing/tests/toolchain/__init__.py      |   0
 support/testing/tests/toolchain/test_external.py | 156 +++++++++++++++++++++++
 2 files changed, 156 insertions(+)
 create mode 100644 support/testing/tests/toolchain/__init__.py
 create mode 100644 support/testing/tests/toolchain/test_external.py

Thomas Petazzoni,

On Sun, Mar 05, 2017 at 12:03 PM, Thomas Petazzoni wrote:

[snip]
> +class TestExternalToolchain(infra.basetest.BRTest):

nose2 recognizes this as a test:
$ support/testing/run-tests --list 2>&1 | grep Toolchain | head -n1
test_run (tests.toolchain.test_external.TestExternalToolchain) ... ok

and it tries to run it when --all is used:
======================================================================
ERROR: test_run (tests.toolchain.test_external.TestExternalToolchain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/testing-v2/support/testing/infra/basetest.py", line 56, in setUp
    self.b.build()
  File "/tmp/testing-v2/support/testing/infra/builder.py", line 23, in build
    cf.write(self.config)
TypeError: expected a string or other character buffer object

> +    def test_run(self):

I think you could avoid this by renaming this method ...

https://nose2.readthedocs.io/en/latest/usage.html#naming-tests
'Within test modules, nose2 will load tests from unittest.TestCase subclasses,
and from test functions (functions whose names begin with "test").'

... to something like this:
    def common_check(self):
...

> +        # Check for broken symlinks
> +        for d in ["lib", "usr/lib"]:
> +            path = os.path.join(self.builddir, "staging", d)
> +            self.assertFalse(check_broken_links(path))
> +            path = os.path.join(self.builddir, "target", d)
> +            self.assertFalse(check_broken_links(path))
> +
> +        interp = infra.get_elf_prog_interpreter(self.builddir,
> +                                                self.toolchain_prefix,
> +                                                "bin/busybox")
> +        interp_path = os.path.join(self.builddir, "target", interp[1:])
> +        self.assertTrue(os.path.exists(interp_path))
> +
> +class TestExternalToolchainSourceryArmv4(TestExternalToolchain):
> +    config = BASIC_CONFIG + \
> +"""
> +BR2_arm=y
> +BR2_arm920t=y
> +BR2_TOOLCHAIN_EXTERNAL=y
> +BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM=y
> +"""
> +    toolchain_prefix = "arm-none-linux-gnueabi"
> +
> +    def test_run(self):
> +        TestExternalToolchain.test_run(self)

... and changing all calls to something like this:
        self.common_check()

> +
> +        # Check the architecture variant
[snip]

Regards,
Ricardo

Hello,

On Sun, 05 Mar 2017 18:30:01 -0300, Ricardo Martincoski wrote:

> and it tries to run it when --all is used:
> ======================================================================
> ERROR: test_run (tests.toolchain.test_external.TestExternalToolchain)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File "/tmp/testing-v2/support/testing/infra/basetest.py", line 56, in setUp
>     self.b.build()
>   File "/tmp/testing-v2/support/testing/infra/builder.py", line 23, in build
>     cf.write(self.config)
> TypeError: expected a string or other character buffer object
> 
> > +    def test_run(self):  
> 
> I think you could avoid this by renaming this method ...
> 
> https://nose2.readthedocs.io/en/latest/usage.html#naming-tests
> 'Within test modules, nose2 will load tests from unittest.TestCase subclasses,
> and from test functions (functions whose names begin with "test").'
> 
> ... to something like this:
>     def common_check(self):
> ...

Ah, yes, indeed.

Speaking of this, there is one thing I am not entirely happy with: it
would be much nicer if we could split some test cases in multiple
test_<foo>() methods, especially the external toolchain tests.

However, the setUp() and tearDown() methods are called before and after
running *each* test_<foo>() method of the current unit test class. This
is clearly not what we want, as we don't want to rebuild/clean up the
whole Buildroot build for each test_<foo>() method.

I saw we have setUpClass() and tearDownClass(), but these being class
methods, I guess you don't have access to members of the object
instance that will be used. And we access a lot of these through
"self." in setUp() and tearDown().

Do you have an idea?

In addition to this, we will at some point need to allow a test case to
do something special before the build is started, i.e in the middle of
the setUp() logic. Should the specific test case override setUp() in
this case, do its own stuff, and call the parent class setUp() method?

Thanks,

Thomas

Thomas,

On Sun, Mar 05, 2017 at 06:36 PM, Thomas Petazzoni wrote:

> Speaking of this, there is one thing I am not entirely happy with: it
> would be much nicer if we could split some test cases in multiple
> test_<foo>() methods, especially the external toolchain tests.
> 
> However, the setUp() and tearDown() methods are called before and after
> running *each* test_<foo>() method of the current unit test class. This
> is clearly not what we want, as we don't want to rebuild/clean up the
> whole Buildroot build for each test_<foo>() method.
> 
> I saw we have setUpClass() and tearDownClass(), but these being class

Beware those methods need python 2.7 at least.
But I don't think it is a huge problem.
Maybe it will be better if 'run-tests' bails out for older python versions if
we start using those methods. When called by an old python interpreter the
tests run but those methods are silently ignored!

I didn't tested this series with older python.

> methods, I guess you don't have access to members of the object
> instance that will be used. And we access a lot of these through
> "self." in setUp() and tearDown().

But it seems to me all accessed values are indirectly coming from the class.
Maybe I am missing something here.

> 
> Do you have an idea?

Please see this *hack*. Notice I don't rename self to cls because I want to
generate a small diff. I also manually edited the diff to make - and +
consecutive.

+++ support/testing/infra/basetest.py
@@ -37,4 +37,5 @@ class BRTest(unittest.TestCase):
 
+    @classmethod
     def show_msg(self, msg):
-        print "[%s/%s/%s] %s" % (os.path.basename(self.__class__.outputdir),
+        print "[%s/%s/%s] %s" % (os.path.basename(self.outputdir),
                                  self.testname,
@@ -42,5 +43,6 @@ class BRTest(unittest.TestCase):
                                  msg)
+    @classmethod
-    def setUp(self):
+    def setUpClass(self):
-        self.testname = self.__class__.__name__
+        self.testname = self.__name__
-        self.builddir = os.path.join(self.__class__.outputdir, self.testname)
+        self.builddir = os.path.join(self.outputdir, self.testname)
         self.runlog = self.builddir + "-run.log"
@@ -48,3 +50,3 @@ class BRTest(unittest.TestCase):
         self.show_msg("Starting")
-        self.b = Builder(self.__class__.config, self.builddir, self.logtofile)
+        self.b = Builder(self.config, self.builddir, self.logtofile)
 
@@ -60,3 +62,4 @@ class BRTest(unittest.TestCase):
 
+    @classmethod
-    def tearDown(self):
+    def tearDownClass(self):
         self.show_msg("Cleaning up")

Something like this can be done in a follow-up patch.

> 
> In addition to this, we will at some point need to allow a test case to
> do something special before the build is started, i.e in the middle of
> the setUp() logic. Should the specific test case override setUp() in
> this case, do its own stuff, and call the parent class setUp() method?

I think in this case the setUp could do only the part before the special
operation. Then the testcase can do the special operation and resume the build
and then do the runtime test.

I plan to do something like this to test the git downloader. I will split the
build() method from Builder into configure() and build().
Probably I will create a new type of test (perhaps DownloadTest or GitTest)
that overrides setUp() to call only configure() and each testcase that inherits
from this type can call self.b.build() at test_run().

We have now only BRTest, but for special cases we can have other classes that
override setUp.

Regards,
Ricardo