mbox series

[RFC,v1,0/1] Add external test support

Message ID 20231222232251.12786-1-colin.foster@in-advantage.com
Headers show
Series Add external test support | expand

Message

Colin Foster Dec. 22, 2023, 11:22 p.m. UTC 
NOTE: This is an RFC that isn't fully functional! Do not merge!

Add the ability to create and maintain tests in a BR2_EXTERNAL
directory.

I recently submitted a simple tftpy test module. My current system is
based on Buildroot 2023.02.x, and I would like to be able to run
verifications without modification to Buildroot and without waiting
until 2024.02.x lands.

There were some Python dependencies that led me to this current
iteration, which is to create a temporary directory for all tests and
pointing nose2 to the temp directory. I'm not sure if that is elegant,
or kludgy... so I'm looking for feedback on that.

I also tested this with my "tftpy" script, but it currently fails.


```
(.venv) $ BR2_EXTERNAL=`pwd`/buildroot-external ./buildroot/support/testing/run-tests -d dl -o output_folder -k tests.package.test_python_tftpy.TestPythonTftpy
17:07:56 TestPythonTftpy                          Starting
17:07:56 TestPythonTftpy                          Building
E
======================================================================
ERROR: test_run (tests.package.test_python_tftpy.TestPythonTftpy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/pwd/buildroot/support/testing/infra/basetest.py", line 73, in setUp
    self.b.build()
  File "/path/to/pwd/buildroot/support/testing/infra/builder.py", line 98, in build
    raise SystemError("Build failed")
SystemError: Build failed

----------------------------------------------------------------------
Ran 1 test in 4.925s

FAILED (errors=1)
```

I figure I should get some feedback on my approach before digging into
this too much further. I did run a simple verification that the
functional tests aren't broken:


```
(.venv) colin@colin-ia-desktop:ia_buildroot_overlay$ BR2_EXTERNAL=`pwd`/buildroot-external ./buildroot/support/testing/run-tests -d dl -o output
_folder -k tests.package.test_python_aexpect.TestPythonPy3Aexpect
17:12:55 TestPythonPy3Aexpect                     Starting
17:13:01 TestPythonPy3Aexpect                     Cleaning up
.
----------------------------------------------------------------------
Ran 1 test in 6.521s

OK
```


Thoughts on this approach?
Is there a different way to overlay the external directory that I'm
overlooking?

If this approach is valid... then why might the test be failing? I'm
surprised to see "/path/to/pwd/buildroot/support/testing/infra/*"
referenced in the output... I'd expect it to be
"/tmp/tmp*/testing/infra/*".


Thanks for any feedback!


Colin Foster (1):
  support/testing/run-tests: add ability to run tests from external

 support/testing/run-tests | 163 +++++++++++++++++++++-----------------
 1 file changed, 91 insertions(+), 72 deletions(-)