mbox series

[RFC,v2,0/3] Add external test support

Message ID 20240103002810.1055951-1-colin.foster@in-advantage.com
Headers show
Series Add external test support | expand

Message

Colin Foster Jan. 3, 2024, 12:28 a.m. UTC 
NOTE: This is an RFC that isn't fully functional! Do not merge!

Add the ability to create and maintain tests in a BR2_EXTERNAL
directory.

I recently submitted a simple tftpy test module. My current system is
based on Buildroot 2023.02.x, and I would like to be able to run
verifications without modification to Buildroot and without waiting
until 2024.02.x lands.

There were some Python dependencies that led me to this current
iteration, which is to create a temporary directory for all tests and
pointing nose2 to the temp directory. I'm not sure if that is elegant,
or kludgy... so I'm looking for feedback on that.

I also tested this with my "tftpy" script, but it currently fails.


```
(.venv) $ BR2_EXTERNAL=`pwd`/buildroot-external ./buildroot/support/testing/run-tests -d dl -o output_folder -k tests.package.test_python_tftpy.TestPythonTftpy
17:07:56 TestPythonTftpy                          Starting
17:07:56 TestPythonTftpy                          Building
E
======================================================================
ERROR: test_run (tests.package.test_python_tftpy.TestPythonTftpy)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/path/to/pwd/buildroot/support/testing/infra/basetest.py", line 73, in setUp
    self.b.build()
  File "/path/to/pwd/buildroot/support/testing/infra/builder.py", line 98, in build
    raise SystemError("Build failed")
SystemError: Build failed

----------------------------------------------------------------------
Ran 1 test in 4.925s

FAILED (errors=1)
```

I figure I should get some feedback on my approach before digging into
this too much further. I did run a simple verification that the
functional tests aren't broken:


```
(.venv) $ BR2_EXTERNAL=`pwd`/buildroot-external ./buildroot/support/testing/run-tests -d dl -o output
_folder -k tests.package.test_python_aexpect.TestPythonPy3Aexpect
17:12:55 TestPythonPy3Aexpect                     Starting
17:13:01 TestPythonPy3Aexpect                     Cleaning up
.
----------------------------------------------------------------------
Ran 1 test in 6.521s

OK
```


Thoughts on this approach?
Is there a different way to overlay the external directory that I'm
overlooking?

If this approach is valid... then why might the test be failing? I'm
surprised to see "/path/to/pwd/buildroot/support/testing/infra/*"
referenced in the output... I'd expect it to be
"/tmp/tmp*/testing/infra/*".


Thanks for any feedback!


*** Notes on V2 ***

There are two other techniques that might make this "cleaner". The first
was suggested by Yann in review, where every test was discovered and
soft-linked.

A second option might be to duplicate a directory into a directory like
buildroot/support/testing/tests/external. This directory could be in
.gitignore and removed upon a clean.


v1 -> v2:
  * Still non-functional
  * Split into three patches for readability
  * Support for multiple BR2_EXTERNALs
  * Remove conditional copy to temp directory - always do it.

Colin Foster (3):
  support/testing/run-tests: add try block for indentation
  support/testing/run-tests: run tests from temp directory
  support/testing/run-tests: add ability to run tests from external

 support/testing/run-tests | 164 +++++++++++++++++++++-----------------
 1 file changed, 91 insertions(+), 73 deletions(-)