[autotest-client-tests] UBUNTU: SAUCE: ubuntu_qrt_kernel_security: improve test granularity

BugLink: https://bugs.launchpad.net/bugs/1976344

For now we have only 2 cases in this ubuntu_qrt_kernel_security test:
  * setup
  * test-kernel-security-test.py

All the 125 sub-tests are executed in test-kernel-security-test.py,
it's tidious to read the report every cycle for some specific failure
and hinting everything there is a bit risky for this security related
test.

We should probably ask security team to fix these broken tests or fix
these issues in our kernel, but there is always an exception.

Improve this test suite by running sub test case one-by-one with:
    python2 test-kernel-security.py -v TestClass.TestMethod

To retrieve tests from test-kernel-security.py, Use grep here is more
straightforward then trying to load a module with '-' in the filename.
We will have to grep for the class and the method (test case), then
we parse and constuct a dictionary with them. Downside is this will be
highly dependent to the script structure, should be ok as this won't
change too much under python unittest framework.

Time cost is the tradeoff here, it will increase from 1m30s to about
5m45s, but I think it defintely worth it.

Patch tested with Focal OEM-5.14.

Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>
---
 ubuntu_qrt_kernel_security/control            | 19 ++++++++++++++++++-
 .../ubuntu_qrt_kernel_security.py             |  8 ++++----
 2 files changed, 22 insertions(+), 5 deletions(-)

On Wed, Jun 01, 2022 at 05:17:25PM +0800, Po-Hsu Lin wrote:
> BugLink: https://bugs.launchpad.net/bugs/1976344
> 
> For now we have only 2 cases in this ubuntu_qrt_kernel_security test:
>   * setup
>   * test-kernel-security-test.py
> 
> All the 125 sub-tests are executed in test-kernel-security-test.py,
> it's tidious to read the report every cycle for some specific failure
> and hinting everything there is a bit risky for this security related
> test.
> 
> We should probably ask security team to fix these broken tests or fix
> these issues in our kernel, but there is always an exception.
> 
> Improve this test suite by running sub test case one-by-one with:
>     python2 test-kernel-security.py -v TestClass.TestMethod
> 
> To retrieve tests from test-kernel-security.py, Use grep here is more
> straightforward then trying to load a module with '-' in the filename.
> We will have to grep for the class and the method (test case), then
> we parse and constuct a dictionary with them. Downside is this will be
> highly dependent to the script structure, should be ok as this won't
> change too much under python unittest framework.
> 
> Time cost is the tradeoff here, it will increase from 1m30s to about
> 5m45s, but I think it defintely worth it.
> 
> Patch tested with Focal OEM-5.14.
> 
> Signed-off-by: Po-Hsu Lin <po-hsu.lin@canonical.com>
Acked-by: Francis Ginther <francis.ginther@canonical.com>
> ---
>  ubuntu_qrt_kernel_security/control            | 19 ++++++++++++++++++-
>  .../ubuntu_qrt_kernel_security.py             |  8 ++++----
>  2 files changed, 22 insertions(+), 5 deletions(-)
> 
> diff --git a/ubuntu_qrt_kernel_security/control b/ubuntu_qrt_kernel_security/control
> index 98a70be3..567db4d3 100644
> --- a/ubuntu_qrt_kernel_security/control
> +++ b/ubuntu_qrt_kernel_security/control
> @@ -12,6 +12,23 @@ result = job.run_test_detail(NAME, test_name='setup', tag='setup', timeout=60*15
>  if result == 'ERROR':
>      print("ERROR: test failed to build, skipping all the sub tests")
>  else:
> -    result = job.run_test_detail(NAME, test_name='test-kernel-security.py', tag='test-kernel-security.py', timeout=(60*60*2))
> +    fn = os.path.join(job.bindir, 'tmp', NAME, 'src', 'qa-regression-testing', 'scripts', 'test-kernel-security.py')
> +    cmd = 'grep -e "class .*Test.*:" -e "def test_.*:" {} | sed "s/(.*)://"'.format(fn)
> +    output = utils.system_output(cmd, verbose=False).splitlines()
> +    all_tests = {}
> +    for line in output:
> +        item = line.strip()
> +        if item.startswith('class '):
> +            category = item.replace('class ', '')
> +            all_tests[category] = []
> +        elif item.startswith('def '):
> +            sub_test = item.replace('def ', '')
> +            all_tests[category].append(sub_test)
> +
> +    for category in all_tests:
> +        if all_tests[category]:
> +            for sub_test in all_tests[category]:
> +                test = '{}.{}'.format(category, sub_test)
> +                job.run_test_detail(NAME, test_name=test, tag=test, timeout=(60*5))
>  
>  # vi:set ts=4 sw=4 expandtab syntax=python:
> diff --git a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> index e6b0543d..f9e39013 100644
> --- a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> +++ b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py
> @@ -51,13 +51,13 @@ class ubuntu_qrt_kernel_security(test.test):
>          print("Test suite HEAD SHA1: {}".format(sha1))
>  
>      def run_once(self, test_name):
> -        scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts')
> -        os.chdir(scripts)
> -
>          if test_name == 'setup':
>              return
>  
> -        cmd = 'python2 ./%s -v' % test_name
> +        scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts')
> +        os.chdir(scripts)
> +
> +        cmd = 'python2 ./test-kernel-security.py -v %s' % test_name
>          self.results = utils.system_output(cmd, retain_output=True)
>  
>  
> -- 
> 2.25.1
> 
> 
> -- 
> kernel-team mailing list
> kernel-team@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team

Applied and pushed.

Thanks
Sam