From patchwork Wed Jun 1 09:17:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Po-Hsu Lin X-Patchwork-Id: 1637850 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=iTa9xwnL; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LCk8b17x6z9s0w for ; Wed, 1 Jun 2022 19:18:21 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1nwKUS-0000Zg-5i; Wed, 01 Jun 2022 09:18:08 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1nwKUQ-0000Yx-3t for kernel-team@lists.ubuntu.com; Wed, 01 Jun 2022 09:18:06 +0000 Received: from mail-pg1-f198.google.com (mail-pg1-f198.google.com [209.85.215.198]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id BCF763F044 for ; Wed, 1 Jun 2022 09:18:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1654075085; bh=jDEdXU2blXf63LtSWM6egPT9oJgUBVugVUZ6ReCinI4=; h=From:To:Subject:Date:Message-Id:MIME-Version; b=iTa9xwnLgpYMqSPcI8PNc82iiZAlSo6qUa/Vn7fAkJVo4fAwkoTmobbA+8iK07iZp W6qUR6lAD13MRBuDPtNYTbb7k8pwMu+tEW6UbVPz/yWCxakI73bdb0AFSaytuGcRL/ dvWdrJrZ7JjmqOIFglaxFUvXcxuJWwn7hA0G12v9vLG2lOdBVjmiaZ4N3jtkeWAlYG SWZMtbokr83/ucdAxzTjx0snPN85phGpGlB1NCEAISAFnG9mwPSG0e2R/4KIjf1CXM 7aVQUjf5I53iwoN1p/NaCq8MycFOF04picIBpBPWX9rWKB62NDX8oNoVtXz7PU7GNW elwkd0Y111vFw== Received: by mail-pg1-f198.google.com with SMTP id 35-20020a631563000000b003fc09751ce5so776050pgv.10 for ; Wed, 01 Jun 2022 02:18:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:subject:date:message-id:mime-version :content-transfer-encoding; bh=jDEdXU2blXf63LtSWM6egPT9oJgUBVugVUZ6ReCinI4=; b=xk0fJmiIkwA/AkHJTmTSeOC22Liq4G+UpHxkqdfL2Ff7+g5SctwV2nHhIE2RbTyUkJ S/XAR48o5zW9lORw4ASLa+3iBa+qjvUuFWcL+/hpnU+USYSaR24C6sJJls8BL24jbP2i QHImdkj4g/e0hmMddgyRpfLVhWNMvoBl1UKjBOnJ4NknkO5p3QXaiws/Q3hDuGlFyNzz 7KM/OlgeJYfF0X1P9Vlw8MFc/UKOrynOdCgo4ZaHrMv6VeuK6sCN7lB74hq61EG4RVSg MP6QUkbbMPoF5iJx6h08IZBHxJtVUmcOWyYT6xicxb7VwLty2XhT84YnDrnLxJOOjUaR 6wRg== X-Gm-Message-State: AOAM532UogE8NWpDxkHc6R2FEV+rnQGVO3typl32gDsUJestzjDRPDdd SLSI/OnRFCNjW7I9krof0efE0m2iIuf89M/lRcAQ/Ui/lKGHsAycXtISU3NwN8iKtuRXAKY5L+9 FjHfQXjTIolvVPNFGR88tlElRH2FVfqiKVmwYs+tH X-Received: by 2002:a17:90b:4d11:b0:1dc:ec4f:a19c with SMTP id mw17-20020a17090b4d1100b001dcec4fa19cmr33019414pjb.117.1654075084493; Wed, 01 Jun 2022 02:18:04 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzMwd1lsLahV4Qw6WRHYDaI/w1NrxmSwx+BvODkFx1eXQ2WqB8QD01umBi2YojPdhKjJsfpOA== X-Received: by 2002:a17:90b:4d11:b0:1dc:ec4f:a19c with SMTP id mw17-20020a17090b4d1100b001dcec4fa19cmr33019386pjb.117.1654075084095; Wed, 01 Jun 2022 02:18:04 -0700 (PDT) Received: from localhost.localdomain (111-71-42-39.emome-ip.hinet.net. [111.71.42.39]) by smtp.gmail.com with ESMTPSA id o4-20020a170902d4c400b0016230703ca3sm1035930plg.231.2022.06.01.02.18.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 02:18:03 -0700 (PDT) From: Po-Hsu Lin To: kernel-team@lists.ubuntu.com Subject: [autotest-client-tests][PATCH] UBUNTU: SAUCE: ubuntu_qrt_kernel_security: improve test granularity Date: Wed, 1 Jun 2022 17:17:25 +0800 Message-Id: <20220601091725.257642-1-po-hsu.lin@canonical.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/1976344 For now we have only 2 cases in this ubuntu_qrt_kernel_security test: * setup * test-kernel-security-test.py All the 125 sub-tests are executed in test-kernel-security-test.py, it's tidious to read the report every cycle for some specific failure and hinting everything there is a bit risky for this security related test. We should probably ask security team to fix these broken tests or fix these issues in our kernel, but there is always an exception. Improve this test suite by running sub test case one-by-one with: python2 test-kernel-security.py -v TestClass.TestMethod To retrieve tests from test-kernel-security.py, Use grep here is more straightforward then trying to load a module with '-' in the filename. We will have to grep for the class and the method (test case), then we parse and constuct a dictionary with them. Downside is this will be highly dependent to the script structure, should be ok as this won't change too much under python unittest framework. Time cost is the tradeoff here, it will increase from 1m30s to about 5m45s, but I think it defintely worth it. Patch tested with Focal OEM-5.14. Signed-off-by: Po-Hsu Lin Acked-by: Francis Ginther --- ubuntu_qrt_kernel_security/control | 19 ++++++++++++++++++- .../ubuntu_qrt_kernel_security.py | 8 ++++---- 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/ubuntu_qrt_kernel_security/control b/ubuntu_qrt_kernel_security/control index 98a70be3..567db4d3 100644 --- a/ubuntu_qrt_kernel_security/control +++ b/ubuntu_qrt_kernel_security/control @@ -12,6 +12,23 @@ result = job.run_test_detail(NAME, test_name='setup', tag='setup', timeout=60*15 if result == 'ERROR': print("ERROR: test failed to build, skipping all the sub tests") else: - result = job.run_test_detail(NAME, test_name='test-kernel-security.py', tag='test-kernel-security.py', timeout=(60*60*2)) + fn = os.path.join(job.bindir, 'tmp', NAME, 'src', 'qa-regression-testing', 'scripts', 'test-kernel-security.py') + cmd = 'grep -e "class .*Test.*:" -e "def test_.*:" {} | sed "s/(.*)://"'.format(fn) + output = utils.system_output(cmd, verbose=False).splitlines() + all_tests = {} + for line in output: + item = line.strip() + if item.startswith('class '): + category = item.replace('class ', '') + all_tests[category] = [] + elif item.startswith('def '): + sub_test = item.replace('def ', '') + all_tests[category].append(sub_test) + + for category in all_tests: + if all_tests[category]: + for sub_test in all_tests[category]: + test = '{}.{}'.format(category, sub_test) + job.run_test_detail(NAME, test_name=test, tag=test, timeout=(60*5)) # vi:set ts=4 sw=4 expandtab syntax=python: diff --git a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py index e6b0543d..f9e39013 100644 --- a/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py +++ b/ubuntu_qrt_kernel_security/ubuntu_qrt_kernel_security.py @@ -51,13 +51,13 @@ class ubuntu_qrt_kernel_security(test.test): print("Test suite HEAD SHA1: {}".format(sha1)) def run_once(self, test_name): - scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts') - os.chdir(scripts) - if test_name == 'setup': return - cmd = 'python2 ./%s -v' % test_name + scripts = os.path.join(self.srcdir, 'qa-regression-testing', 'scripts') + os.chdir(scripts) + + cmd = 'python2 ./test-kernel-security.py -v %s' % test_name self.results = utils.system_output(cmd, retain_output=True)