From patchwork Wed Nov 6 10:29:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Po-Hsu Lin X-Patchwork-Id: 1190261 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 477N7Y5PCVz9sPk; Wed, 6 Nov 2019 21:29:29 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1iSIZ3-0005Ke-Ql; Wed, 06 Nov 2019 10:29:25 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1iSIZ1-0005Jv-PD for kernel-team@lists.ubuntu.com; Wed, 06 Nov 2019 10:29:23 +0000 Received: from mail-pl1-f197.google.com ([209.85.214.197]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1iSIZ1-0003s5-CS for kernel-team@lists.ubuntu.com; Wed, 06 Nov 2019 10:29:23 +0000 Received: by mail-pl1-f197.google.com with SMTP id f7so4404319plj.12 for ; Wed, 06 Nov 2019 02:29:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=4Y9lmH/MVE8rpKddJtc3qC4QgnRU4FtFqtV8j6sjAJ4=; b=WyC7QKj7X6ZVC85IE01c5Jut/tU5KxU8m0pACgZXiYrLzs00cK7Ou3h5i3Fxeoo47R CtYVsvre5PnQZuf3827qDkYbDfoGeaj41g8DAGggqcaWanklIGvTxlhnfvYIa3NEGuFN sMlpS31imsexYf0filNF1D7/xJnpaFbNTkQi0OQOz6ZpiWN4727nJxihyboJqqp2c3g2 ec269TjwiZTl/kB0Gobf2KuIswvGnQN1+mCoBwx+LMZrZyar7DyEW3/NVJzFiC4aPQNW KXJ9PX0cpAuDp13ADDAOW5mEXtFlMoyV6CQ2AiOH1KL1p1w2Mgm8+mDTc87h8hVr8VPT sIPQ== X-Gm-Message-State: APjAAAX4Tp0gKpImocbgp+2mMpRdmAB5QtPYia2l9NnBiFggXtbxQtB+ amZtwGEG3fO5Izco9Bly1aDlkPs3sUXI7awwTPONtNEvN+TBtudfEDY2Pz3AfXgBVXxsg0hLQsA 7USbdxFDl7Tdvo0GBRY+E8/8h2YIgQ4wiiHr9thAO X-Received: by 2002:aa7:9467:: with SMTP id t7mr2457681pfq.142.1573036161549; Wed, 06 Nov 2019 02:29:21 -0800 (PST) X-Google-Smtp-Source: APXvYqxjBQIMuFfa/fp7rNGgHyUwwOCSwDgGzV/wGudObClbPG4A2zHLjs0dtl6NvNnIEwsHGqEkIQ== X-Received: by 2002:aa7:9467:: with SMTP id t7mr2457655pfq.142.1573036161295; Wed, 06 Nov 2019 02:29:21 -0800 (PST) Received: from Leggiero.taipei.internal (61-220-137-37.HINET-IP.hinet.net. [61.220.137.37]) by smtp.gmail.com with ESMTPSA id x10sm19872046pgl.53.2019.11.06.02.29.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Nov 2019 02:29:20 -0800 (PST) From: Po-Hsu Lin To: kernel-team@lists.ubuntu.com Subject: [Bionic][SRU][PATCH 4/4] selftests: lib.mk: add SKIP handling to RUN_TESTS define Date: Wed, 6 Nov 2019 18:29:06 +0800 Message-Id: <20191106102906.28118-5-po-hsu.lin@canonical.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20191106102906.28118-1-po-hsu.lin@canonical.com> References: <20191106102906.28118-1-po-hsu.lin@canonical.com> X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "Shuah Khan (Samsung OSG)" BugLink: https://bugs.launchpad.net/bugs/1812352 RUN_TESTS which is the common function that implements run_tests target, treats all non-zero return codes from tests as failures. When tests are skipped with non-zero return code, because of unmet dependencies and/or unsupported configuration, it reports them as failed. This will lead to too many false negatives even on the tests that couldn't be run. RUN_TESTS is changed to test for SKIP=4 return from tests to enable the framework for individual tests to return special SKIP code. Tests will be changed as needed to report SKIP instead FAIL/PASS when they get skipped. Signed-off-by: Shuah Khan (Samsung OSG) (cherry picked from commit 3f4435b5149372b3bbc5acab5c835d490490d6bc) Signed-off-by: Po-Hsu Lin --- tools/testing/selftests/lib.mk | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk index 0b76e4a..02273bf 100644 --- a/tools/testing/selftests/lib.mk +++ b/tools/testing/selftests/lib.mk @@ -34,6 +34,7 @@ endif define RUN_TESTS @export KSFT_TAP_LEVEL=`echo 1`; \ test_num=`echo 0`; \ + skip=`echo 4`; \ echo "TAP version 13"; \ for TEST in $(1); do \ BASENAME_TEST=`basename $$TEST`; \ @@ -46,9 +47,19 @@ define RUN_TESTS else \ cd `dirname $$TEST` > /dev/null; \ if [ "X$(summary)" != "X" ]; then \ - (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests: $$BASENAME_TEST [FAIL]"; \ + (./$$BASENAME_TEST > /tmp/$$BASENAME_TEST 2>&1 && \ + echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || \ + (if [ $$? -eq $$skip ]; then \ + echo "not ok 1..$$test_num selftests: $$BASENAME_TEST [SKIP]"; \ + else echo "not ok 1..$$test_num selftests: $$BASENAME_TEST [FAIL]"; \ + fi;) \ else \ - (./$$BASENAME_TEST && echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || echo "not ok 1..$$test_num selftests: $$BASENAME_TEST [FAIL]"; \ + (./$$BASENAME_TEST && \ + echo "ok 1..$$test_num selftests: $$BASENAME_TEST [PASS]") || \ + (if [ $$? -eq $$skip ]; then \ + echo "not ok 1..$$test_num selftests: $$BASENAME_TEST [SKIP]"; \ + else echo "not ok 1..$$test_num selftests: $$BASENAME_TEST [FAIL]"; \ + fi;) \ fi; \ cd - > /dev/null; \ fi; \