From patchwork Tue Mar 31 19:41:54 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218255 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=mTEqAbBy; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=WZlbCfzN; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldmw72GSz1yCp for ; Wed, 01 Apr 2026 06:42:20 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=rrsCu9qjymXrEqKnY0GhwJMBfZPl4QJ9pbACV/z62I8=; b=mTEqAbByNYZWRFjsD8N9qPPyno gsSMe3K1Sgr4fw+QwEcJMwNBiqfFaCoKXd0QltqbTZ8JSvp6dvC5WIJlio7go0g5RylV6EslVBhzY lvlEchavHt3GL5cFXwHj0A9UCaPAtBlErawnYtdvCyxYYQVlYaIxiB7Dc7i4H5oV+5x1DJVlS4Qff VRIa9ZhkKrNdEB51Y9K8qiOmbR3U0FtNp29iqkBgzo/zA/Hxj/6F2ADBFGVSO9wckaa3tR9rJCjTS 3vgXLEsVMNmywbZ6iMfHx7byYsNQrDVeDkLtcrRnoCia2k3szk8Fw4xs8LwNZaPkeK4amTKzD0kEA AoS2a1BA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyf-0000000DULz-2bSX; Tue, 31 Mar 2026 19:42:17 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyd-0000000DUL2-2037 for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:16 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-35842aa350fso17796062a91.0 for ; Tue, 31 Mar 2026 12:42:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986134; x=1775590934; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=77Z+jquwwk0zk6Vwhrc3psuAUoaKxjDwCdLNMqluKDo=; b=WZlbCfzN84qSI7exEWNkjk5FIGo8lw1MM/H6uEC0aidExQ1cxhizbkwpx7TQRpPYyX /e+clcUxUPL8anHod4/W1eYuZtxTOF6tXp+TLbsPv+5VAbo3B4tLIN4/Saw+1UGGsQif gER/NeYCvHU5leEyaBAjnZ55EFdUUtaJ9CE7HmWvQWdO/8ZLx2sDfS2TAvGU/KSX5sKe GoPLx33sWbJr/8B9wfO4j8ho0gd1e+K0ORA7kWqpeTfRaDrJ5/UucpIzAIUQOr33F7nH Q4wVrX7vrAxnz1Kb0hIPay97HTWBbNr39GYy7SiqoJMVKc49GO16HCLi+mPy2iXvuNM6 uJpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986134; x=1775590934; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=77Z+jquwwk0zk6Vwhrc3psuAUoaKxjDwCdLNMqluKDo=; b=LyKwRhOORFcXpMPAzJ4aENIyFkoZXZvTV9gELlB9bSu12WOh1MOh+5jRqNn+bR3Zcw tyZzQAUjUotAOqN8rsGDvJR+mKlT81D0SVW+g8tMZLyXQFtgXMd40aQ2J1NpOjzM67dC 2sOChxYSZdo9f3AhBQJaiZ8yndCv/rF3sZTkngIN61NdLzk4eoXcBjPWoH37TdqeA1DN fLWZoEmq19GMdyKx6AXGnJcwqE/S5boLTJ6ax8p0zlKZ7ji46+m1XY6nIrXE1fOgn4nl +CH6bVtFVWL//vCQ4JEMAAghrPRb/Ztl2ltBBtJEN0lBiBWMBe+eU51r/6dnDWYRpoZp Dqgw== X-Forwarded-Encrypted: i=1; AJvYcCWg09hOauduWDgXyurr8fQNFMmybEDdWlDMTUtV+RhX8JW34eceF3G9UP8QOWvChb5/ssTLMgdueao=@lists.infradead.org X-Gm-Message-State: AOJu0Ywsqy0qsvlpMdZe6o3IdDJqr9oF0goebP4USRTaHyByZHQMaHlq yiaoQ1yqu8cExpa8Uyz+tQe1NSLQTia8ZY/xqu/Dvg7fRHpA2P08j1YEeizC7+95O2BVU5CJcS0 J00y+TXnpIA== X-Received: from pjbmz13.prod.google.com ([2002:a17:90b:378d:b0:35c:217b:ecd7]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2ecb:b0:35b:ea35:c3ce with SMTP id 98e67ed59e1d1-35dc6f93abfmr482770a91.27.1774986134029; Tue, 31 Mar 2026 12:42:14 -0700 (PDT) Date: Tue, 31 Mar 2026 12:41:54 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-2-vipinsh@google.com> Subject: [PATCH v4 1/9] KVM: selftest: Create KVM selftest runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124215_525381_A35DC632 X-CRM114-Status: GOOD ( 18.39 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Implement a basic KVM selftest runner in Python to run selftests. Add command line options to select individual testcase file or a directory containing multiple testcase files. After selecting the tests to run, start their execution and print their final execution status (passed, failed, skipped, no run), stdout and stderr on terminal. Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1049 listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Implement a basic KVM selftest runner in Python to run selftests. Add command line options to select individual testcase file or a directory containing multiple testcase files. After selecting the tests to run, start their execution and print their final execution status (passed, failed, skipped, no run), stdout and stderr on terminal. Print execution status in colors on the terminals where it is supported to easily distinguish different statuses of the tests execution. If a test fails or times out, then return with a non-zero exit code after all of the tests execution have completed. If none of the tests fails or times out then exit with status 0 Provide some sample test configuration files to demonstrate the execution of the runner. Runner can be started from tools/testing/selftests/kvm directory as: python3 runner --dirs tests OR python3 runner --testcases \ tests/dirty_log_perf_test/no_dirty_log_protect.test This is a very basic implementation of the runner. Next patches will enhance the runner by adding more features like parallelization, dumping output to file system, time limit, out-of-tree builds run, etc. Signed-off-by: Vipin Sharma --- tools/testing/selftests/kvm/.gitignore | 4 +- .../testing/selftests/kvm/runner/__main__.py | 99 +++++++++++++++++++ .../testing/selftests/kvm/runner/selftest.py | 66 +++++++++++++ .../selftests/kvm/runner/test_runner.py | 37 +++++++ .../2slot_5vcpu_10iter.test | 1 + .../no_dirty_log_protect.test | 1 + 6 files changed, 207 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/kvm/runner/__main__.py create mode 100644 tools/testing/selftests/kvm/runner/selftest.py create mode 100644 tools/testing/selftests/kvm/runner/test_runner.py create mode 100644 tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test create mode 100644 tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index 1d41a046a7bf..95af97b1ff9e 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -3,10 +3,12 @@ !/**/ !*.c !*.h +!*.py !*.S !*.sh +!*.test !.gitignore !config !settings !Makefile -!Makefile.kvm \ No newline at end of file +!Makefile.kvm diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py new file mode 100644 index 000000000000..db87f426331d --- /dev/null +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -0,0 +1,99 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2025 Google LLC +# +# Author: vipinsh@google.com (Vipin Sharma) + +import argparse +import logging +import os +import sys + +from test_runner import TestRunner +from selftest import SelftestStatus + + +def cli(): + parser = argparse.ArgumentParser( + prog="KVM Selftests Runner", + formatter_class=argparse.RawTextHelpFormatter, + allow_abbrev=False + ) + + parser.add_argument("-t", + "--testcases", + nargs="*", + default=[], + help="Testcases to run. Provide the space separated testcases paths") + + parser.add_argument("-d", + "--dirs", + nargs="*", + default=[], + help="Run the testcases present in the given directory and all of its sub directories. Provide the space separated paths to add multiple directories.") + + return parser.parse_args() + + +def setup_logging(): + class TerminalColorFormatter(logging.Formatter): + reset = "\033[0m" + red_bold = "\033[31;1m" + green = "\033[32m" + yellow = "\033[33m" + blue = "\033[34m" + + COLORS = { + SelftestStatus.PASSED: green, + SelftestStatus.NO_RUN: blue, + SelftestStatus.SKIPPED: yellow, + SelftestStatus.FAILED: red_bold + } + + def __init__(self, fmt=None, datefmt=None): + super().__init__(fmt, datefmt) + + def format(self, record): + return (self.COLORS.get(record.levelno, "") + + super().format(record) + self.reset) + + logger = logging.getLogger("runner") + logger.setLevel(logging.INFO) + + ch = logging.StreamHandler() + ch_formatter = TerminalColorFormatter(fmt="%(asctime)s | %(message)s", + datefmt="%H:%M:%S") + ch.setFormatter(ch_formatter) + logger.addHandler(ch) + + +def fetch_testcases_in_dirs(dirs): + testcases = [] + for dir in dirs: + for root, child_dirs, files in os.walk(dir): + for file in files: + testcases.append(os.path.join(root, file)) + return testcases + + +def fetch_testcases(args): + testcases = args.testcases + testcases.extend(fetch_testcases_in_dirs(args.dirs)) + # Remove duplicates + testcases = list(dict.fromkeys(testcases)) + return testcases + + +def main(): + args = cli() + setup_logging() + testcases = fetch_testcases(args) + return TestRunner(testcases).start() + + +if __name__ == "__main__": + PYTHON_VERSION = (3, 6) + if sys.version_info < PYTHON_VERSION: + print(f"Minimum required python version {PYTHON_VERSION}, found {sys.version}") + sys.exit(1) + + sys.exit(main()) diff --git a/tools/testing/selftests/kvm/runner/selftest.py b/tools/testing/selftests/kvm/runner/selftest.py new file mode 100644 index 000000000000..5a86f9ceedd4 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/selftest.py @@ -0,0 +1,66 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2025 Google LLC +# +# Author: vipinsh@google.com (Vipin Sharma) + +import pathlib +import enum +import os +import subprocess + +class SelftestStatus(enum.IntEnum): + """ + Selftest Status. Integer values are just +1 to the logging.INFO level. + """ + + PASSED = 21 + NO_RUN = 22 + SKIPPED = 23 + FAILED = 24 + + def __str__(self): + return str.__str__(self.name) + +class Selftest: + """ + Represents a single selftest. + + Extract the test execution command from test file and executes it. + """ + + def __init__(self, test_path): + test_command = pathlib.Path(test_path).read_text().strip() + if not test_command: + raise ValueError("Empty test command in " + test_path) + + test_command = os.path.join(".", test_command) + self.exists = os.path.isfile(test_command.split(maxsplit=1)[0]) + self.test_path = test_path + self.command = test_command + self.status = SelftestStatus.NO_RUN + self.stdout = "" + self.stderr = "" + + def run(self): + if not self.exists: + self.stderr = "File doesn't exist." + return + + run_args = { + "universal_newlines": True, + "shell": True, + "stdout": subprocess.PIPE, + "stderr": subprocess.PIPE + } + proc = subprocess.run(self.command, **run_args) + + out, err = proc.stdout, proc.stderr + self.stdout = out.decode("utf-8", "replace") if isinstance(out, bytes) else (out or "") + self.stderr = err.decode("utf-8", "replace") if isinstance(err, bytes) else (err or "") + + if proc.returncode == 0: + self.status = SelftestStatus.PASSED + elif proc.returncode == 4: + self.status = SelftestStatus.SKIPPED + else: + self.status = SelftestStatus.FAILED diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py new file mode 100644 index 000000000000..4418777d75e3 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -0,0 +1,37 @@ +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2025 Google LLC +# +# Author: vipinsh@google.com (Vipin Sharma) + +import logging +from selftest import Selftest +from selftest import SelftestStatus + +logger = logging.getLogger("runner") + + +class TestRunner: + def __init__(self, testcases): + self.tests = [] + + for testcase in testcases: + self.tests.append(Selftest(testcase)) + + def _log_result(self, test_result): + logger.info("*** stdout ***\n" + test_result.stdout) + logger.info("*** stderr ***\n" + test_result.stderr) + logger.log(test_result.status, + f"[{test_result.status.name}] {test_result.test_path}") + + def start(self): + ret = 0 + + for test in self.tests: + test.run() + self._log_result(test) + + if (test.status not in [SelftestStatus.PASSED, + SelftestStatus.NO_RUN, + SelftestStatus.SKIPPED]): + ret = 1 + return ret diff --git a/tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test new file mode 100644 index 000000000000..5b8d56b44a75 --- /dev/null +++ b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/2slot_5vcpu_10iter.test @@ -0,0 +1 @@ +dirty_log_perf_test -x 2 -v 5 -i 10 diff --git a/tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test new file mode 100644 index 000000000000..ed3490b1d1a1 --- /dev/null +++ b/tools/testing/selftests/kvm/tests/dirty_log_perf_test/no_dirty_log_protect.test @@ -0,0 +1 @@ +dirty_log_perf_test -g From patchwork Tue Mar 31 19:41:55 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218254 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=kC8efRlw; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=B8zFQLjL; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldmx009Nz1yHD for ; Wed, 01 Apr 2026 06:42:21 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8ULEXUPklnWTetqY+rw+6+1lXkJPaOA8XLEvrOcD/BY=; b=kC8efRlwRYxqCMqoo+cmIXx/H3 Wd7QrbV8n9wZx/TiHnGz3VelCHcZHgs/3v+y2BTx9VSm3+lXWm/f0Szfw8QGUZP87dmFsuInlfWAj aWueJHNokKE2reYjbF3YGxNs38ugb+12GqCWIyB+KrZM7EDAdJqp7Z3onitQ3hZtY1t0LLlwN9JIm NnDw0/0jwKEZdv8Sbd/nHCxk0nIxk4jnv9Ju5NtTBlqTZlxJYp4Arg8tMoLI+exXmFqmQnvkgPexj cnF/9HM78b8eMODm5NVoKcustAkbm0Du/ijTG21Pp3AszZwv8eegn8usfJEmAHpZzdLGmtcmPS4tD Lvb3O+XA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyh-0000000DUNY-3pHl; Tue, 31 Mar 2026 19:42:19 +0000 Received: from mail-pl1-x64a.google.com ([2607:f8b0:4864:20::64a]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyf-0000000DULo-2kVe for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:18 +0000 Received: by mail-pl1-x64a.google.com with SMTP id d9443c01a7336-2b249541063so31964375ad.3 for ; Tue, 31 Mar 2026 12:42:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986136; x=1775590936; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=okyRWg6SJ1Nbhs5yC14NmJAon29R3bVuhvecgVY39ps=; b=B8zFQLjLv4FH0W/NQdatOkyDLLnxcb3wYl++HLIoVjkzQ3dYlWmEK7a2Bwyre7W1+v sovmrazy2SmjLmK4Xu3ATkwEAoMWC7KzSVXd2Fw6DK1mxd3AyZ1098NA7oC0jKEV3FUC rJt2crfXDsxBqCgFWS7S8XWijXwt0QXgNrsNkIPjvruNYEDppR7O4iwLTFyTY955CHpy gRiazwfyGc26L2Pe8TjNwXISu37uxjIsjYkI0SfEw1yiuRV33yUHt1fsBfgnnfIMK6cv Vy3LOTmgWvYfScF9XKTYEA9Ht8Budhzxk2e6O/xzkekNiU+nDH0tGcw6LkZ8Upzv8xNg 49wA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986136; x=1775590936; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=okyRWg6SJ1Nbhs5yC14NmJAon29R3bVuhvecgVY39ps=; b=NvzAQL/X/iYVEZaDMpZh3uG34dC1p2vccnB5zWY9CC2PxdKG4JovogcvGBjaV4TFDc H8dtNSsxQfjuKglGief1FwJCVtquTIc8zinN+SOQcNyhvKlXQ23lq1K8v2ufACOlxE/m vMmULs9N4bQ59JolzLskln1FjlX8Ys+2fAdjhjM72Va6WUOY7R5olZW5rHLKIsEF/P9X 8qZi5pq2K0/qcRTUisx8Rr6m2eTWmOv/0Z/W0Ca+NV6CDO6BSEH4Y4RcIplTO1hYgu9I lH/gwrjDZm8EulwimHKY5qLxztoUDuJrAtXA2fiTDJ6YQZpAa+xSS+NvNJzuoDtIwYgC /PPQ== X-Forwarded-Encrypted: i=1; AJvYcCU4KU77WKvk8k0fL3ufJW3ACTMzYtc6G9T0abLy+elLlncTpYtTQlJqlMkMSLLPbyqcqwpwnMlsAGM=@lists.infradead.org X-Gm-Message-State: AOJu0Yw6NFm5wV5HKgMvYqGmErkgjbNSCfoE5KJ4NQuyh/zSP/KL2yAc /lDeysQ3IaWgqq9fH/WxXMJi5dDND/bLLO6ESxQUsByKlmA/N2g9UStusVQPXcp95wp2f+5zdpB V3rqeUaBzAQ== X-Received: from plbmm7.prod.google.com ([2002:a17:903:a07:b0:2b0:4b14:f0e9]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:fb0:b0:2ae:5eee:7a5 with SMTP id d9443c01a7336-2b269aca7a0mr4064425ad.12.1774986135979; Tue, 31 Mar 2026 12:42:15 -0700 (PDT) Date: Tue, 31 Mar 2026 12:41:55 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-3-vipinsh@google.com> Subject: [PATCH v4 2/9] KVM: selftests: Provide executables path option to the KVM selftest runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124217_717543_6363944A X-CRM114-Status: GOOD ( 11.42 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Add command line option, -p/--path, to specify the directory where test binaries exist. If this option is not provided then default to the current directory. Example: python3 runner --dirs test -p ~/build/selftests Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:64a listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Add command line option, -p/--path, to specify the directory where test binaries exist. If this option is not provided then default to the current directory. Example: python3 runner --dirs test -p ~/build/selftests This option enables executing tests from out-of-tree builds. Signed-off-by: Vipin Sharma --- tools/testing/selftests/kvm/runner/__main__.py | 8 +++++++- tools/testing/selftests/kvm/runner/selftest.py | 4 ++-- tools/testing/selftests/kvm/runner/test_runner.py | 4 ++-- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py index db87f426331d..9b6c78e69c64 100644 --- a/tools/testing/selftests/kvm/runner/__main__.py +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -31,6 +31,12 @@ def cli(): default=[], help="Run the testcases present in the given directory and all of its sub directories. Provide the space separated paths to add multiple directories.") + parser.add_argument("-p", + "--path", + nargs='?', + default=".", + help="Finds the test executables in the given path. Default is the current directory.") + return parser.parse_args() @@ -87,7 +93,7 @@ def main(): args = cli() setup_logging() testcases = fetch_testcases(args) - return TestRunner(testcases).start() + return TestRunner(testcases, args).start() if __name__ == "__main__": diff --git a/tools/testing/selftests/kvm/runner/selftest.py b/tools/testing/selftests/kvm/runner/selftest.py index 5a86f9ceedd4..c1720897c629 100644 --- a/tools/testing/selftests/kvm/runner/selftest.py +++ b/tools/testing/selftests/kvm/runner/selftest.py @@ -28,12 +28,12 @@ class Selftest: Extract the test execution command from test file and executes it. """ - def __init__(self, test_path): + def __init__(self, test_path, path): test_command = pathlib.Path(test_path).read_text().strip() if not test_command: raise ValueError("Empty test command in " + test_path) - test_command = os.path.join(".", test_command) + test_command = os.path.join(path, test_command) self.exists = os.path.isfile(test_command.split(maxsplit=1)[0]) self.test_path = test_path self.command = test_command diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py index 4418777d75e3..acc9fb3dabde 100644 --- a/tools/testing/selftests/kvm/runner/test_runner.py +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -11,11 +11,11 @@ logger = logging.getLogger("runner") class TestRunner: - def __init__(self, testcases): + def __init__(self, testcases, args): self.tests = [] for testcase in testcases: - self.tests.append(Selftest(testcase)) + self.tests.append(Selftest(testcase, args.path)) def _log_result(self, test_result): logger.info("*** stdout ***\n" + test_result.stdout) From patchwork Tue Mar 31 19:41:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218253 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=QunV8Wft; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=XPQiGqe4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldmw74C3z1yGw for ; Wed, 01 Apr 2026 06:42:23 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Y8en7RrU9TKo+wcGtT46H6bZHTSHUsG3Z/zvqdDXL7s=; b=QunV8WftmoM1fY1b13tMqhmjU6 3Z4dMqy/AHG1wH+wgikKuupFOdoJMsa48NB+t93OJiToC/87Jfco4jVNWw3QTYmXSpsVz8g9KevyR /W9xeVP8YVB0uj3zEAI8hJGygst5xitLDEKh48sPkUeaZ/VOTGnkamTWNAFpNe0QK6yXFX0HJO/x9 fjGnEmEomPbuqcYPKLZkj2C0axysRz7ncXQJ9Nn2eY1/IpCn0hXd4QWtEToQS5nk0JrIEMGTd1PX3 RvzL7V+zXwa2F8yNlHBzcToFEY7Su9J8Nw9laiOC/hct/SgTXX6xCRibgO24hB84UwMxCtpZAcS8H F2gdk6ww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyk-0000000DUOw-1bLs; Tue, 31 Mar 2026 19:42:22 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyh-0000000DUMn-36Vp for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:20 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-35678f99c6eso4452663a91.1 for ; Tue, 31 Mar 2026 12:42:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986138; x=1775590938; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MqTZM8Atvognfn8r5xuaknIcSyhPV2VEo05zIHjpojQ=; b=XPQiGqe4Vn5S8/5MQ8pVFlCl9REO3KTz8FpLFhqpqvj+TXRP8eOwfQTUFlKAAj/ezc 95/UthwcMvDAJHgFkqeKHzVk4aMtWFlG46Ln/uFmTCk77pulqDK9nPMK9Ap650sqYD1w 9PWeBpDPROfvcgYvfz+d07/Y/oBJML4cD+04Q2jHSN5uObqaUJD8xZUwPJluyYoC5BoB AQpknnzMrsfh4c1xfLB8A9ic2UelZLeD0AWOZHBVpjDpFN2qRfnIR+17GJqdD324Zbxw Od/5y1c82u/MZgkaazYsxbBQ7WW663zkCx+5vkKUsLJx3vft1GNa4MqUpjGjWstVdslW 2oDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986138; x=1775590938; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MqTZM8Atvognfn8r5xuaknIcSyhPV2VEo05zIHjpojQ=; b=sr48nw3j7Yl7B1QBbmWrtRbO+AI4YmYUAfbCSlLniu1QjxF9KIMxrhzKXBaPbZbun6 +h2Q1jgi2H3NBkxp//k7HHNSK9Fsm+bELJ9bTKuAKYlGb0nntlNGawog2iTvEHGOlrxT 5bhaLOln9JgjerdEn/v7jzersXfje1E+1v2YA3152cYv4ECQBdr5r35J4zMFRWqvSXiT Gsxqkt1vzNXzuECHBSDY9emZqmiNCQ2A7oFYNjSTZF3QN5dkE0SGQe1wvLixmAAhe5p+ yS6jBBOHZFKdTpNc7xRc/jBvnHVKTDiVtSIgm7P8ZcKkN25Pgs3MUxETe5mwXzjyZIC7 YJ+w== X-Forwarded-Encrypted: i=1; AJvYcCUdOMmyGMMLyXzr1BStnK8MTjyqsn6nqJXl/SYrhLGOH8G0qZMJQv0+22TjSfUiPsZnUYLZ/9zns2E=@lists.infradead.org X-Gm-Message-State: AOJu0Yxzg1pxzPi+mwzfJu/ocD3wSSSG6hQZsKRi2Elaalbk4xD4iFMz GDSvJulTpNzgZ+UFLr5emFlm0bd//uSfTUCLkwwWHOzZtZ2bj3kw9AW7y2idEoDkSG+MI46BDUy 8cHBRICvDsA== X-Received: from pjbqi4.prod.google.com ([2002:a17:90b:2744:b0:35b:993f:f936]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:57c4:b0:35c:d37:6024 with SMTP id 98e67ed59e1d1-35dc6edbfd0mr555200a91.11.1774986138004; Tue, 31 Mar 2026 12:42:18 -0700 (PDT) Date: Tue, 31 Mar 2026 12:41:56 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-4-vipinsh@google.com> Subject: [PATCH v4 3/9] KVM: selftests: Add timeout option in selftests runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124219_784870_A9BB27EF X-CRM114-Status: GOOD ( 12.89 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Add a command line argument in KVM selftest runner to limit amount of time (seconds) given to a test for execution. Kill the test if it exceeds the given timeout. Define a new SelftestStatus.TIMED_OUT [...] Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1049 listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Add a command line argument in KVM selftest runner to limit amount of time (seconds) given to a test for execution. Kill the test if it exceeds the given timeout. Define a new SelftestStatus.TIMED_OUT to denote a selftest final result. Add terminal color for status messages of timed out tests. Set the default value of 120 seconds for all tests. Signed-off-by: Vipin Sharma --- .../testing/selftests/kvm/runner/__main__.py | 9 +++++- .../testing/selftests/kvm/runner/selftest.py | 29 ++++++++++++------- .../selftests/kvm/runner/test_runner.py | 2 +- 3 files changed, 27 insertions(+), 13 deletions(-) diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py index 9b6c78e69c64..82c543d11c34 100644 --- a/tools/testing/selftests/kvm/runner/__main__.py +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -37,6 +37,11 @@ def cli(): default=".", help="Finds the test executables in the given path. Default is the current directory.") + parser.add_argument("--timeout", + default=120, + type=int, + help="Timeout, in seconds, before runner kills the running test. (Default: 120 seconds)") + return parser.parse_args() @@ -44,6 +49,7 @@ def setup_logging(): class TerminalColorFormatter(logging.Formatter): reset = "\033[0m" red_bold = "\033[31;1m" + red = "\033[31;1m" green = "\033[32m" yellow = "\033[33m" blue = "\033[34m" @@ -52,7 +58,8 @@ def setup_logging(): SelftestStatus.PASSED: green, SelftestStatus.NO_RUN: blue, SelftestStatus.SKIPPED: yellow, - SelftestStatus.FAILED: red_bold + SelftestStatus.FAILED: red_bold, + SelftestStatus.TIMED_OUT: red } def __init__(self, fmt=None, datefmt=None): diff --git a/tools/testing/selftests/kvm/runner/selftest.py b/tools/testing/selftests/kvm/runner/selftest.py index c1720897c629..e3924ee40ab3 100644 --- a/tools/testing/selftests/kvm/runner/selftest.py +++ b/tools/testing/selftests/kvm/runner/selftest.py @@ -17,6 +17,7 @@ class SelftestStatus(enum.IntEnum): NO_RUN = 22 SKIPPED = 23 FAILED = 24 + TIMED_OUT = 25 def __str__(self): return str.__str__(self.name) @@ -28,7 +29,7 @@ class Selftest: Extract the test execution command from test file and executes it. """ - def __init__(self, test_path, path): + def __init__(self, test_path, path, timeout): test_command = pathlib.Path(test_path).read_text().strip() if not test_command: raise ValueError("Empty test command in " + test_path) @@ -37,6 +38,7 @@ class Selftest: self.exists = os.path.isfile(test_command.split(maxsplit=1)[0]) self.test_path = test_path self.command = test_command + self.timeout = timeout self.status = SelftestStatus.NO_RUN self.stdout = "" self.stderr = "" @@ -50,17 +52,22 @@ class Selftest: "universal_newlines": True, "shell": True, "stdout": subprocess.PIPE, - "stderr": subprocess.PIPE + "stderr": subprocess.PIPE, + "timeout": self.timeout, } - proc = subprocess.run(self.command, **run_args) + try: + proc = subprocess.run(self.command, **run_args) + out, err = proc.stdout, proc.stderr + + if proc.returncode == 0: + self.status = SelftestStatus.PASSED + elif proc.returncode == 4: + self.status = SelftestStatus.SKIPPED + else: + self.status = SelftestStatus.FAILED + except subprocess.TimeoutExpired as e: + self.status = SelftestStatus.TIMED_OUT + out, err = e.stdout, e.stderr - out, err = proc.stdout, proc.stderr self.stdout = out.decode("utf-8", "replace") if isinstance(out, bytes) else (out or "") self.stderr = err.decode("utf-8", "replace") if isinstance(err, bytes) else (err or "") - - if proc.returncode == 0: - self.status = SelftestStatus.PASSED - elif proc.returncode == 4: - self.status = SelftestStatus.SKIPPED - else: - self.status = SelftestStatus.FAILED diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py index acc9fb3dabde..bea82c6239cd 100644 --- a/tools/testing/selftests/kvm/runner/test_runner.py +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -15,7 +15,7 @@ class TestRunner: self.tests = [] for testcase in testcases: - self.tests.append(Selftest(testcase, args.path)) + self.tests.append(Selftest(testcase, args.path, args.timeout)) def _log_result(self, test_result): logger.info("*** stdout ***\n" + test_result.stdout) From patchwork Tue Mar 31 19:41:57 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218256 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=sOr2xerJ; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=m9AYZPGp; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldmy1TtDz1yGw for ; Wed, 01 Apr 2026 06:42:26 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=hNh4Xs0AucfkIhmM5nww+AGOLdzkoQA1D4Bnpt8evlU=; b=sOr2xerJO660IUTRbe9o8c3w19 eJz8CLKJ8eq1ZDoopOSaU8EheRUvTwuIIpZcaA2Yuah6YcHHPNZZMv4HHGnLHNI202P1iPALnLMTl peTrBH/OHN49vvpNSeqhZxda8iL0SrVroy1blyhG3XYZmcKDi5zNwsuYpPfWT8aGItvGT59W6f7jZ oVFvlICdTWhR3uMqKtU4I3+ZQs7C4POu+KFQW52lXg33G28cyYtKhkdmo7m2wS7IhSNVkiba3FM7N ZTG7ZQgdB/PFhiXGlIAV1XXUfR2KDNxdti2f1DkF/ybKHB3uBNL1O45v8hEWRKTsuqS0FH0brW3Pz Mk0du0YQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eym-0000000DUQA-2kMz; Tue, 31 Mar 2026 19:42:24 +0000 Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyj-0000000DUO0-1a4j for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:22 +0000 Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-35da86144daso2790453a91.3 for ; Tue, 31 Mar 2026 12:42:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986140; x=1775590940; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ka0uEckvF83qn5GmvpHw9HtNOmToQnEieGf8Vdd2qVw=; b=m9AYZPGpS2iy8rSnRh2PoMJqsui9xQ9P079wLFLObfCoEE3by0sGQ5dbM4dE8df7lD 6LUSR+Z//Xq3WWgeSsMdb2aZdZULwqh0xGiuvyENt/uTjpc2vPrGWs0+PF8q7qsm65H3 uiHEKlcGWJINbjSZ35JNMZCz9lDhoZPnilPeIrEd3hrf0IJc8UOUdQQaPLsy0rIeU1hI ze74LTdImc9IMCkf3aTZYA8dFu9neQbd98sFOLseouFNYmS+ZNiHlaowpCs+AbDih6Df 16wliqPs0345bQO7tm+PkX/BWzbtTidr0q9O56DNe5/Z1ASV+zw/G/WPfZfxGZh+A0ij JtnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986140; x=1775590940; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ka0uEckvF83qn5GmvpHw9HtNOmToQnEieGf8Vdd2qVw=; b=qh+i9TTr5ZZci7YD3jC+2SL4qrifXJg5CnW8uJk3xgtXcPTmAq2L0Y3pt7arI1E4lr K6EBhGwqkkWor0GZnKQcvsIYrM05RmTqoUuJ54og+0IXLRfqz7wFzUW6MxOdnHGrG9Hh 5w0H+SQ3Xu7FjVnXUDAzEzVO81qPCwqG77ILpn0I2c2zCe7oHHxX4utBBJjWF1LnYFdf e6OgjTdZSU0R5Z8TsKM11PFiJsA8ghhODsHoDTZ1PGGR40OnFwdUEOLITTS6GdU1uyMn oJ7ZmkOYIey0JUIHzeN1QUGBy6r3MKneWcjee6mdxtvg6E9UAJTxHgEQUWTMgEioV/2M uiYw== X-Forwarded-Encrypted: i=1; AJvYcCUSBQrz99agIxiJyTc1jAPQAvEvFjulYVaIZS+E1G3kih5r2CNzHtw5HEtQmvWeQTY0p7uekddrP5g=@lists.infradead.org X-Gm-Message-State: AOJu0YzyPA/CyAhTs/FOyRalu0BbBZvK1zDEWtf+YqbCu8Xzf5WT32wF otkm2fPZSPz5X4TF3iRweWcSQa0TdUnHBZUo9HuGesykejvfVfbBw5O+lmgfMWeJk9QPh9upqIv H6kcOkO6rbQ== X-Received: from pjblp17.prod.google.com ([2002:a17:90b:4a91:b0:35d:a80b:8d43]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4cc2:b0:35b:e554:53a6 with SMTP id 98e67ed59e1d1-35dc6ea1153mr570258a91.11.1774986139702; Tue, 31 Mar 2026 12:42:19 -0700 (PDT) Date: Tue, 31 Mar 2026 12:41:57 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-5-vipinsh@google.com> Subject: [PATCH v4 4/9] KVM: selftests: Add option to save selftest runner output to a directory From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124221_424533_A635E3E4 X-CRM114-Status: GOOD ( 12.57 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Add a command line flag, -o/--output, to selftest runner which enables it to save individual tests output (stdout & stderr) stream to a directory in a hierarchical way. Create folder hierarchy same as [...] Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:104a listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Add a command line flag, -o/--output, to selftest runner which enables it to save individual tests output (stdout & stderr) stream to a directory in a hierarchical way. Create folder hierarchy same as tests hieararcy given by --testcases and --dirs. Also, add a command line flag, --append-output-time, which will append timestamp (format YYYY.M.DD.HH.MM.SS) to the directory name given in --output flag. Example: python3 runner --dirs test -o test_result --append_output_time This will create test_result.2025.06.06.08.45.57 directory. Signed-off-by: Vipin Sharma --- .../testing/selftests/kvm/runner/__main__.py | 34 +++++++++++++++-- .../testing/selftests/kvm/runner/selftest.py | 38 ++++++++++++++++--- .../selftests/kvm/runner/test_runner.py | 4 +- 3 files changed, 65 insertions(+), 11 deletions(-) diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py index 82c543d11c34..5ea5ee8957c4 100644 --- a/tools/testing/selftests/kvm/runner/__main__.py +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -7,6 +7,8 @@ import argparse import logging import os import sys +import datetime +import pathlib from test_runner import TestRunner from selftest import SelftestStatus @@ -42,10 +44,20 @@ def cli(): type=int, help="Timeout, in seconds, before runner kills the running test. (Default: 120 seconds)") + parser.add_argument("-o", + "--output", + nargs='?', + help="Dumps test runner output which includes each test execution result, their stdouts and stderrs hierarchically in the given directory.") + + parser.add_argument("--append-output-time", + action="store_true", + default=False, + help="Appends timestamp to the output directory.") + return parser.parse_args() -def setup_logging(): +def setup_logging(args): class TerminalColorFormatter(logging.Formatter): reset = "\033[0m" red_bold = "\033[31;1m" @@ -72,12 +84,26 @@ def setup_logging(): logger = logging.getLogger("runner") logger.setLevel(logging.INFO) + formatter_args = { + "fmt": "%(asctime)s | %(message)s", + "datefmt": "%H:%M:%S" + } + ch = logging.StreamHandler() - ch_formatter = TerminalColorFormatter(fmt="%(asctime)s | %(message)s", - datefmt="%H:%M:%S") + ch_formatter = TerminalColorFormatter(**formatter_args) ch.setFormatter(ch_formatter) logger.addHandler(ch) + if args.output != None: + if (args.append_output_time): + args.output += datetime.datetime.now().strftime(".%Y.%m.%d.%H.%M.%S") + pathlib.Path(args.output).mkdir(parents=True, exist_ok=True) + logging_file = os.path.join(args.output, "log") + fh = logging.FileHandler(logging_file) + fh_formatter = logging.Formatter(**formatter_args) + fh.setFormatter(fh_formatter) + logger.addHandler(fh) + def fetch_testcases_in_dirs(dirs): testcases = [] @@ -98,7 +124,7 @@ def fetch_testcases(args): def main(): args = cli() - setup_logging() + setup_logging(args) testcases = fetch_testcases(args) return TestRunner(testcases, args).start() diff --git a/tools/testing/selftests/kvm/runner/selftest.py b/tools/testing/selftests/kvm/runner/selftest.py index e3924ee40ab3..76c3e6d463ee 100644 --- a/tools/testing/selftests/kvm/runner/selftest.py +++ b/tools/testing/selftests/kvm/runner/selftest.py @@ -7,6 +7,7 @@ import pathlib import enum import os import subprocess +import contextlib class SelftestStatus(enum.IntEnum): """ @@ -29,7 +30,7 @@ class Selftest: Extract the test execution command from test file and executes it. """ - def __init__(self, test_path, path, timeout): + def __init__(self, test_path, path, timeout, output_dir): test_command = pathlib.Path(test_path).read_text().strip() if not test_command: raise ValueError("Empty test command in " + test_path) @@ -39,15 +40,14 @@ class Selftest: self.test_path = test_path self.command = test_command self.timeout = timeout + if output_dir is not None: + output_dir = os.path.join(output_dir, test_path.lstrip("./")) + self.output_dir = output_dir self.status = SelftestStatus.NO_RUN self.stdout = "" self.stderr = "" - def run(self): - if not self.exists: - self.stderr = "File doesn't exist." - return - + def _run(self, output=None, error=None): run_args = { "universal_newlines": True, "shell": True, @@ -70,4 +70,30 @@ class Selftest: out, err = e.stdout, e.stderr self.stdout = out.decode("utf-8", "replace") if isinstance(out, bytes) else (out or "") + if output is not None: + output.write(self.stdout) + self.stderr = err.decode("utf-8", "replace") if isinstance(err, bytes) else (err or "") + if error is not None: + error.write(self.stderr) + + def run(self): + if not self.exists: + self.stderr = "File doesn't exist." + return + + if self.output_dir is not None: + pathlib.Path(self.output_dir).mkdir(parents=True, exist_ok=True) + + output = None + error = None + with contextlib.ExitStack() as stack: + if self.output_dir is not None: + output_path = os.path.join(self.output_dir, "stdout") + output = stack.enter_context( + open(output_path, encoding="utf-8", mode="w")) + + error_path = os.path.join(self.output_dir, "stderr") + error = stack.enter_context( + open(error_path, encoding="utf-8", mode="w")) + return self._run(output, error) diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py index bea82c6239cd..b9101f0e0432 100644 --- a/tools/testing/selftests/kvm/runner/test_runner.py +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -13,9 +13,11 @@ logger = logging.getLogger("runner") class TestRunner: def __init__(self, testcases, args): self.tests = [] + self.output_dir = args.output for testcase in testcases: - self.tests.append(Selftest(testcase, args.path, args.timeout)) + self.tests.append(Selftest(testcase, args.path, args.timeout, + args.output)) def _log_result(self, test_result): logger.info("*** stdout ***\n" + test_result.stdout) From patchwork Tue Mar 31 19:41:58 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218257 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=Xz7Re44J; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=mcSDuaNw; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldmy3drHz1yCp for ; Wed, 01 Apr 2026 06:42:26 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=PxdfncEcoHLJVPg+4ylJBZ7BASy7OrhH/P8uTeU9wN4=; b=Xz7Re44Jk+R8swAbmGL0Xd7B34 NwlitgB3HjwooeJKppGGyIL/fEIZ/qkI4cYjIEFMQGumVJEoll7xSOhgRcLqcRV0XFQVHY6wVG7zt LxJBkMIe0lSZemgagJ3LSN3j6arb3eOpQEIMWyYxXL9A5SD2k20Ta1oLCiYk9zC5HJgo4Wh40bie8 0UnsAq87gqVDBIX3pjc9HhMWuOFYlaKHwRZuZjXfcngaQyRZSvDdDdIZEbIKsRl3UZeFzses5QSk0 nG8G6CZAgFAIKlZ9nh5gCWTfNczVpE3lP0qm0gsRvk4H/rZ+xQIO3Y7IClfS0B1Fhaik/rClLfVWs tQCXy8pA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eym-0000000DUQY-3rS3; Tue, 31 Mar 2026 19:42:24 +0000 Received: from mail-pf1-x44a.google.com ([2607:f8b0:4864:20::44a]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyl-0000000DUOv-14Vn for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:24 +0000 Received: by mail-pf1-x44a.google.com with SMTP id d2e1a72fcca58-82c9364612fso10001164b3a.2 for ; Tue, 31 Mar 2026 12:42:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986142; x=1775590942; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=moQABiS7cv82z1Yz7qoQD99nIXpPdwRqLpnQn6Ns6AY=; b=mcSDuaNwN2xZ6R7Fz+EyGlSRylCg5sgDlpt/vw+dULZeP5Mm9F/j7B8KVE4D6NT9yn x5eH1F0moDKf1bbAhnWu17gztkLE9QIN/vGjChPj0ylIv5PTyxjPpn1AIMfDyMw52aFa CXrAww66zIss7cO8cJUoY8PeNfBL1hbPrbZuXYkUlYs8pPoMsdrvSoZLzgHXkh2zjVXy i6qw/4qFAGzWVfmLP5K3KCSWM4JALG6N+H9efLmaTzgUdHHQVG5OwGivLx1ytWReaZsM hixO9DbhhgE4jt8yPTbdgOzrV5zNHvCXsuBHb3rCg2xhZ24eFljdKQJsbaHoqSZjdaRa Ggrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986142; x=1775590942; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=moQABiS7cv82z1Yz7qoQD99nIXpPdwRqLpnQn6Ns6AY=; b=cISLRaRIpUl5EfUsumMoYPQ6y6NXYKYenV6Mxl4YyO5HZiwz3aaYR1JIyELnYUr/vf mRPUuhXtlqiGbYg5FF2qdT/ZRcvGwWHGIlh/VvePsoe6LpOPWsD31ouKSli6UBxYTq6j gR8LIf6XkAyUJ0sKxwB3oIWOPYT40D5RMznCQSqpVaU6+5CIFB+ec6c6JsRleJZT8J4u bJCNmIpbF6h3astugiZpDQ7n+b3k7OC4PpjTNSgCPid3We2c2Qu+dhA9GWOWoUbl9ofc 3V02ot9msvef0nEZH7bBWdvpXiALbElY6hSjWhwaIjs9X6O5cyMzkBVPqfjIsVMc3XC6 KtLA== X-Forwarded-Encrypted: i=1; AJvYcCVvkFa49eJCWFIxeZuO5t2xifWIOqgG/0NPNrHKzVJcEufqmKRboFWG8bo9IedyXd4CSpAa5E1Lhn0=@lists.infradead.org X-Gm-Message-State: AOJu0Ywokh6L9EMzQD26MNgED4jFK71F8PX0CbzPxwADp/kp69/5FzLX nMvZihTjNw7Kapzr/ZFDXA9kQ9XcIcvigJEvUCTeuMoVvldhbBl7ow6+GbxKPlkYhrfCrYMrAkr Wclo882USvQ== X-Received: from pfbjf1.prod.google.com ([2002:a05:6a00:6e41:b0:821:82a1:fe7d]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:600d:b0:827:3946:a23c with SMTP id d2e1a72fcca58-82ce88d3bd7mr811773b3a.10.1774986141556; Tue, 31 Mar 2026 12:42:21 -0700 (PDT) Date: Tue, 31 Mar 2026 12:41:58 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-6-vipinsh@google.com> Subject: [PATCH v4 5/9] KVM: selftests: Run tests concurrently in KVM selftests runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124223_296137_0F73ACE3 X-CRM114-Status: GOOD ( 11.39 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Add a command line argument, --jobs, to specify how many tests can execute concurrently. Set default to 1. Example: python3 runner --test-dirs tests -j 10 Signed-off-by: Vipin Sharma --- .../testing/selftests/kvm/runner/__main__.py | 6 ++++ .../selftests/kvm/runner/test_runner.py | 28 +++++++++++++------ 2 files changed, 26 insertio [...] Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:44a listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Add a command line argument, --jobs, to specify how many tests can execute concurrently. Set default to 1. Example: python3 runner --test-dirs tests -j 10 Signed-off-by: Vipin Sharma --- .../testing/selftests/kvm/runner/__main__.py | 6 ++++ .../selftests/kvm/runner/test_runner.py | 28 +++++++++++++------ 2 files changed, 26 insertions(+), 8 deletions(-) diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py index 5ea5ee8957c4..96402c89aea9 100644 --- a/tools/testing/selftests/kvm/runner/__main__.py +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -54,6 +54,12 @@ def cli(): default=False, help="Appends timestamp to the output directory.") + parser.add_argument("-j", + "--jobs", + default=1, + type=int, + help="Maximum number of tests that can be run concurrently. (Default: 1)") + return parser.parse_args() diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py index b9101f0e0432..92eec18fe5c6 100644 --- a/tools/testing/selftests/kvm/runner/test_runner.py +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -4,6 +4,8 @@ # Author: vipinsh@google.com (Vipin Sharma) import logging +import concurrent.futures + from selftest import Selftest from selftest import SelftestStatus @@ -14,11 +16,16 @@ class TestRunner: def __init__(self, testcases, args): self.tests = [] self.output_dir = args.output + self.jobs = args.jobs for testcase in testcases: self.tests.append(Selftest(testcase, args.path, args.timeout, args.output)) + def _run_test(self, test): + test.run() + return test + def _log_result(self, test_result): logger.info("*** stdout ***\n" + test_result.stdout) logger.info("*** stderr ***\n" + test_result.stderr) @@ -28,12 +35,17 @@ class TestRunner: def start(self): ret = 0 - for test in self.tests: - test.run() - self._log_result(test) - - if (test.status not in [SelftestStatus.PASSED, - SelftestStatus.NO_RUN, - SelftestStatus.SKIPPED]): - ret = 1 + with concurrent.futures.ProcessPoolExecutor(max_workers=self.jobs) as executor: + all_futures = [] + for test in self.tests: + future = executor.submit(self._run_test, test) + all_futures.append(future) + + for future in concurrent.futures.as_completed(all_futures): + test_result = future.result() + self._log_result(test_result) + if (test_result.status not in [SelftestStatus.PASSED, + SelftestStatus.NO_RUN, + SelftestStatus.SKIPPED]): + ret = 1 return ret From patchwork Tue Mar 31 19:41:59 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218258 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=aX0z9yh5; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=rZ/M8s4c; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldn06vcVz1yCp for ; Wed, 01 Apr 2026 06:42:28 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=MBsW3Dp0QIVeCccLJZQ2oe/Uzf3avP0QdYpC50oO9qw=; b=aX0z9yh58nwlM2QxzZiCH6c1Px UlhG216AeZSToONz4XC8v2s/RyEySdFIx1pWbcv0XrjMDq19ZTtjFM3izQnIFnCQaVjWdkgaB05/H RPzvR/1jOIbdRk2qxOYPa6cBMsjzDMF8XoIQT9c6isoqXxKirCZUxEeLHmFpf3obsDf8850bVOYvq BL4ReMFJre42SQ5LCp/umEdL8n0EN+kdOzq0CmhinMFpd+r1D6x2J0BW471anwQGkTiRPEhcy6dSd 7m7ZYlmkZFSEpaKvYaO6NOP26khUtynbKuXUKek0gN5PNyrRfTO4OCPhtqUp6BmIiTsXBbWkXANoU hYCUIGNg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyp-0000000DUSV-1lmE; Tue, 31 Mar 2026 19:42:27 +0000 Received: from mail-pj1-x104a.google.com ([2607:f8b0:4864:20::104a]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eym-0000000DUPr-3fGV for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:26 +0000 Received: by mail-pj1-x104a.google.com with SMTP id 98e67ed59e1d1-35c0cbe0f64so17569357a91.0 for ; Tue, 31 Mar 2026 12:42:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986144; x=1775590944; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=SmS1uHrhcJAkZ/77nv6KxNChDlZ5TAlHKr2qEzAIUF0=; b=rZ/M8s4cw/r1itpg6KkUZzVQbk64QLvYxDY595vFo04Tb76+tQY+a1eCThrCYce7Pz 6e7Vb8t+uHedqFOcuLiQfmFBX9fMx0oAUQKfipolSVvQVSn9LO8ix931s51Ooq/CFFpn ND9kRswLtB+0fY7UEEJOGh/SJy9a9e9vzeHBHdXmTeD1tBO2rcnE2n+KsxlsYkV340Tr CjiQ819ycqZJm6tgEyXGNQoB8D+ZJw6wiYw3oKttPjCivhBX42JulSUqyc9HDB1NCSAk PKIRbfe9awPh+C0goYbxO5CviT+B6P6qTPE/ai1FoklrrTrVu5/X5NxoqSQ/ciTkq5tl kNWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986144; x=1775590944; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=SmS1uHrhcJAkZ/77nv6KxNChDlZ5TAlHKr2qEzAIUF0=; b=e+rlPMPmWyOLQS3TaMXTAAagGRvoMY9YPK9n+LIGE1qU/I/vaLRnjxV+LTKI/OnkJ2 Xjc7M+JU0ZUJ+OoKbzjd4m8gSJZnKe+1jN9GubtTdv8ZS9Qn4E5qjHQ0i4TjgAVSwPP0 B7Rbd3Xlu8PHeo6g+iJfr4Jyb8zottpm8LZBbUGz5E55RQjPputFXW44iFQJijeKrwta IArVNUcqvz74SIMXKnv5lfCl1KZv+AausK/zdKxfJ4KpiErY+k1B2X+HqvCaGZeqhNGV yQ2axJsWeFIsaWU7+2NmFGgFTUDiNF9nj2aFcKXLxtoQNovHTLZ/m2t40Y1wqLAiUTUL rQYg== X-Forwarded-Encrypted: i=1; AJvYcCU605Vip9Q7pg6cKioWUeFSWznDRU8+0dqIT6FjsojLU3GsKOoD5w7du/LAxeNDIXw5LLF36dvA0DU=@lists.infradead.org X-Gm-Message-State: AOJu0Yx3yj9NqfFJ+wJgeSZgRFo16zbEbA6zNsaTKtOrLUrlEKKDD5mU MLxNhSSvs+YntxRCT19JnJit1IJgZciGxGCFbQ9t9u7OzoD+ZC9gPBjPtCRENmN2MXKSpoRVWnJ eh8hTUQ75Pw== X-Received: from pjbng13.prod.google.com ([2002:a17:90b:1a8d:b0:35c:23cf:b494]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2ec4:b0:34c:904a:d92 with SMTP id 98e67ed59e1d1-35dc6fdd803mr478428a91.26.1774986143568; Tue, 31 Mar 2026 12:42:23 -0700 (PDT) Date: Tue, 31 Mar 2026 12:41:59 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-7-vipinsh@google.com> Subject: [PATCH v4 6/9] KVM: selftests: Add various print flags to KVM selftest runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124224_922439_0DFFE2C0 X-CRM114-Status: GOOD ( 10.22 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Add various print flags to selectively print outputs on terminal based on test execution status (passed, failed, timed out, skipped, no run). For each status provide further options (off, full, stderr, stdout, status) to choose verbosity of their prints. Make "full" the default choice. Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:104a listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Add various print flags to selectively print outputs on terminal based on test execution status (passed, failed, timed out, skipped, no run). For each status provide further options (off, full, stderr, stdout, status) to choose verbosity of their prints. Make "full" the default choice. Example: To print stderr for the failed tests and only status for the passed test: python3 runner --test-dirs tests --print-failed stderr \ --print-passed status Above command with disable print off skipped, timed out, and no run tests. Signed-off-by: Vipin Sharma --- .../testing/selftests/kvm/runner/__main__.py | 45 +++++++++++++++++++ .../selftests/kvm/runner/test_runner.py | 19 ++++++-- 2 files changed, 60 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/kvm/runner/__main__.py b/tools/testing/selftests/kvm/runner/__main__.py index 96402c89aea9..13f4cbea1fa7 100644 --- a/tools/testing/selftests/kvm/runner/__main__.py +++ b/tools/testing/selftests/kvm/runner/__main__.py @@ -9,6 +9,7 @@ import os import sys import datetime import pathlib +import textwrap from test_runner import TestRunner from selftest import SelftestStatus @@ -60,6 +61,50 @@ def cli(): type=int, help="Maximum number of tests that can be run concurrently. (Default: 1)") + status_choices = ["off", "full", "stdout", "stderr", "status"] + status_help_text = textwrap.dedent('''\ + Control output of the {} test (default is full). + off : dont print anything. + full : print stdout, stderr, and status of the test. + stdout: print stdout and status of the test. + stderr: print stderr and status of the test. + status: only print the status of test execution and no other output.'''); + + parser.add_argument("--print-passed", + default="full", + const="full", + nargs='?', + choices=status_choices, + help = status_help_text.format("passed")) + + parser.add_argument("--print-failed", + default="full", + const="full", + nargs='?', + choices=status_choices, + help = status_help_text.format("failed")) + + parser.add_argument("--print-skipped", + default="full", + const="full", + nargs='?', + choices=status_choices, + help = status_help_text.format("skipped")) + + parser.add_argument("--print-timed-out", + default="full", + const="full", + nargs='?', + choices=status_choices, + help = status_help_text.format("timed-out")) + + parser.add_argument("--print-no-run", + default="full", + const="full", + nargs='?', + choices=status_choices, + help = status_help_text.format("no-run")) + return parser.parse_args() diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py index 92eec18fe5c6..e8e8fd91c1ad 100644 --- a/tools/testing/selftests/kvm/runner/test_runner.py +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -17,6 +17,13 @@ class TestRunner: self.tests = [] self.output_dir = args.output self.jobs = args.jobs + self.print_stds = { + SelftestStatus.PASSED: args.print_passed, + SelftestStatus.FAILED: args.print_failed, + SelftestStatus.SKIPPED: args.print_skipped, + SelftestStatus.TIMED_OUT: args.print_timed_out, + SelftestStatus.NO_RUN: args.print_no_run + } for testcase in testcases: self.tests.append(Selftest(testcase, args.path, args.timeout, @@ -27,10 +34,14 @@ class TestRunner: return test def _log_result(self, test_result): - logger.info("*** stdout ***\n" + test_result.stdout) - logger.info("*** stderr ***\n" + test_result.stderr) - logger.log(test_result.status, - f"[{test_result.status.name}] {test_result.test_path}") + print_level = self.print_stds.get(test_result.status, "full") + + if (print_level == "full" or print_level == "stdout"): + logger.info("*** stdout ***\n" + test_result.stdout) + if (print_level == "full" or print_level == "stderr"): + logger.info("*** stderr ***\n" + test_result.stderr) + if (print_level != "off"): + logger.log(test_result.status, f"[{test_result.status.name}] {test_result.test_path}") def start(self): ret = 0 From patchwork Tue Mar 31 19:42:00 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218259 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=JuCsAW3d; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=uZm3GbkG; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldn31gtZz1yCp for ; Wed, 01 Apr 2026 06:42:31 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=J9BqIiVY+NeaOjcWMKzG5oTixTxMLbzZXDC8nsGO2xQ=; b=JuCsAW3dMwJ1AD1tKEeLyH2zUI ovdCPcznNCncxBGYX2drUgl3zH4wYomxVyoQijcS9hs3ULZd+7VLYtmbXYZTjLpVWu5j0IWxMRZKL MUQwLNe16s5prG4JqdFSDe7w68IEzYV4amv4Wow9ZEMfRDKFpfg/q6ejXNkWtTHVqB7SL4Dp/w5j2 d5DjswYCNDa8ej5ZIR5NUs0tHZikZFUi8EzXXyHnsZl6P+WXLlRXsTIWCsNfDaode1efbA1kvmKox SNZPw/fuwXduk7p70/Qb2SHJ6xNZrO0QzChqTVmouzsOBbydkHSF5BwiaxFRRj/3PfrZKcdIQgohm kxONzc/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyr-0000000DUU7-2oYk; Tue, 31 Mar 2026 19:42:29 +0000 Received: from mail-pf1-x449.google.com ([2607:f8b0:4864:20::449]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyo-0000000DURQ-3gSG for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:28 +0000 Received: by mail-pf1-x449.google.com with SMTP id d2e1a72fcca58-82a88a2704fso106991b3a.0 for ; Tue, 31 Mar 2026 12:42:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986145; x=1775590945; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EDxS3RLGSrG2H/pkS7vidJMr8Di538wm6I+VqwzPhms=; b=uZm3GbkG3QrOu4yQAS1lwze0IiGdsEUPytuJ0iMwscyAgNyTr5GzADZKS1bNSM4npH 4FtRIe9EUCuH5+y3BqjfOUfi+cb1Zvlo28nbMa7tomAVQkAiOWzD1vQm3QEnIMPyOK+g Vex4MQ7ImEv8XC1zfvUoOn7FzPvsfKdfx8dej8hnZovVTswvXLZS+G3e3AENrI7Utl7d gajwrvU4ZCQLr4iAE72vcOui7zhD98PinZ75xgQGlWMpE/h2H5yEHftge5JT7hypmrRe OC5bBXd090gCVp0U4EHTB//62NbsxNNAZkE1SjftQxfuBK7n0egL46/MFHmbcylNT5XZ HHtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986145; x=1775590945; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EDxS3RLGSrG2H/pkS7vidJMr8Di538wm6I+VqwzPhms=; b=r/hZULERu6UZSGyRqNlfbmB98azwRzUp9ayrVv8g1pDHp7gPjra1l+g84D++J7kH+K ojOGFUwlClfHKePU3onlmtPOGrPmaXm86XuEOlsb/Z712iRZSbVYJ9dZ5Nyinm0hm7tT 29rpmuyZmBUN99BHFGvOn05rced2jdDCBssFHGiGz1ZcUIsHVOyWpH3lMsCz4NPcD5md +hLcoIqOWn/t0u3E8ISEuxPsYLpY35Ma29L3Chbc9JLbkSBF7+T3TbDUO6nUmk7+RT3X kNlN4Wxsglmb3e5SftR5Oc3Ith30lfN5/M+oOf9LBGU8S4BgeYCsOHCPyjSUx5db77fa EGbg== X-Forwarded-Encrypted: i=1; AJvYcCVhDk3zyMMixOt61BcjXDqyRkWOpkfNGs6S/lND/naJEKXC26t1qOVU7fUjS/3cTuLrLdvWroUPN4o=@lists.infradead.org X-Gm-Message-State: AOJu0YznqrlHRkLyiPo1szo9k6N68frSucQybEOOF3zMNetdSGtFEQ7N ya4KWbwmcXsOgkoHNG172eHRqfzdhbhCDO0RglnTDR/NBlgGmVkKpg4C4hsa7/ZNwNF+6lKV458 Y+oqu37ckSg== X-Received: from pfud21.prod.google.com ([2002:a05:6a00:10d5:b0:82a:1d94:6f65]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:1ca8:b0:82a:15a1:14d with SMTP id d2e1a72fcca58-82ceaa5635emr441207b3a.14.1774986145378; Tue, 31 Mar 2026 12:42:25 -0700 (PDT) Date: Tue, 31 Mar 2026 12:42:00 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-8-vipinsh@google.com> Subject: [PATCH v4 7/9] KVM: selftests: Print sticky KVM selftests runner status at bottom From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124227_032599_BD712DA5 X-CRM114-Status: UNSURE ( 9.32 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Print current state of the KVM selftest runner during its execution. Show it as the bottom most line, make it sticky and colored. Provide the following information: - Total number of tests selected fo [...] Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:449 listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Print current state of the KVM selftest runner during its execution. Show it as the bottom most line, make it sticky and colored. Provide the following information: - Total number of tests selected for run. - How many have executed. - Total for each end state. Example: Total: 3/3 Passed: 1 Failed: 1 Skipped: 0 Timed Out: 0 No Run: 1 Signed-off-by: Vipin Sharma --- .../selftests/kvm/runner/test_runner.py | 21 +++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/tools/testing/selftests/kvm/runner/test_runner.py b/tools/testing/selftests/kvm/runner/test_runner.py index e8e8fd91c1ad..66a3c3711a8a 100644 --- a/tools/testing/selftests/kvm/runner/test_runner.py +++ b/tools/testing/selftests/kvm/runner/test_runner.py @@ -15,6 +15,7 @@ logger = logging.getLogger("runner") class TestRunner: def __init__(self, testcases, args): self.tests = [] + self.status = {x: 0 for x in SelftestStatus} self.output_dir = args.output self.jobs = args.jobs self.print_stds = { @@ -33,9 +34,19 @@ class TestRunner: test.run() return test + def _sticky_update(self): + print(f"\r\033[1mTotal: {self.tests_ran}/{len(self.tests)}" \ + f"\033[32;1m Passed: {self.status[SelftestStatus.PASSED]}" \ + f"\033[31;1m Failed: {self.status[SelftestStatus.FAILED]}" \ + f"\033[33;1m Skipped: {self.status[SelftestStatus.SKIPPED]}"\ + f"\033[91;1m Timed Out: {self.status[SelftestStatus.TIMED_OUT]}"\ + f"\033[34;1m No Run: {self.status[SelftestStatus.NO_RUN]}\033[0m", + end="\r", flush=True) + def _log_result(self, test_result): print_level = self.print_stds.get(test_result.status, "full") + print("\033[2K", end="\r", flush=True) if (print_level == "full" or print_level == "stdout"): logger.info("*** stdout ***\n" + test_result.stdout) if (print_level == "full" or print_level == "stderr"): @@ -43,8 +54,16 @@ class TestRunner: if (print_level != "off"): logger.log(test_result.status, f"[{test_result.status.name}] {test_result.test_path}") + self.status[test_result.status] += 1 + # Sticky bottom line + self._sticky_update() + def start(self): ret = 0 + self.tests_ran = 0 + + # Show initial progress + self._sticky_update() with concurrent.futures.ProcessPoolExecutor(max_workers=self.jobs) as executor: all_futures = [] @@ -54,9 +73,11 @@ class TestRunner: for future in concurrent.futures.as_completed(all_futures): test_result = future.result() + self.tests_ran += 1 self._log_result(test_result) if (test_result.status not in [SelftestStatus.PASSED, SelftestStatus.NO_RUN, SelftestStatus.SKIPPED]): ret = 1 + print("\n") return ret From patchwork Tue Mar 31 19:42:01 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218260 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=cHARjlSG; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=gzO5z9K9; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldn43Lppz1yCp for ; Wed, 01 Apr 2026 06:42:32 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=4bfBpegrOc6cbcwEOTyxI8D9TVw8cpb3vaobTYrj6PI=; b=cHARjlSG3aoaxEifbMWOf2v5Dt zMCJXYlne1bCgj0bSkNlz0stnQ3g/Vf6KyCE++lSGGEP462n3Cbqw5Tt9BZA9Qp4CHzS3HQjcNnqI uWAsQrpbQp6Fwxd+bLZN5lQvPxBjkxe2RGVoB8Lp2CGo9eQdAHjcwp9sNMyw7pr+aLdEtNGxtvJ+j UXKiNewdfNoOvTXMO+PKPq8E96UC+j8rHxKy8rnqBPj2iHxdqxMgZc63ZQq7BJBnRBluUaBDp/Sz4 0ml0jLHyCdCoLQAWnZAkne41QkjZ6Q977QlIvfOYRD6Osb1sIdM5OHLHUh6f5o9sC2lT8sBJS1uq/ 2J715HYg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eys-0000000DUVE-3vso; Tue, 31 Mar 2026 19:42:30 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyq-0000000DUSv-3Nrm for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:29 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-35c1874336aso17493852a91.3 for ; Tue, 31 Mar 2026 12:42:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986147; x=1775590947; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=EMAEEYc33pbDyh5jLNn4+2ckaUXnuUr3hoaO2wyMBMY=; b=gzO5z9K9ITmLRK3JYOrqK7cUhfk8Q/+j4IJzmPD9oD5m+Cp8DutM6TosHkdJRBk8Ji s8CKjDXmRUT2RmWCpRJfBvew9LbaHg33J8igP2belO8nOxg6mSeV8iQxpwFfEaN+iAoD 3JEdsNGSqSi0gFXPgW4wnbrmJ+xaDX9tDUem2BRNbM44GK8XrsWWcHAgIHW/Wn3ZmL19 JzcaOtT0ov7mjACdFLZl11bO2qZjjb1zShMT8OjkdoIz9RGVh43K3hvPjEAL7PWoGHZh m0N8BMbmtUIUQF7AHP53wZbFmbiB1gbVsvmOOgOB+Dixyd6yTqvaB+hKZ+RRiPUV/1L6 JVjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986147; x=1775590947; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EMAEEYc33pbDyh5jLNn4+2ckaUXnuUr3hoaO2wyMBMY=; b=FS4G6zYoIU1AgVeMLHVwbX6sKG0tWTCbsgI4HJioGXA4hfccx1Q1pK0lZ5NvkHQ0GZ dq5NMEsP/Mjb4y6qqeHSNC7iiKKHXet+5oV35ejq61yzAoP56e6lqmIOy6rh5+SOYMD7 hmDZzXtyR00I8T9TKHGolrZDbh6FqcGU64bNKM4MIlqbY0uEAS3LqV/LLfZReZLur6oi ABcwybVCdcx50EB3aL/T9bf/416SdaBKlKN3Gal5hSmCsmLRXIV6VrEUaAfSTHGmCQfy Dpojuq/N3RKqxplp7rpfFMBR6iLXfI2xMfhS+O9qlgEfoRHalGwbXLHxLtL2pRUODfCl e8Jg== X-Forwarded-Encrypted: i=1; AJvYcCV6/vDTyxdVaPYvylXlqwbCDiSVL1n/TUVdU6urgEJcKIquMv6uNXuPpV+d+JI8IwmOJtSdmqbEnUE=@lists.infradead.org X-Gm-Message-State: AOJu0YzOTwwlf2ZZUpOSDje59Fwub+kWkgBZIL9iOXK77Juk/PC6ZDEe 3ys8T6Lg9lOA1h4FUSuW3l2phLdKjdiP9Y6E2ixw35uN4/ihJDn6hyGvYDoM50pS2ex39mNs4gj LBQWMXCf6IA== X-Received: from pjpe5.prod.google.com ([2002:a17:90a:9a85:b0:35d:9d42:3153]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1850:b0:359:ff8a:ee4f with SMTP id 98e67ed59e1d1-35dc6e9bc88mr571073a91.7.1774986147217; Tue, 31 Mar 2026 12:42:27 -0700 (PDT) Date: Tue, 31 Mar 2026 12:42:01 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-9-vipinsh@google.com> Subject: [PATCH v4 8/9] KVM: selftests: Add rule to generate default tests for KVM selftests runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124228_846700_923CAF3F X-CRM114-Status: GOOD ( 10.93 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Add 'tests_install' rule in the Makefile.kvm to auto generate default testcases for KVM selftests runner. Preserve the hierarchy of test executables for autogenerated files. Remove these testcases on [...] Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1049 listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Add 'tests_install' rule in the Makefile.kvm to auto generate default testcases for KVM selftests runner. Preserve the hierarchy of test executables for autogenerated files. Remove these testcases on invocation of 'make clean'. Autogeneration of default test files allows runner to execute default testcases easily. These default testcases don't need to be checked in as they are just executing the test without any command line options. Signed-off-by: Vipin Sharma --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/Makefile.kvm | 26 +++++++++++++++++++++++- 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index 95af97b1ff9e..548d435bde2f 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -7,6 +7,7 @@ !*.S !*.sh !*.test +default.test !.gitignore !config !settings diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm index 6471fa214a9f..fb9439cb5f3d 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only include ../../../build/Build.include -all: +all: tests_install LIBKVM += lib/assert.c LIBKVM += lib/elf.c @@ -330,11 +330,15 @@ $(SPLIT_TEST_GEN_PROGS): $(OUTPUT)/%: $(OUTPUT)/%.o $(OUTPUT)/$(ARCH)/%.o $(SPLIT_TEST_GEN_OBJ): $(OUTPUT)/$(ARCH)/%.o: $(ARCH)/%.c $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -c $< -o $@ +# Default testcases for KVM selftests runner will be generated in this directory. +DEFAULT_TESTCASES = testcases_default_gen + EXTRA_CLEAN += $(GEN_HDRS) \ $(LIBKVM_OBJS) \ $(SPLIT_TEST_GEN_OBJ) \ $(TEST_DEP_FILES) \ $(TEST_GEN_OBJ) \ + $(OUTPUT)/$(DEFAULT_TESTCASES) \ cscope.* $(LIBKVM_C_OBJ): $(OUTPUT)/%.o: %.c $(GEN_HDRS) @@ -363,3 +367,23 @@ cscope: find . -name '*.c' \ -exec realpath --relative-base=$(PWD) {} \;) | sort -u > cscope.files cscope -b + +# Generate runner testcases in DEFAULT_TESTCASES directory. +# $(OUTPUT) is either CWD or specified in the make command. +tests_install: list_progs = $(patsubst $(OUTPUT)/%,%,$(TEST_GEN_PROGS)) +tests_install: + $(foreach tc, $(TEST_PROGS), \ + $(shell mkdir -p $(OUTPUT)/$(DEFAULT_TESTCASES)/$(patsubst %.sh,%,$(tc)))) + $(foreach tc, $(TEST_PROGS), \ + $(shell echo $(tc) > $(patsubst %.sh,$(OUTPUT)/$(DEFAULT_TESTCASES)/%/default.test,$(tc)))) + + $(foreach tc, $(list_progs), \ + $(shell mkdir -p $(OUTPUT)/$(DEFAULT_TESTCASES)/$(tc))) + $(foreach tc, $(list_progs), \ + $(shell echo $(tc) > $(patsubst %,$(OUTPUT)/$(DEFAULT_TESTCASES)/%/default.test,$(tc)))) + + @if [ ! -d $(OUTPUT)/runner ]; then \ + cp -r $(selfdir)/kvm/runner $(OUTPUT); \ + fi + + @: From patchwork Tue Mar 31 19:42:02 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vipin Sharma X-Patchwork-Id: 2218261 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=kxWgIAP4; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20251104 header.b=Dlr76GlW; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4fldn55ChPz1yCp for ; Wed, 01 Apr 2026 06:42:33 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Message-ID: References:Mime-Version:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=SGlaVB4oAo7dAktCVacBA2SiDnegAUgUHaHuLvhTH0g=; b=kxWgIAP4Cr8au8veNGpYKcusNh A/s3SNXeSS887LkuYF+rFUBclXKf5T0Zaje7nGfAPgJwIJjK9dPJ8oVGo4eynZzfPIcqpWzAObuD6 Y671NSk5nzZPP70Bv0rdnazg8rTzY3wyt0fTecOxZ13CB0VxsPTtWzsXPA09wdWOUt2zNxoaPpx9V XYv3H5tYQXid5NV3qDnRJKVvuRiPcZxDc5DVMmtcN6f9ghard8l0IuctfYWik7XqCq45dvWLQnnQZ LDZfe7piSHUqHcho1ZTLBaoMNcq2XVKSNyvnZDz6djsgYecd2jt4V+QVh6LtwL6Xxi7RDYkmLJUoz jD64ajMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyu-0000000DUWD-0l80; Tue, 31 Mar 2026 19:42:32 +0000 Received: from mail-pj1-x1049.google.com ([2607:f8b0:4864:20::1049]) by bombadil.infradead.org with esmtps (Exim 4.98.2 #2 (Red Hat Linux)) id 1w7eyr-0000000DUTz-434L for kvm-riscv@lists.infradead.org; Tue, 31 Mar 2026 19:42:31 +0000 Received: by mail-pj1-x1049.google.com with SMTP id 98e67ed59e1d1-35dacc40f92so2441437a91.2 for ; Tue, 31 Mar 2026 12:42:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774986149; x=1775590949; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=4VWnO9ZqBqweDz2fb12798lX7BIjzp/risNDY0rLawk=; b=Dlr76GlWnhxf6RuwFAL+0HDjrpGazbgfVGrrJ3uwy7pDBSv3jecs5LFltHTHE9SMHq qwXLQofMduUoH6YHSx/FIIcp8dUdEFsmUzMurJ53X6cjO1rtZBayY2UnoBspmBDwQ1H+ M4pJyREfKKx/HoScIu08tnTy/+cnMyYuiDXp5BCmIlYw9rh+/ZjLzLW7w8aGwIeI6o57 2nO56CgoxW7NSt25EAJ0f07/y1JIWfysYMDyJTlkYvU1+SgmfK8T4eaNtLlVpWX100qg wJXPTWxK3h/kFriIf88Snic8UbiOE4qSfoMgcukt/byOy8jQO7x/kQhOKN720Y+m7i/i wR5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774986149; x=1775590949; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=4VWnO9ZqBqweDz2fb12798lX7BIjzp/risNDY0rLawk=; b=HENRuS/jjukvgbwGstMZGQ3y+xCRorNLR5GMmL/luC0Q6XgcJ+kVEoqXtMTesf0vM4 7q65iZDMb7FaM1O0amN9/Tfp4dJUJDh3X5vg073bK8C3xNFmbRk5Q/3iT6f8KAvmJ4Oa RpihMpIOwnZvXWSPgIf30ds8ftOr2Kvf3aN2V+IFws+kUmu7htrvkqWNH7PnbAW95GOg ao4kahfYG4NJ8RVx1Puz2GOmrQgROgkw523akSaFiOyCs4VcFRy5+nRkXaXjSpelQGU3 VHT3xYwatfaRL0tj9YzdLu9kfYW+3ettOF+vjz2w8ZPu3f1zUA6cA2qQGBtJDGitAN8M LeOQ== X-Forwarded-Encrypted: i=1; AJvYcCUHgZwYc+9Ey/r74xoJBwqSELVFekS6++zuJSb6a5T9mesOTemvSWu8eFlPCzVSIAM3rfucoUqwe6E=@lists.infradead.org X-Gm-Message-State: AOJu0Yy8Dpb+vVHgx38p0q6gRDoVsemJTJeL5ZFtv74kHGEx2EAlozUR TbfVMkZH7vO40Pv05vzFXQc8p/+bOLtcua7AFewBnNJl6GsRefoJ3yvvw89+dziVokWuLHD+IHT L3mfrC9S1Cw== X-Received: from pjyw24.prod.google.com ([2002:a17:90a:ea18:b0:35d:a917:a7bf]) (user=vipinsh job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:528c:b0:35d:a3b4:2f0d with SMTP id 98e67ed59e1d1-35dc6e77eb0mr554628a91.6.1774986148828; Tue, 31 Mar 2026 12:42:28 -0700 (PDT) Date: Tue, 31 Mar 2026 12:42:02 -0700 In-Reply-To: <20260331194202.1722082-1-vipinsh@google.com> Mime-Version: 1.0 References: <20260331194202.1722082-1-vipinsh@google.com> X-Mailer: git-send-email 2.53.0.1118.gaef5881109-goog Message-ID: <20260331194202.1722082-10-vipinsh@google.com> Subject: [PATCH v4 9/9] KVM: selftests: Provide README.rst for KVM selftests runner From: Vipin Sharma To: kvm@vger.kernel.org, kvmarm@lists.linux.dev, kvm-riscv@lists.infradead.org Cc: seanjc@google.com, pbonzini@redhat.com, borntraeger@linux.ibm.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, anup@brainfault.org, atish.patra@linux.dev, zhaotianrui@loongson.cn, maobibo@loongson.cn, chenhuacai@kernel.org, maz@kernel.org, oliver.upton@linux.dev, ajones@ventanamicro.com, Vipin Sharma X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260331_124230_010292_07356739 X-CRM114-Status: GOOD ( 16.32 ) X-Spam-Score: -9.5 (---------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Add README.rst for KVM selftest runner and explain how to use the runner. Signed-off-by: Vipin Sharma --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/runner/README.rst | 95 +++++++++++++++++++ 2 files changed, 96 insertions( [...] Content analysis details: (-9.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1049 listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM welcome-list 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org Add README.rst for KVM selftest runner and explain how to use the runner. Signed-off-by: Vipin Sharma --- tools/testing/selftests/kvm/.gitignore | 1 + tools/testing/selftests/kvm/runner/README.rst | 95 +++++++++++++++++++ 2 files changed, 96 insertions(+) create mode 100644 tools/testing/selftests/kvm/runner/README.rst diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore index 548d435bde2f..83aa2fe01bac 100644 --- a/tools/testing/selftests/kvm/.gitignore +++ b/tools/testing/selftests/kvm/.gitignore @@ -4,6 +4,7 @@ !*.c !*.h !*.py +!*.rst !*.S !*.sh !*.test diff --git a/tools/testing/selftests/kvm/runner/README.rst b/tools/testing/selftests/kvm/runner/README.rst new file mode 100644 index 000000000000..e264c78ace68 --- /dev/null +++ b/tools/testing/selftests/kvm/runner/README.rst @@ -0,0 +1,95 @@ +KVM Selftest Runner +=================== + +Execute KVM selftests with high configurability. The runner supports parallel +execution, hierarchical output storage, console output control, and execution +status reporting. + +Compatibility with KVM Selftests +=============================== + +The runner acts as a non-intrusive wrapper around the existing KVM selftests. +It does not modify the underlying test binaries or the way they are built. +You can still run KVM selftests using traditional methods without any conflict: + +- Execute binaries directly: ``./dirty_log_perf_test -v 4`` +- Use the standard kselftest framework: ``make -C tools/testing/selftests TARGETS=kvm run_tests`` + +Comparison with kselftest framework +=================================== + +While the standard ``kselftest`` framework provides basic execution and reporting, +this runner is designed for more advanced testing scenarios and offers several +advantages: + +- **Flexible Test Selection**: Unlike some ``kselftest`` options that may use a + fixed ``suite:test`` syntax for selection, this runner leverages the + filesystem. You can group tests into directories (acting as suites) or + provide a list of specific test files, offering a more intuitive way to + organize and execute subsets of tests. +- **Parallel Execution**: Run tests concurrently using the ``-j`` flag to + significantly reduce total execution time. +- **Granular Output Control**: Control exactly what gets printed to the console + for passed, failed, skipped, or timed-out tests. +- **Hierarchical Logging**: Automatically save the stdout and stderr of every + test into a structured directory tree, making it easy to debug failures in + large test runs. +- **Out-of-Tree Support**: Easily run test binaries located in a different + directory using the ``-p`` flag. + +Generate Default Tests +====================== + +Generate the default test case files using the provided make target:: + + # make tests_install + +This creates the ``testcases_default_gen`` directory containing ``default.test`` +files. Each KVM selftest has a directory with a ``default.test`` file. This file +contains the executable path relative to the KVM selftest root directory +(``/tools/testing/selftests/kvm``). For example, the ``dirty_log_perf_test`` +entry looks like:: + + # cat testcases_default_gen/dirty_log_perf_test/default.test + dirty_log_perf_test + +In above testcase, the runner executes ``dirty_log_perf_test``. Testcase files +can provide extra arguments to the test:: + + # cat tests/dirty_log_perf_test/2slot_5vcpu_10iter.test + dirty_log_perf_test -x 2 -v 5 -i 10 + +In this case, the runner executes ``dirty_log_perf_test`` with the specified +options. + +Examples +======== + +Display all options:: + + # python3 runner -h + +Run all default tests:: + + # python3 runner -d testcases_default_gen + +Run tests in parallel:: + + # python3 runner -d testcases_default_gen -j 40 + +Print only passed test status and failed test stderr:: + + # python3 runner -d testcases_default_gen --print-passed status \ + --print-failed stderr + +Run test binaries from a different directory (e.g., out-of-tree builds):: + + # python3 runner -d testcases_default_gen -p /path/to/binaries + +Save all test outputs (stdout, stderr, status) to a directory:: + + # python3 runner -d testcases_default_gen -o test_outputs + +Run specific testcase files:: + + # python3 runner -t tests/dirty_log_perf_test/2slot_5vcpu_10iter.test