From patchwork Wed Nov 28 19:36:30 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Higgins X-Patchwork-Id: 1004803 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=linux-um-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="a25eV2mq"; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="HZFAhEFp"; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 434rYV12h7z9s9G for ; Thu, 29 Nov 2018 06:38:38 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X7r5+lJxvtz7gI+gejxbjcW1bls8j/voTtv4bYu9zSA=; b=a25eV2mqKsN4YK 0K3r9NDRDd0Ay2/wzRPHjDrUgg9sbWre8mnVSE8MCOikZFcZZh2Eikr8iV5cJR7WynWSLBRJSXnTP MT9ZKItjtG6UEon5IQ/hRpvTAFMxBypbGYjvfTd4boZodhP9p3DwTh8MvS32ef51NthSRJPvPYc3b US9ltai8nmYM1s+N+phG399QYBKxKeMkqOtyo/AJe+O/Y2L6zyeNqpvpLWHx36Q5Ytj64hd34nyKn gFU3lZOWlZrUPHUidjiZl3kByRLVc0mt7tf/d5ErAqPUTf/iPUfx26Aki1zk/yVEzqE+5GQi8dvMt BHxBEh6mE+42H9eZZGQw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gS5fK-0003ol-Nk; Wed, 28 Nov 2018 19:38:30 +0000 Received: from mail-it1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gS5fH-0003gK-BP for linux-um@lists.infradead.org; Wed, 28 Nov 2018 19:38:29 +0000 Received: by mail-it1-x149.google.com with SMTP id d1-v6so4291265itj.8 for ; Wed, 28 Nov 2018 11:38:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=vAK0p7vKTuMPpezuXE/k7QGLFOufe8xVPHMWIxnzcOw=; b=HZFAhEFpsemMuzBPrmsGrjv52lptKO71UvLKo8R2cqvpqs1fsoXvzEW9M82vVbiG4I 8lgm+sQr+xKpgGcI7n57+Iqp4EL/gw1tAT1r5AzBwEMUCY6uDx4HumzI5JusEDB9i0gT 1aXleqwgLn8rIP+5OiJ8PMWwF9NrssiZgN5lGGKq+tgV+qhjf0N0IRW6EXo5QmczSDFv sfTS7DnEtq0yWfymSBjFNaRT2AnoJ0F1kMTq6CJvrMF6HNzvjc4JFjUQ1ZL1V67M+DbU KMzcuowcCU7kdr/QxmqyuYiHmsdeq/7YReCEQcungg8JXo/tErb0d4cugphcf/gfr7a4 3+YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=vAK0p7vKTuMPpezuXE/k7QGLFOufe8xVPHMWIxnzcOw=; b=fMhTUkMwbPLHbfoIS1sz1wdPmcY2V7vW80RnBJjBnDT+ROqfyGMs03V5BUO9Vibck1 Ba7cNVYn9XLbazjzFK+aXh8qyESnzSt04irHmX5yK4zOXJsbU3NyZTi2A1MakBqeikpR X6vY7f/1xMHX2Mo7ASh32VM4skt1saPadM4Ge+3AtTpHcFStXSOKxY62+zXtrPz0jyp1 GXyme+YvPeqNY9sqEVYWirG2BKMniZD+qvst/skusHoUlgLVw/IzLgSwKNdWOOQtgH4D zgtYm6Zw0ODzbyeTBhMyR91nDE0c6wTj6WlDkAEUXKzVzIHZguNubCz7bdaQsvkQLjKW O7aA== X-Gm-Message-State: AA+aEWaTfgcyZG/u5Vzka27+BTcARx/tO8vT2CndvJTHAPik4DDAyH9d Qv8XDp2jjFvB476X14woOUHogP4ekTMU7iokUOHZhw== X-Google-Smtp-Source: AFSGD/Ujo8L+z++cqBiX0TR/cZSdTP71zFmLdi9OlccxU3BUppjqS2CtoJDToemuwTkNGpAvvuqMQL2o2zW3kWe3LvqXRA== X-Received: by 2002:a24:ac6c:: with SMTP id m44-v6mr3565488iti.40.1543433895704; Wed, 28 Nov 2018 11:38:15 -0800 (PST) Date: Wed, 28 Nov 2018 11:36:30 -0800 In-Reply-To: <20181128193636.254378-1-brendanhiggins@google.com> Message-Id: <20181128193636.254378-14-brendanhiggins@google.com> Mime-Version: 1.0 References: <20181128193636.254378-1-brendanhiggins@google.com> X-Mailer: git-send-email 2.20.0.rc0.387.gc7a69e6b6c-goog Subject: [RFC v3 13/19] kunit: improve output from python wrapper From: Brendan Higgins To: gregkh@linuxfoundation.org, keescook@google.com, mcgrof@kernel.org, shuah@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181128_113827_398299_4A9303E3 X-CRM114-Status: GOOD ( 10.19 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:149 listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.0 DKIMWL_WL_MED DKIMwl.org - Whitelisted Medium sender X-BeenThere: linux-um@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: brakmo@fb.com, jdike@addtoit.com, dri-devel@lists.freedesktop.org, linux-kselftest@vger.kernel.org, frowand.list@gmail.com, robh@kernel.org, linux-nvdimm@lists.01.org, richard@nod.at, knut.omang@oracle.com, kieran.bingham@ideasonboard.com, Felix Guo , joel@jms.id.au, khilman@baylibre.com, Brendan Higgins , Tim.Bird@sony.com, linux-um@lists.infradead.org, rostedt@goodmis.org, julia.lawall@lip6.fr, dan.j.williams@intel.com, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, daniel@ffwll.ch, mpe@ellerman.id.au, joe@perches.com Sender: "linux-um" Errors-To: linux-um-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org - add colors to displayed output - add timing and summary Signed-off-by: Felix Guo Signed-off-by: Brendan Higgins --- tools/testing/kunit/kunit.py | 27 ++++++++- tools/testing/kunit/kunit_parser.py | 93 ++++++++++++++++++++++++++++- 2 files changed, 115 insertions(+), 5 deletions(-) diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py index 1356be404996b..0b8e8c20a746e 100755 --- a/tools/testing/kunit/kunit.py +++ b/tools/testing/kunit/kunit.py @@ -6,6 +6,7 @@ import argparse import sys import os +import time import kunit_config import kunit_kernel @@ -21,20 +22,40 @@ parser.add_argument('--timeout', help='maximum number of seconds to allow for ' 'build the tests.', type=int, default=300, metavar='timeout') +parser.add_argument('--jobs', + help='As in the make command, "Specifies the number of ' + 'jobs (commands) to run simultaneously."', + type=int, default=8, metavar='jobs') + cli_args = parser.parse_args() linux = kunit_kernel.LinuxSourceTree() +config_start = time.time() success = linux.build_reconfig() +config_end = time.time() if not success: quit() -print('Building KUnit Kernel ...') -success = linux.build_um_kernel() +kunit_parser.print_with_timestamp('Building KUnit Kernel ...') + +build_start = time.time() +success = linux.build_um_kernel(jobs=cli_args.jobs) +build_end = time.time() if not success: quit() -print('Starting KUnit Kernel ...') +kunit_parser.print_with_timestamp('Starting KUnit Kernel ...') +test_start = time.time() + if cli_args.raw_output: kunit_parser.raw_output(linux.run_kernel(timeout=cli_args.timeout)) else: kunit_parser.parse_run_tests(linux.run_kernel(timeout=cli_args.timeout)) + +test_end = time.time() + +kunit_parser.print_with_timestamp(( + "Elapsed time: %.3fs total, %.3fs configuring, %.3fs " + + "building, %.3fs running.\n") % (test_end - config_start, + config_end - config_start, build_end - build_start, + test_end - test_start)) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 1dff3adb73bd3..d9051e407d5a7 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 import re +from datetime import datetime kunit_start_re = re.compile('console .* enabled') kunit_end_re = re.compile('List of all partitions:') @@ -19,6 +20,94 @@ def raw_output(kernel_output): for line in kernel_output: print(line) +DIVIDER = "=" * 30 + +RESET = '\033[0;0m' + +def red(text): + return '\033[1;31m' + text + RESET + +def yellow(text): + return '\033[1;33m' + text + RESET + +def green(text): + return '\033[1;32m' + text + RESET + +def print_with_timestamp(message): + print('[%s] %s' % (datetime.now().strftime('%H:%M:%S'), message)) + +def print_log(log): + for m in log: + print_with_timestamp(m) + def parse_run_tests(kernel_output): - for output in isolate_kunit_output(kernel_output): - print(output) + test_case_output = re.compile('^kunit .*?: (.*)$') + + test_module_success = re.compile('^kunit .*: all tests passed') + test_module_fail = re.compile('^kunit .*: one or more tests failed') + + test_case_success = re.compile('^kunit (.*): (.*) passed') + test_case_fail = re.compile('^kunit (.*): (.*) failed') + test_case_crash = re.compile('^kunit (.*): (.*) crashed') + + total_tests = set() + failed_tests = set() + crashed_tests = set() + + def get_test_name(match): + return match.group(1) + ":" + match.group(2) + + current_case_log = [] + def end_one_test(match, log): + log.clear() + total_tests.add(get_test_name(match)) + + print_with_timestamp(DIVIDER) + for line in isolate_kunit_output(kernel_output): + # Ignore module output: + if (test_module_success.match(line) or + test_module_fail.match(line)): + print_with_timestamp(DIVIDER) + continue + + match = re.match(test_case_success, line) + if match: + print_with_timestamp(green("[PASSED] ") + + get_test_name(match)) + end_one_test(match, current_case_log) + continue + + match = re.match(test_case_fail, line) + # Crashed tests will report as both failed and crashed. We only + # want to show and count it once. + if match and get_test_name(match) not in crashed_tests: + failed_tests.add(get_test_name(match)) + print_with_timestamp(red("[FAILED] " + + get_test_name(match))) + print_log(map(yellow, current_case_log)) + print_with_timestamp("") + end_one_test(match, current_case_log) + continue + + match = re.match(test_case_crash, line) + if match: + crashed_tests.add(get_test_name(match)) + print_with_timestamp(yellow("[CRASH] " + + get_test_name(match))) + print_log(current_case_log) + print_with_timestamp("") + end_one_test(match, current_case_log) + continue + + # Strip off the `kunit module-name:` prefix + match = re.match(test_case_output, line) + if match: + current_case_log.append(match.group(1)) + else: + current_case_log.append(line) + + fmt = green if (len(failed_tests) + len(crashed_tests) == 0) else red + print_with_timestamp( + fmt("Testing complete. %d tests run. %d failed. %d crashed." % + (len(total_tests), len(failed_tests), len(crashed_tests)))) +