From patchwork Thu Sep 7 22:07:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831152 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Zb6qb/ms; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYM66hZqz1yhG for ; Fri, 8 Sep 2023 08:09:54 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229568AbjIGWJy (ORCPT ); Thu, 7 Sep 2023 18:09:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232116AbjIGWJp (ORCPT ); Thu, 7 Sep 2023 18:09:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C4DE1BC7 for ; Thu, 7 Sep 2023 15:08:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124526; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iI3VAqFHNCcX6DtHVKxcNqkEC6+kea2I34uN1fS8kcM=; b=Zb6qb/msa8pOjsxuhQicvYXPA8Yqb9VvNcRYweSqrPxpc0z3EOJutrpN44sy5hOu772Qiv Lj5g1PGaSWDxlNjkl1lsoTxLstsMHaBXbIcMBxBEZM6djJCsvkdVY4O+mPlXHeSXXHgcOv aUpl8JKHNm1M5O3l5vf7hrg4OLRlIQg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-631-zw3UKMaVNiu-6FX-P9Rnrw-1; Thu, 07 Sep 2023 18:08:45 -0400 X-MC-Unique: zw3UKMaVNiu-6FX-P9Rnrw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 38643800888 for ; Thu, 7 Sep 2023 22:08:45 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 838537B62; Thu, 7 Sep 2023 22:08:44 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 01/11] tests/shell: cleanup result handling in "test-wrapper.sh" Date: Fri, 8 Sep 2023 00:07:13 +0200 Message-ID: <20230907220833.2435010-2-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org The previous code was mostly correct, but hard to understand. Rework it. Also, on failure now always write "rc-failed-exit", which is the exit code that "test-wrapper.sh" reports to "run-test.sh". Note that this error code may not be the same as the one returned by the TEST binary. The latter you can find in one of the files "rc-{ok,skipped,failed}". In general, you can search the directory with test results for those "rc-*" files. If you find a "rc-failed" file, it was counted as failure. There might be other "rc-failed-*" files, depending on whether the diff failed or kernel got tainted. Also, reserve all the error codes 118 - 124 for the "test-wrapper.sh". For example, 124 means a dump difference and 123 means kernel got tainted. In the future, 122 will mean a valgrind error. Other numbers are not reserved. If a test command fails with such an reserved code, "test-wrapper.sh" modifies it to 125, so that "run-test.sh" does not get the wrong idea about the failure reason. This is not new in this patch, except that the reserved range was extended for future additions. Signed-off-by: Thomas Haller --- tests/shell/helpers/test-wrapper.sh | 65 ++++++++++++++++++----------- 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh index 43b3aa09ef26..f8b27b1e9291 100755 --- a/tests/shell/helpers/test-wrapper.sh +++ b/tests/shell/helpers/test-wrapper.sh @@ -51,7 +51,6 @@ DUMPPATH="$TESTDIR/dumps" DUMPFILE="$DUMPPATH/$TESTBASE.nft" dump_written= -rc_dump= # The caller can request a re-geneating of the dumps, by setting # DUMPGEN=y. @@ -66,42 +65,60 @@ if [ "$rc_test" -eq 0 -a "$DUMPGEN" = y -a -d "$DUMPPATH" ] ; then cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "$DUMPFILE" fi +rc_dump=0 if [ "$rc_test" -ne 77 -a -f "$DUMPFILE" ] ; then - rc_dump=0 if [ "$dump_written" != y ] ; then - $DIFF -u "$DUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff" || rc_dump=$? - if [ "$rc_dump" -eq 0 ] ; then + if ! $DIFF -u "$DUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff" ; then + rc_dump=124 rm -f "$NFT_TEST_TESTTMPDIR/ruleset-diff" fi fi fi +if [ "$rc_dump" -ne 0 ] ; then + echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump" +fi +rc_tainted=0 if [ "$tainted_before" != "$tainted_after" ] ; then echo "$tainted_after" > "$NFT_TEST_TESTTMPDIR/rc-failed-tainted" + rc_tainted=123 fi -rc_exit="$rc_test" -if [ -n "$rc_dump" ] && [ "$rc_dump" -ne 0 ] ; then - echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump" - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed" - if [ "$rc_exit" -eq 0 ] ; then - # Special exit code to indicate dump diff. - rc_exit=124 - fi -elif [ "$rc_test" -eq 77 ] ; then - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-skipped" -elif [ "$rc_test" -eq 0 -a "$tainted_before" = "$tainted_after" ] ; then - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-ok" +if [ "$rc_tainted" -ne 0 ] ; then + rc_exit="$rc_tainted" +elif [ "$rc_test" -ge 118 -a "$rc_test" -le 124 ] ; then + # Special exit codes are reserved. Coerce them. + rc_exit="125" +elif [ "$rc_test" -ne 0 ] ; then + rc_exit="$rc_test" +elif [ "$rc_dump" -ne 0 ] ; then + rc_exit="$rc_dump" else - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed" - if [ "$rc_test" -eq 0 -a "$tainted_before" != "$tainted_after" ] ; then - # Special exit code to indicate tainted. - rc_exit=123 - elif [ "$rc_test" -eq 124 -o "$rc_test" -eq 123 ] ; then - # These exit codes are reserved - rc_exit=125 - fi + rc_exit="0" +fi + + +# We always write the real exit code of the test ($rc_test) to one of the files +# rc-{ok,skipped,failed}, depending on which it is. +# +# Note that there might be other rc-failed-{dump,tainted} files with additional +# errors. Note that if such files exist, the overall state will always be +# failed too (and an "rc-failed" file exists). +# +# On failure, we also write the combined "$rc_exit" code from "test-wrapper.sh" +# to "rc-failed-exit" file. +# +# This means, failed tests will have a "rc-failed" file, and additional +# "rc-failed-*" files exist for further information. +if [ "$rc_exit" -eq 0 ] ; then + RC_FILENAME="rc-ok" +elif [ "$rc_exit" -eq 77 ] ; then + RC_FILENAME="rc-skipped" +else + RC_FILENAME="rc-failed" + echo "$rc_exit" > "$NFT_TEST_TESTTMPDIR/rc-failed-exit" fi +echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/$RC_FILENAME" END_TIME="$(cut -d ' ' -f1 /proc/uptime)" WALL_TIME="$(awk -v start="$START_TIME" -v end="$END_TIME" "BEGIN { print(end - start) }")" From patchwork Thu Sep 7 22:07:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831150 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Z+28To7d; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYM60PB9z1yhG for ; Fri, 8 Sep 2023 08:09:54 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234718AbjIGWJx (ORCPT ); Thu, 7 Sep 2023 18:09:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229568AbjIGWJp (ORCPT ); Thu, 7 Sep 2023 18:09:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61BAD1BC8 for ; Thu, 7 Sep 2023 15:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124527; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CVCv5AB+p3+Inv0VY0+UNivHlZexnHKpe+VtRkmQID4=; b=Z+28To7dwdgHW05fLyAMzsaEMOhXZornBlAkoXh3ShfX6YG0XCX9M286JoWF7ezw2DB1e0 BoG8dGoDOpMboO6Up8PaDCujgorxDhDREBJRX/JZj1T7Lol3oBNXmMIIbjDwj5/1YXAKRZ ofMjGU8g4CrKniyFfQTXfaq+s/MYYCE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-605-9BLtVBDiNZ-DDajYMfNh6g-1; Thu, 07 Sep 2023 18:08:46 -0400 X-MC-Unique: 9BLtVBDiNZ-DDajYMfNh6g-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 045F2181792D for ; Thu, 7 Sep 2023 22:08:46 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 727C963F6C; Thu, 7 Sep 2023 22:08:45 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 02/11] tests/shell: cleanup print_test_result() and show TAINTED error code Date: Fri, 8 Sep 2023 00:07:14 +0200 Message-ID: <20230907220833.2435010-3-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org We will add more special error codes (122 for VALGRIND). Minor refactor of print_test_result() to make it easier to extend for that. Also, we will soon colorize the output. This preparation patch makes that easier too. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 423c5465c4d4..e0adb27ad104 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -471,25 +471,27 @@ print_test_result() { shift 3 local result_msg_level="I" - local result_msg_status="OK" local result_msg_suffix="" local result_msg_files=( "$NFT_TEST_TESTTMPDIR/testout.log" "$NFT_TEST_TESTTMPDIR/ruleset-diff" ) + local result_msg_status if [ "$rc_got" -eq 0 ] ; then ((ok++)) - elif [ "$rc_got" -eq 124 ] ; then - ((failed++)) - result_msg_level="W" - result_msg_status="DUMP FAIL" + result_msg_status="OK" elif [ "$rc_got" -eq 77 ] ; then ((skipped++)) - result_msg_level="I" result_msg_status="SKIPPED" else ((failed++)) result_msg_level="W" - result_msg_status="FAILED" - result_msg_suffix="got $rc_got" + if [ "$rc_got" -eq 123 ] ; then + result_msg_status="TAINTED" + elif [ "$rc_got" -eq 124 ] ; then + result_msg_status="DUMP FAIL" + else + result_msg_status="FAILED" + result_msg_suffix="got $rc_got" + fi result_msg_files=( "$NFT_TEST_TESTTMPDIR/testout.log" ) fi From patchwork Thu Sep 7 22:07:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831153 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FTVZtiDs; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYM71yBcz1ynh for ; Fri, 8 Sep 2023 08:09:55 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232116AbjIGWJy (ORCPT ); Thu, 7 Sep 2023 18:09:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230519AbjIGWJp (ORCPT ); Thu, 7 Sep 2023 18:09:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19E0B1BC9 for ; Thu, 7 Sep 2023 15:08:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124528; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=p42J59YFB1q7R4cqDRkoV31TIvR4OX+3ZsNaguGhBug=; b=FTVZtiDsRyw5mr3gK8bgVAZ8jZv4bnuSXfuG5G7BlW4V2Yx9LWRVvJm6YLY7nJhNcff45p cf/aJWhlu8Brl/SHV2SEYuntDsYUMbiceG+ZxFuXsZRfU/36ojXgbBtwxtKQu9eoDWvXF9 Lf3KiVqkHg3Tzl5ZgXaIFVMl6d3V9A0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-671-JWlHeQ0NNOCjXwT8DxJYQw-1; Thu, 07 Sep 2023 18:08:47 -0400 X-MC-Unique: JWlHeQ0NNOCjXwT8DxJYQw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C75308007CE for ; Thu, 7 Sep 2023 22:08:46 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 412377B62; Thu, 7 Sep 2023 22:08:46 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 03/11] tests/shell: colorize terminal output with test result Date: Fri, 8 Sep 2023 00:07:15 +0200 Message-ID: <20230907220833.2435010-4-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Colors help to see what is important. It honors the common NO_COLOR= to disable coloring. It also does not colorize, if [ -t 1 ] indicates that stdout is not a terminal. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 71 ++++++++++++++++++++++++++++++++++------ 1 file changed, 61 insertions(+), 10 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index e0adb27ad104..c8688587bbc4 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -1,15 +1,26 @@ #!/bin/bash +GREEN="" +YELLOW="" +RED="" +RESET="" +if [[ -t 1 && -z "$NO_COLOR" ]] ; then + GREEN=$'\e[32m' + YELLOW=$'\e[33m' + RED=$'\e[31m' + RESET=$'\e[0m' +fi + _msg() { local level="$1" shift - local msg - msg="$level: $*" - if [ "$level" = E -o "$level" = W ] ; then - printf '%s\n' "$msg" >&2 + if [ "$level" = E ] ; then + printf '%s\n' "$RED$level$RESET: $*" >&2 + elif [ "$level" = W ] ; then + printf '%s\n' "$YELLOW$level$RESET: $*" >&2 else - printf '%s\n' "$msg" + printf '%s\n' "$level: $*" fi if [ "$level" = E ] ; then exit 1 @@ -28,6 +39,39 @@ msg_info() { _msg I "$@" } +align_text() { + local _OUT_VARNAME="$1" + local _LEFT_OR_RIGHT="$2" + local _INDENT="$3" + shift 3 + local _text="$*" + local _text_plain + local _text_align + local _text_result + local _i + + # This function is needed, because "$text" might contain color escape + # sequences. A plain `printf '%12s' "$text"` will not align properly. + + # strip escape sequences + _text_plain="${_text//$'\e['[0-9]m/}" + _text_plain="${_text_plain//$'\e['[0-9][0-9]m/}" + + _text_align="" + for (( _i = "${#_text_plain}" ; "$_i" < "$_INDENT" ; _i++ )) ; do + _text_align="$_text_align " + done + + if [ "$_LEFT_OR_RIGHT" = left ] ; then + _text_result="$(printf "%s$_text_align-" "$_text")" + else + _text_result="$(printf "$_text_align%s-" "$_text")" + fi + _text_result="${_text_result%-}" + + eval "$_OUT_VARNAME=\"\$_text_result\"" +} + bool_n() { case "$1" in n|N|no|No|NO|0|false|False|FALSE) @@ -459,8 +503,7 @@ print_test_header() { local suffix="$4" local text - text="[$status]" - text="$(printf '%-12s' "$text")" + align_text text left 12 "[$status]" _msg "$msglevel" "$text $testfile${suffix:+: $suffix}" } @@ -477,10 +520,10 @@ print_test_result() { if [ "$rc_got" -eq 0 ] ; then ((ok++)) - result_msg_status="OK" + result_msg_status="${GREEN}OK$RESET" elif [ "$rc_got" -eq 77 ] ; then ((skipped++)) - result_msg_status="SKIPPED" + result_msg_status="${YELLOW}SKIPPED$RESET" else ((failed++)) result_msg_level="W" @@ -492,6 +535,7 @@ print_test_result() { result_msg_status="FAILED" result_msg_suffix="got $rc_got" fi + result_msg_status="$RED$result_msg_status$RESET" result_msg_files=( "$NFT_TEST_TESTTMPDIR/testout.log" ) fi @@ -578,7 +622,14 @@ echo "" kmemleak_found=0 check_kmemleak_force -msg_info "results: [OK] $ok [SKIPPED] $skipped [FAILED] $failed [TOTAL] $((ok+skipped+failed))" +if [ "$failed" -gt 0 ] ; then + RR="$RED" +elif [ "$skipped" -gt 0 ] ; then + RR="$YELLOW" +else + RR="$GREEN" +fi +msg_info "${RR}results$RESET: [OK] $GREEN$ok$RESET [SKIPPED] $YELLOW$skipped$RESET [FAILED] $RED$failed$RESET [TOTAL] $((ok+skipped+failed))" kernel_cleanup From patchwork Thu Sep 7 22:07:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831154 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FEnZ+zzl; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYM742MJz1yhh for ; Fri, 8 Sep 2023 08:09:55 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230519AbjIGWJz (ORCPT ); Thu, 7 Sep 2023 18:09:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42066 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234721AbjIGWJp (ORCPT ); Thu, 7 Sep 2023 18:09:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECC481BD2 for ; Thu, 7 Sep 2023 15:08:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124529; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=pDr2XROpy94VkfvsmuTPCE+mjhl1XAhL4pRKgrGnOd8=; b=FEnZ+zzl+FoDpNOns4Mc34RzF1erFcQpn1KuPYkgHXPboDa568j96vv9uDINs7CTzlQIaz AKe8mS04FjLxBRYAZWSSujVimUqv4eVblFm9FlBcacmlTRLfDKb0Cizp7EeHQfTXKio9B2 DFvVecx9LNqO7Y1N1o3KYa1tuck9pZo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-446-hpxP5DG5OZGvB5kQyWpeXQ-1; Thu, 07 Sep 2023 18:08:47 -0400 X-MC-Unique: hpxP5DG5OZGvB5kQyWpeXQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 974D72820543 for ; Thu, 7 Sep 2023 22:08:47 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0FF6F63F78; Thu, 7 Sep 2023 22:08:46 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 04/11] tests/shell: fix handling failures with VALGRIND=y Date: Fri, 8 Sep 2023 00:07:16 +0200 Message-ID: <20230907220833.2435010-5-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org With VALGRIND=y, on memleaks the tests did not fail. Fix that by passing "--error-exitcode=122" to valgrind. But just returning 122 from $NFT command may not correctly fail the test. Instead, ensure to write a "rc-failed-valrind" file, which is picked up by "test-wrapper.sh" to properly handle the valgrind failure (and fail with error code 122 itself). Also, accept NFT_TEST_VALGRIND_OPTS variable to a pass additional arguments to valgrind. For example a "--suppressions" file. Also show the special error code [VALGRIND] in "run-test.sh". Signed-off-by: Thomas Haller --- tests/shell/helpers/nft-valgrind-wrapper.sh | 15 ++++++++++++++- tests/shell/helpers/test-wrapper.sh | 13 +++++++++---- tests/shell/run-tests.sh | 4 +++- 3 files changed, 26 insertions(+), 6 deletions(-) diff --git a/tests/shell/helpers/nft-valgrind-wrapper.sh b/tests/shell/helpers/nft-valgrind-wrapper.sh index 9da50d4d9d1d..ad8cc74bc781 100755 --- a/tests/shell/helpers/nft-valgrind-wrapper.sh +++ b/tests/shell/helpers/nft-valgrind-wrapper.sh @@ -1,6 +1,6 @@ #!/bin/bash -e -SUFFIX="$(date '+%Y%m%d-%H%M%S.%6N')" +SUFFIX="$(date "+%Y%m%d-%H%M%S.%6N.$$")" rc=0 libtool \ @@ -10,8 +10,21 @@ libtool \ --trace-children=yes \ --leak-check=full \ --show-leak-kinds=all \ + --num-callers=100 \ + --error-exitcode=122 \ + $NFT_TEST_VALGRIND_OPTS \ "$NFT_REAL" \ "$@" \ || rc=$? +if [ "$rc" -eq 122 ] ; then + shopt -s nullglob + FILES=( "$NFT_TEST_TESTTMPDIR/valgrind.$SUFFIX."*".log" ) + shopt -u nullglob + ( + printf '%s\n' "args: $*" + printf '%s\n' "${FILES[*]}" + ) >> "$NFT_TEST_TESTTMPDIR/rc-failed-valgrind" +fi + exit $rc diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh index f8b27b1e9291..405e70c86751 100755 --- a/tests/shell/helpers/test-wrapper.sh +++ b/tests/shell/helpers/test-wrapper.sh @@ -78,13 +78,18 @@ if [ "$rc_dump" -ne 0 ] ; then echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump" fi +rc_valgrind=0 +[ -f "$NFT_TEST_TESTTMPDIR/rc-failed-valgrind" ] && rc_valgrind=122 + rc_tainted=0 if [ "$tainted_before" != "$tainted_after" ] ; then echo "$tainted_after" > "$NFT_TEST_TESTTMPDIR/rc-failed-tainted" rc_tainted=123 fi -if [ "$rc_tainted" -ne 0 ] ; then +if [ "$rc_valgrind" -ne 0 ] ; then + rc_exit="$rc_valgrind" +elif [ "$rc_tainted" -ne 0 ] ; then rc_exit="$rc_tainted" elif [ "$rc_test" -ge 118 -a "$rc_test" -le 124 ] ; then # Special exit codes are reserved. Coerce them. @@ -101,9 +106,9 @@ fi # We always write the real exit code of the test ($rc_test) to one of the files # rc-{ok,skipped,failed}, depending on which it is. # -# Note that there might be other rc-failed-{dump,tainted} files with additional -# errors. Note that if such files exist, the overall state will always be -# failed too (and an "rc-failed" file exists). +# Note that there might be other rc-failed-{dump,tainted,valgrind} files with +# additional errors. Note that if such files exist, the overall state will +# always be failed too (and an "rc-failed" file exists). # # On failure, we also write the combined "$rc_exit" code from "test-wrapper.sh" # to "rc-failed-exit" file. diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index c8688587bbc4..ab91fd4d9053 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -527,7 +527,9 @@ print_test_result() { else ((failed++)) result_msg_level="W" - if [ "$rc_got" -eq 123 ] ; then + if [ "$rc_got" -eq 122 ] ; then + result_msg_status="VALGRIND" + elif [ "$rc_got" -eq 123 ] ; then result_msg_status="TAINTED" elif [ "$rc_got" -eq 124 ] ; then result_msg_status="DUMP FAIL" From patchwork Thu Sep 7 22:07:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831157 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=cMtrwrmJ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYMs1wvWz1ygR for ; Fri, 8 Sep 2023 08:10:33 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239006AbjIGWKf (ORCPT ); Thu, 7 Sep 2023 18:10:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237078AbjIGWKe (ORCPT ); Thu, 7 Sep 2023 18:10:34 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19DE41BD5 for ; Thu, 7 Sep 2023 15:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124530; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LzRJMC35G2BEDlECvoqimMR3Bpfmq/VCP6LOxQd86pk=; b=cMtrwrmJdcpOV69WNnFH9hnjroFJPdwtCMM6ahItIQbO25v5qGl6SWVBGOkdXroeEWa5TN iFf4sijTD7Yn929tIKy0OEt3G76tzB6UBQo/11hBuiPP9upR9Zp5A8aieh59CY4O375hst gifUttNOf2gfd2Rn1Gp9MTNwi4YLpzw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-102-ssaCps8_MViRyq8Wv005gQ-1; Thu, 07 Sep 2023 18:08:48 -0400 X-MC-Unique: ssaCps8_MViRyq8Wv005gQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6DD093802BAC for ; Thu, 7 Sep 2023 22:08:48 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D5B9C7B62; Thu, 7 Sep 2023 22:08:47 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 05/11] tests/shell: print the NFT setting with the VALGRIND=y wrapper Date: Fri, 8 Sep 2023 00:07:17 +0200 Message-ID: <20230907220833.2435010-6-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org With this we see in the info output I: info: NFT=./tests/shell/helpers/nft-valgrind-wrapper.sh Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index ab91fd4d9053..4f0df3217b76 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -411,6 +411,11 @@ echo msg_info "info: NFT_TEST_BASEDIR=$(printf '%q' "$NFT_TEST_BASEDIR")" msg_info "info: NFT_TEST_TMPDIR=$(printf '%q' "$NFT_TEST_TMPDIR")" +if [ "$VALGRIND" == "y" ]; then + NFT="$NFT_TEST_BASEDIR/helpers/nft-valgrind-wrapper.sh" + msg_info "info: NFT=$(printf '%q' "$NFT")" +fi + kernel_cleanup() { if [ "$NFT_TEST_JOBS" -ne 0 ] ; then # When we run jobs in parallel (even with only one "parallel" @@ -442,10 +447,6 @@ kernel_cleanup() { nft_xfrm } -if [ "$VALGRIND" == "y" ]; then - NFT="$NFT_TEST_BASEDIR/helpers/nft-valgrind-wrapper.sh" -fi - echo "" ok=0 skipped=0 From patchwork Thu Sep 7 22:07:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831155 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Z1GODcxH; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYMp65T4z1ygR for ; Fri, 8 Sep 2023 08:10:30 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235201AbjIGWKc (ORCPT ); Thu, 7 Sep 2023 18:10:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234721AbjIGWKc (ORCPT ); Thu, 7 Sep 2023 18:10:32 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80B221BD6 for ; Thu, 7 Sep 2023 15:08:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124530; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WsAjjTR0dF1l/kivAtXilgwtm02gadSuwZIdMrf/k5o=; b=Z1GODcxHR7bmcY7xApqRZhDTPHRstDYwrtIdfenQyLhXojfBOuzersCZx5lOT7J12I2dXc WyhlZH5FsHXm5NWx7X0RDlc/7dfOt9LvCG6U0Ws7vMt0EAFcedRPV6wgTNlHPT1XoHBQ0L ASTvztX5l/wxcAiqveist1oJGPE1Igg= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-608-FtGDooY9NxWcao-O5WkhCw-1; Thu, 07 Sep 2023 18:08:49 -0400 X-MC-Unique: FtGDooY9NxWcao-O5WkhCw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3CF222820543 for ; Thu, 7 Sep 2023 22:08:49 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id AAA587B62; Thu, 7 Sep 2023 22:08:48 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 06/11] tests/shell: don't redirect error/warning messages to stderr Date: Fri, 8 Sep 2023 00:07:18 +0200 Message-ID: <20230907220833.2435010-7-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Writing some messages to stderr and some to stdout is not helpful. Once they are written to separate streams, it's hard to be sure about their relative order. Use grep to filter messages. Also, next we will redirect the entire output also to a file. There the output is also not split in two files. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 4f0df3217b76..e4efbb2de540 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -16,9 +16,9 @@ _msg() { shift if [ "$level" = E ] ; then - printf '%s\n' "$RED$level$RESET: $*" >&2 + printf '%s\n' "$RED$level$RESET: $*" elif [ "$level" = W ] ; then - printf '%s\n' "$YELLOW$level$RESET: $*" >&2 + printf '%s\n' "$YELLOW$level$RESET: $*" else printf '%s\n' "$level: $*" fi From patchwork Thu Sep 7 22:07:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831156 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=H5Q4Ox5w; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYMq1MmRz1yhh for ; Fri, 8 Sep 2023 08:10:31 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234721AbjIGWKc (ORCPT ); Thu, 7 Sep 2023 18:10:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229918AbjIGWKc (ORCPT ); Thu, 7 Sep 2023 18:10:32 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 640451BD7 for ; Thu, 7 Sep 2023 15:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124531; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SD8Q7oHO7IVT5bZSApzfZPV/CLTK+NLOyeYYDsbTj08=; b=H5Q4Ox5wd+qEG1gq4uGdp4zvm0nv2J40J0LNSaRjyDFU+yTJ9+6yMCHmGh5dSNs5EkEA88 xdqrcC9MXWxqtizwqksiF3o26Po+Wxz1w72OU1/cCGR6Q9Wkn8QMx+Z8Ao8uLOuw3HdNKl QM5AhDJVJPTjKCnNFY6oQOraKr2Qbao= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-148-GuZ1i-jfPce3JE9IF2HwKg-1; Thu, 07 Sep 2023 18:08:50 -0400 X-MC-Unique: GuZ1i-jfPce3JE9IF2HwKg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0DA98181792A for ; Thu, 7 Sep 2023 22:08:50 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7B0777B62; Thu, 7 Sep 2023 22:08:49 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 07/11] tests/shell: redirect output of test script to file too Date: Fri, 8 Sep 2023 00:07:19 +0200 Message-ID: <20230907220833.2435010-8-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org It's useful to keep around for later. Redirect to the temporary directory. Note that the file content may be colorized too. `less -R` helps with that. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index e4efbb2de540..4c1ab29b8536 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -382,6 +382,8 @@ NFT_TEST_TMPDIR="$(mktemp --tmpdir="$_TMPDIR" -d "nft-test.$(date '+%Y%m%d-%H%M% msg_error "Failure to create temp directory in \"$_TMPDIR\"" chmod 755 "$NFT_TEST_TMPDIR" +exec &> >(tee "$NFT_TEST_TMPDIR/test.log") + NFT_REAL="${NFT_REAL-$NFT}" msg_info "conf: NFT=$(printf '%q' "$NFT")" From patchwork Thu Sep 7 22:07:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831160 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Q6g0jXPt; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYMt1dBfz1ygR for ; Fri, 8 Sep 2023 08:10:34 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239551AbjIGWKg (ORCPT ); Thu, 7 Sep 2023 18:10:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238962AbjIGWKf (ORCPT ); Thu, 7 Sep 2023 18:10:35 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1695C1BD8 for ; Thu, 7 Sep 2023 15:08:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124532; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wkvpMeZsoh9t6GwznWDqkUKuyzOG0O91IO2nwsaEmrU=; b=Q6g0jXPty6zvcVyWz7Ksq61MDWF1rcGS6VZBt3/iuJkIbGolIG1BsyG1iT96k/nt3G35PU 7wFC4cYTbxY9kwxQRDUGyw2e+3JixtHl/5PSMxuwZ7LcY2DQTjPIHL2EsujisXRPovTDOT QPn9Op2ZmTl8Ksnlm0LA3BuhV0YDr9E= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-610-3EXJGVQ7Pb6ATbDDzZVpHw-1; Thu, 07 Sep 2023 18:08:51 -0400 X-MC-Unique: 3EXJGVQ7Pb6ATbDDzZVpHw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D33E31817929 for ; Thu, 7 Sep 2023 22:08:50 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4A7117B62; Thu, 7 Sep 2023 22:08:50 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 08/11] tests/shell: print "kernel is tainted" separate from test result Date: Fri, 8 Sep 2023 00:07:20 +0200 Message-ID: <20230907220833.2435010-9-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Once the kernel is tainted, it stays until reboot. It would not be useful to fail the entire test run based on that (and we don't do that). But then, it seems odd to print this in the same style as the test results, because a [FAILED] of a test counts as an overall failure. Instead, print this warning in a different style. Previously: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: [FAILED] kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Now: $ ./tests/shell/run-tests.sh -- /usr/bin/true ... W: kernel is tainted I: [OK] /usr/bin/true I: results: [OK] 1 [SKIPPED] 0 [FAILED] 0 [TOTAL] 1 Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 4c1ab29b8536..6abb6c0c73a0 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -496,7 +496,8 @@ check_kmemleak() read kernel_tainted < /proc/sys/kernel/tainted if [ "$kernel_tainted" -ne 0 ] ; then - msg_warn "[FAILED] kernel is tainted" + msg_warn "kernel is tainted" + echo fi print_test_header() { From patchwork Thu Sep 7 22:07:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831158 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=I5Sk2SC1; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYMs4F34z1yhh for ; Fri, 8 Sep 2023 08:10:33 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237078AbjIGWKf (ORCPT ); Thu, 7 Sep 2023 18:10:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229918AbjIGWKe (ORCPT ); Thu, 7 Sep 2023 18:10:34 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17D481BD9 for ; Thu, 7 Sep 2023 15:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2wDAQHWyyBKQ6JN1XwPkoxksbMSgc1OxGbmM3XvGO24=; b=I5Sk2SC1o9KFRRoQI2G4N+nBgUzJ6LcV7jVFjVcjJMaawMlfAYrMzHeaHPALiBu0R0p1Jf 3a3KyqyPTZeqODt5LKjxMloPSHsW+9uAFV2fjYTW66Nn9O2F8DZ1acz4zt1YY+0EPAsWIR KqOWg7lGQnswWnibdqO1e0qtCV1XLr8= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-622-B3BeGPkLPVWtYh8rUTCx5A-1; Thu, 07 Sep 2023 18:08:52 -0400 X-MC-Unique: B3BeGPkLPVWtYh8rUTCx5A-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BF4272820543 for ; Thu, 7 Sep 2023 22:08:51 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2F4357B62; Thu, 7 Sep 2023 22:08:50 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 09/11] tests/shell: no longer enable verbose output when selecting a test Date: Fri, 8 Sep 2023 00:07:21 +0200 Message-ID: <20230907220833.2435010-10-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Previously, when selecting a test on the command line, it would also enable verbose output (except if the "--" separator was used). This convenience feature seems not great because the output from the test badly clutters the "run-test.sh" output. Now that the test results are all on disk, you can search them after the run with great flexibility (grep). Additionally, in previous versions, command line argument parsing was more restrictive, requiring that "-v" always be placed first. Now, the order does not matter, so it's easy to edit the command prompt and append a "-v", if that is what you want. Or if you really like verbose output, then `export VERBOSE=y`. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 6abb6c0c73a0..bb73a771dfdc 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -100,7 +100,7 @@ usage() { echo "OPTIONS:" echo " -h|--help : Print usage." echo " -L|--list-tests : List test names and quit." - echo " -v : Sets VERBOSE=y. Specifying tests without \"--\" enables verbose mode." + echo " -v : Sets VERBOSE=y." echo " -g : Sets DUMPGEN=y." echo " -V : Sets VALGRIND=y." echo " -K : Sets KMEMLEAK=y." @@ -218,10 +218,7 @@ while [ $# -gt 0 ] ; do shift $# ;; *) - # Any unrecognized option is treated as a test name, and also - # enable verbose tests. TESTS+=( "$A" ) - VERBOSE=y ;; esac done From patchwork Thu Sep 7 22:07:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831161 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=hz0ZlPeA; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYN03dz7z1ygR for ; Fri, 8 Sep 2023 08:10:40 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240004AbjIGWKl (ORCPT ); Thu, 7 Sep 2023 18:10:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45996 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237290AbjIGWKl (ORCPT ); Thu, 7 Sep 2023 18:10:41 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3CC51BDA for ; Thu, 7 Sep 2023 15:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124534; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j4aAUBnOj+3N9iemaB01OvKvoXA8L5ay7wQO+OpOSaI=; b=hz0ZlPeA3th+rw6ObH5dsYSH1UzcMPueQ/4MV+m2oNeKlAGB92Ghr4zIS9FVT+rgjPKxxz fcQK0XaoRRB2znA+K7PlN+ttKTmITbaKezn4cLihv7LZCUJWYc6QHhLKQrk6Sje8TGXo1P MMkWaokZcXamIs0qfrf2IGz51chH6jY= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-407-CUWCZPTaMLWThMPEAIRGZw-1; Thu, 07 Sep 2023 18:08:52 -0400 X-MC-Unique: CUWCZPTaMLWThMPEAIRGZw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 938753C0EACC for ; Thu, 7 Sep 2023 22:08:52 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0A06C7B62; Thu, 7 Sep 2023 22:08:51 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 10/11] tests/shell: record wall time of test run in result data Date: Fri, 8 Sep 2023 00:07:22 +0200 Message-ID: <20230907220833.2435010-11-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org When running tests, it's useful to see how long it took. Keep track if the timestamps in a "times" file. Try: ( \ for d in /tmp/nft-test.latest.*/test-*/ ; do \ printf '%10.2f %s\n' \ "$(sed '1!d' "$d/times")" \ "$(cat "$d/name")" ; \ done \ | sort -n \ | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' ; \ printf '%10.2f wall time\n' "$(sed '1!d' /tmp/nft-test.latest.*/times)" \ ) Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index bb73a771dfdc..35621bac569d 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -249,6 +249,8 @@ if [ "$DO_LIST_TESTS" = y ] ; then exit 0 fi +START_TIME="$(cut -d ' ' -f1 /proc/uptime)" + _TMPDIR="${TMPDIR:-/tmp}" if [ "$NFT_TEST_HAS_REALROOT" = "" ] ; then @@ -636,6 +638,20 @@ msg_info "${RR}results$RESET: [OK] $GREEN$ok$RESET [SKIPPED] $YELLOW$skipped$RES kernel_cleanup +# ( \ +# for d in /tmp/nft-test.latest.*/test-*/ ; do \ +# printf '%10.2f %s\n' \ +# "$(sed '1!d' "$d/times")" \ +# "$(cat "$d/name")" ; \ +# done \ +# | sort -n \ +# | awk '{print $0; s+=$1} END{printf("%10.2f\n", s)}' ; \ +# printf '%10.2f wall time\n' "$(sed '1!d' /tmp/nft-test.latest.*/times)" \ +# ) +END_TIME="$(cut -d ' ' -f1 /proc/uptime)" +WALL_TIME="$(awk -v start="$START_TIME" -v end="$END_TIME" "BEGIN { print(end - start) }")" +printf "%s\n" "$WALL_TIME" "$START_TIME" "$END_TIME" > "$NFT_TEST_TMPDIR/times" + if [ "$failed" -gt 0 -o "$NFT_TEST_KEEP_LOGS" = y ] ; then msg_info "check the temp directory \"$NFT_TEST_TMPDIR\" (\"$NFT_TEST_LATEST\")" msg_info " ls -lad \"$NFT_TEST_LATEST\"/*/*" From patchwork Thu Sep 7 22:07:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1831159 X-Patchwork-Delegate: fw@strlen.de Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=AmvcvhYD; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RhYMs6M9Hz1ynh for ; Fri, 8 Sep 2023 08:10:33 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229918AbjIGWKf (ORCPT ); Thu, 7 Sep 2023 18:10:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237290AbjIGWKe (ORCPT ); Thu, 7 Sep 2023 18:10:34 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 51C531BDC for ; Thu, 7 Sep 2023 15:08:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1694124535; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YeS+cw9GRyeHvD7gpqPCu9ZofWdnkynaCj0lhD1IbS4=; b=AmvcvhYDQ7To2p5wYJzFdjHkrYJxqo9ICVDcN7xKSyQIECIMZvQpkfE7+GGJKEhNI3gsts zfSEelO6m8htaQz0nbleEZ4hYvC7gtEfvB8laisRlwZYL6y04f/p+9to0VZlvnGJXj2vr2 Fo2t9b23p2QN6yy4M2lKsRWDGQiOuSw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-460-WmQzQEMkObOsfkMH6r8ZAA-1; Thu, 07 Sep 2023 18:08:53 -0400 X-MC-Unique: WmQzQEMkObOsfkMH6r8ZAA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 69B8E3802AD1 for ; Thu, 7 Sep 2023 22:08:53 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.3]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D0FB463F6C; Thu, 7 Sep 2023 22:08:52 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft 11/11] tests/shell: set NFT_TEST_JOBS based on $(nproc) Date: Fri, 8 Sep 2023 00:07:23 +0200 Message-ID: <20230907220833.2435010-12-thaller@redhat.com> In-Reply-To: <20230907220833.2435010-1-thaller@redhat.com> References: <20230907220833.2435010-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Choose 150% of $(nproc) for the default vlaue of NFT_TEST_JOBS (rounded up). The minimal value chosen by default is 2. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 35621bac569d..c622c1509500 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -154,7 +154,7 @@ usage() { echo " mount namespace." echo " This is only honored when \$NFT_TEST_UNSHARE_CMD= is set. Otherwise it's detected." echo " NFT_TEST_KEEP_LOGS=*|y: Keep the temp directory. On success, it will be deleted by default." - echo " NFT_TEST_JOBS=: number of jobs for parallel execution. Defaults to \"12\" for parallel run." + echo " NFT_TEST_JOBS=: number of jobs for parallel execution. Defaults to \"\$(nproc)*1.5\" for parallel run." echo " Setting this to \"0\" or \"1\", means to run jobs sequentially." echo " Setting this to \"0\" means also to perform global cleanups between tests (remove" echo " kernel modules)." @@ -167,13 +167,17 @@ NFT_TEST_BASEDIR="$(dirname "$0")" # Export the base directory. It may be used by tests. export NFT_TEST_BASEDIR +_NFT_TEST_JOBS_DEFAULT="$(nproc)" +[ "$_NFT_TEST_JOBS_DEFAULT" -gt 0 ] 2>/dev/null || _NFT_TEST_JOBS_DEFAULT=1 +_NFT_TEST_JOBS_DEFAULT="$(( _NFT_TEST_JOBS_DEFAULT + (_NFT_TEST_JOBS_DEFAULT + 1) / 2 ))" + VERBOSE="$(bool_y "$VERBOSE")" DUMPGEN="$(bool_y "$DUMPGEN")" VALGRIND="$(bool_y "$VALGRIND")" KMEMLEAK="$(bool_y "$KMEMLEAK")" NFT_TEST_KEEP_LOGS="$(bool_y "$NFT_TEST_KEEP_LOGS")" NFT_TEST_HAS_REALROOT="$NFT_TEST_HAS_REALROOT" -NFT_TEST_JOBS="${NFT_TEST_JOBS:-12}" +NFT_TEST_JOBS="${NFT_TEST_JOBS:-$_NFT_TEST_JOBS_DEFAULT}" DO_LIST_TESTS= TESTS=() @@ -345,7 +349,7 @@ export NFT_TEST_HAS_UNSHARED_MOUNT # normalize the jobs number to be an integer. case "$NFT_TEST_JOBS" in - ''|*[!0-9]*) NFT_TEST_JOBS=12 ;; + ''|*[!0-9]*) NFT_TEST_JOBS=_NFT_TEST_JOBS_DEFAULT ;; esac if [ -z "$NFT_TEST_UNSHARE_CMD" -a "$NFT_TEST_JOBS" -gt 1 ] ; then NFT_TEST_JOBS=1