From patchwork Mon May 11 21:48:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Burgess X-Patchwork-Id: 1288101 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=gcc-patches-bounces@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=embecosm.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=embecosm.com header.i=@embecosm.com header.a=rsa-sha256 header.s=google header.b=cdas6YCr; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49LZM00tZPz9sRK for ; Tue, 12 May 2020 07:48:41 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B4A98388E810; Mon, 11 May 2020 21:48:38 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by sourceware.org (Postfix) with ESMTPS id 96F25386F465 for ; Mon, 11 May 2020 21:48:33 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 96F25386F465 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=embecosm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=andrew.burgess@embecosm.com Received: by mail-wm1-x344.google.com with SMTP id u16so21061592wmc.5 for ; Mon, 11 May 2020 14:48:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=embecosm.com; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=eHf8p+DlZdtEGP0HV5hWSvULuhqzRcuN8m64zgffBLQ=; b=cdas6YCr6QD8PeHq+4GdjG0zbfGZGvyVs531cwNDL6fZVa8y/QNa3Pn+k8CzsKh6TT LaFbhEkIropDgzr7dDj/+kqn054ggY2ua+kSnDp5O0wFRT2MQxuj9An+9XX7Gg9QDnW3 O5IrR/ZdlXD7asfWbZH7HENLYPJ0q6aoV7IWOknIwA9qw4dZ6reY38ZrO/lzJvEI7MMJ 3E2xKn5an8vSRGTGqjiA2Sd+qJ90ZoD1oDQwz5tuv1SLxN28jnrUKk8JoCO+ryK1R8Tz IdX8we9khr6OBFi3o+RWEdcY2Am1EiGgKUDLrenoRNetsTU/cr3smItI9vuuzdp8ILZ1 Y5SQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=eHf8p+DlZdtEGP0HV5hWSvULuhqzRcuN8m64zgffBLQ=; b=bGJWO6261e/c3T1qjd+ToSb06M2RyZ0ZQKZTTkMoTJ59aFtXaUP7jZT7lmh5jeiomI cUwr/JPYAd9lzxdK5E34UXlyZC31BVb/EDbHrUjObxXtZpB9OQvagWLH+8PQvlmZHuev Q+dXtJZE2U+PL747fWTIg+WAIZoOrEkpurerDboS5+SqlDPLNf1LPeVP3CFxycE2Qr1N CuhXt3PqbWLLFZTCtFSPPVbmB9WQkki4lY0sZewpOVlPz5GBslhhIOfu9rab65FH31/J WGiTdtk4Han5ShoJy9rxN8x0LHeh7K8vEy95K43NBzQhYHT/XJ/DbAVm5kaERQVqYbHd pW8w== X-Gm-Message-State: AGi0PubWi4n4TLis0aOYbjukjkqHGwURQFjLA+FLHEbtXoOYAFRSl5sk p+k1zQ++23ctnO2QT32Hu78Lp1xwQVA= X-Google-Smtp-Source: APiQypIWiEWmkSCqE30YQpQNNaGXGmneIld1dqOvkI1rjLtH/mimgULFXDFpT6/XIY6lCN/gdih/Uw== X-Received: by 2002:a7b:c8c1:: with SMTP id f1mr22921296wml.66.1589233712020; Mon, 11 May 2020 14:48:32 -0700 (PDT) Received: from localhost (host81-151-181-184.range81-151.btcentralplus.com. [81.151.181.184]) by smtp.gmail.com with ESMTPSA id o205sm14744610wmo.32.2020.05.11.14.48.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 11 May 2020 14:48:31 -0700 (PDT) From: Andrew Burgess To: gcc-patches@gcc.gnu.org Subject: [PATCH] contrib: Handle GDB specific test result types Date: Mon, 11 May 2020 22:48:25 +0100 Message-Id: <20200511214825.825295-1-andrew.burgess@embecosm.com> X-Mailer: git-send-email 2.25.3 MIME-Version: 1.0 X-Spam-Status: No, score=-17.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces@gcc.gnu.org Sender: "Gcc-patches" This commit is for the benefit of GDB, but as the binutils-gdb repository shares the contrib/ directory with gcc, this commit must first be applied to gcc then copied back to binutils-gdb. This commit extends the two scripts contrib/dg-extract-results.{py,sh} to handle some new, GDB specific test result types. These test results types should never appear in GCC, or any other tool that shares the contrib/ directly, so this change should be harmless. In this patch series: https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html changes were made in GDB's use of Dejagnu so that two additional conditions could be detected, these are: 1. Test names that contain either the build or source paths. Such test names make it difficult to compare the results of two test runs of GDB from two different directories, and 2. Duplicate test names. Duplicates make it difficult to track down exactly which test has failed. When running Dejagnu on GDB we can now (sometimes) see two additional test result types matching the above conditions, these are '# of paths in test names' and '# of duplicate test names'. If the test is run in parallel mode (make -j...) then these extra test results will appear in the individual test summary files, but are not merged into the final summary file. Additionally, within the summary file there are now two new types of test summary line, these are 'PATH: ...' and 'DUPLICATE: ...', these allow users to quickly search the test summary to track down where the offending test names are. These lines are similarly not merged into the unified gdb.sum file after a parallel test run. This commit extends the dg-extract-results.* scripts to calculate the totals for the two new result types, and to copy the new test summary lines into the unified summary file. contrib/ChangeLog: * dg-extract-results.py: Handle GDB specific test types. * dg-extract-results.sh: Likewise. --- contrib/ChangeLog | 5 +++++ contrib/dg-extract-results.py | 6 ++++-- contrib/dg-extract-results.sh | 12 +++++++++++- 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/contrib/dg-extract-results.py b/contrib/dg-extract-results.py index 7100794d42a..30aa68771d4 100644 --- a/contrib/dg-extract-results.py +++ b/contrib/dg-extract-results.py @@ -117,7 +117,7 @@ class Prog: self.tool_re = re.compile (r'^\t\t=== (.*) tests ===$') self.result_re = re.compile (r'^(PASS|XPASS|FAIL|XFAIL|UNRESOLVED' r'|WARNING|ERROR|UNSUPPORTED|UNTESTED' - r'|KFAIL|KPASS):\s*(.+)') + r'|KFAIL|KPASS|PATH|DUPLICATE):\s*(.+)') self.completed_re = re.compile (r'.* completed at (.*)') # Pieces of text to write at the head of the output. # start_line is a pair in which the first element is a datetime @@ -143,7 +143,9 @@ class Prog: '# of known failures\t\t', '# of untested testcases\t\t', '# of unresolved testcases\t', - '# of unsupported tests\t\t' + '# of unsupported tests\t\t', + '# of paths in test names\t', + '# of duplicate test names\t' ] self.runs = dict() diff --git a/contrib/dg-extract-results.sh b/contrib/dg-extract-results.sh index f948088370e..ff6c50d029c 100755 --- a/contrib/dg-extract-results.sh +++ b/contrib/dg-extract-results.sh @@ -326,7 +326,7 @@ BEGIN { } } /^\t\t=== .* ===$/ { curvar = ""; next } -/^(PASS|XPASS|FAIL|XFAIL|UNRESOLVED|WARNING|ERROR|UNSUPPORTED|UNTESTED|KFAIL|KPASS):/ { +/^(PASS|XPASS|FAIL|XFAIL|UNRESOLVED|WARNING|ERROR|UNSUPPORTED|UNTESTED|KFAIL|KPASS|PATH|DUPLICATE):/ { testname=\$2 # Ugly hack for gfortran.dg/dg.exp if ("$TOOL" == "gfortran" && testname ~ /^gfortran.dg\/g77\//) @@ -400,6 +400,7 @@ BEGIN { variant="$VAR" tool="$TOOL" passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; kpasscnt=0; kfailcnt=0; unsupcnt=0; unrescnt=0; dgerrorcnt=0; + pathcnt=0; dupcnt=0 curvar=""; insummary=0 } /^Running target / { curvar = \$3; next } @@ -414,6 +415,8 @@ BEGIN { /^# of untested testcases/ { if (insummary == 1) untstcnt += \$5; next; } /^# of unresolved testcases/ { if (insummary == 1) unrescnt += \$5; next; } /^# of unsupported tests/ { if (insummary == 1) unsupcnt += \$5; next; } +/^# of paths in test names/ { if (insummary == 1) pathcnt += \$7; next; } +/^# of duplicate test names/ { if (insummary == 1) dupcnt += \$6; next; } /^$/ { if (insummary == 1) { insummary = 0; curvar = "" } next @@ -431,6 +434,8 @@ END { if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt) if (unrescnt != 0) printf ("# of unresolved testcases\t%d\n", unrescnt) if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt) + if (pathcnt != 0) printf ("# of paths in test names\t%d\n", pathcnt) + if (dupcnt != 0) printf ("# of duplicate test names\t%d\n", dupcnt) } EOF @@ -452,6 +457,7 @@ cat << EOF > $TOTAL_AWK BEGIN { tool="$TOOL" passcnt=0; failcnt=0; untstcnt=0; xpasscnt=0; xfailcnt=0; kfailcnt=0; unsupcnt=0; unrescnt=0; dgerrorcnt=0 + pathcnt=0; dupcnt=0 } /^# of DejaGnu errors/ { dgerrorcnt += \$5 } /^# of expected passes/ { passcnt += \$5 } @@ -463,6 +469,8 @@ BEGIN { /^# of untested testcases/ { untstcnt += \$5 } /^# of unresolved testcases/ { unrescnt += \$5 } /^# of unsupported tests/ { unsupcnt += \$5 } +/^# of paths in test names/ { pathcnt += \$7 } +/^# of duplicate test names/ { dupcnt += \$6 } END { printf ("\n\t\t=== %s Summary ===\n\n", tool) if (dgerrorcnt != 0) printf ("# of DejaGnu errors\t\t%d\n", dgerrorcnt) @@ -475,6 +483,8 @@ END { if (untstcnt != 0) printf ("# of untested testcases\t\t%d\n", untstcnt) if (unrescnt != 0) printf ("# of unresolved testcases\t%d\n", unrescnt) if (unsupcnt != 0) printf ("# of unsupported tests\t\t%d\n", unsupcnt) + if (pathcnt != 0) printf ("# of paths in test names\t%d\n", pathcnt) + if (dupcnt != 0) printf ("# of duplicate test names\t%d\n", dupcnt) } EOF