From patchwork Tue Jun 5 00:28:31 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Diego Novillo X-Patchwork-Id: 162885 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) by ozlabs.org (Postfix) with SMTP id 8FD85B6F13 for ; Tue, 5 Jun 2012 10:28:53 +1000 (EST) Comment: DKIM? See http://www.dkim.org DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=gcc.gnu.org; s=default; x=1339460933; h=Comment: DomainKey-Signature:Received:Received:Received:Received:Received: Received:Received:Received:Received:To:Subject:Message-Id:Date: From:Mailing-List:Precedence:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:Sender:Delivered-To; bh=FZpjbt6 KtuVfvCYuExBi/SsRLl0=; b=aEaDrCkqIhAfzM2VrNM+GvAok13Ajdit8S1KAEe 6fqR21vGLRYCBl4pqC0O0yoBo/i4GTx/c0E/HxGQ/A4EygDwDM3FXcE9SkkJxIN5 O4nkjbe4khz/o+yzk4KcBBxfGP0hdbikvYxhyXXVvmlqrcc8J34eOlRDdcjolu0H nN/4= Comment: DomainKeys? See http://antispam.yahoo.com/domainkeys DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=gcc.gnu.org; h=Received:Received:X-SWARE-Spam-Status:X-Spam-Check-By:Received:Received:X-Google-DKIM-Signature:Received:Received:Received:Received:Received:To:Subject:Message-Id:Date:From:X-Gm-Message-State:X-IsSubscribed:Mailing-List:Precedence:List-Id:List-Unsubscribe:List-Archive:List-Post:List-Help:Sender:Delivered-To; b=JSBbPtojknlX3/o77fbo9RdbSJe/o1jPSZ/jL9kOGdB2kTNqvAixRJGgBCvuVM Z793htozfOIJNsAdwIET3XLYz28KayoO8LYjINLrMVlV9qFeugOb+V7rpQV0obiB Di3GTC1Z7SEqrOGw5ZSpQophgKNUUClRcRMcRj1+iHJ+Y=; Received: (qmail 29831 invoked by alias); 5 Jun 2012 00:28:50 -0000 Received: (qmail 29822 invoked by uid 22791); 5 Jun 2012 00:28:48 -0000 X-SWARE-Spam-Status: No, hits=-4.3 required=5.0 tests=AWL, BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, KHOP_RCVD_TRUST, RCVD_IN_DNSWL_LOW, RCVD_IN_HOSTKARMA_YE, T_RP_MATCHES_RCVD X-Spam-Check-By: sourceware.org Received: from mail-fa0-f73.google.com (HELO mail-fa0-f73.google.com) (209.85.161.73) by sourceware.org (qpsmtpd/0.43rc1) with ESMTP; Tue, 05 Jun 2012 00:28:34 +0000 Received: by faas14 with SMTP id s14so360864faa.2 for ; Mon, 04 Jun 2012 17:28:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=to:subject:message-id:date:from:x-gm-message-state; bh=ANeXTd5RsMHc2l7/uFvLkYthhBCQu/+O8Ra7qXQPjUY=; b=cuod+i8XLKtNsTk2Uh37NvdjBnowOkhnG4MTGiiL30Sc6WfnTBxCIIwRGou/f3HSGN QO1fLuhvJYbU/rS7pAO7VkyzJRjsVs2YH1xCsbAt/dqhc9Qw/qQdi2AfZW1BZyfkTCkF 2tIWOo4pSDt2kcqbUqs2m6X4p965xFbTjax36TBzRsqJHFweUmMLX84k+13vtuJ/FFKV AfECxLevs/3aECMjylytAvjYBkEYngldOppLVhP4uIqyjJsV3HEOQaWfXByrYqm8AJXh OWrly7l1uxLKuZyYmPbs77vC0IaXcuX56JbI0K1fqv4SxnyYiy3LBir5bYUeVkZCnARr s1iw== Received: by 10.14.101.2 with SMTP id a2mr4378600eeg.6.1338856112606; Mon, 04 Jun 2012 17:28:32 -0700 (PDT) Received: by 10.14.101.2 with SMTP id a2mr4378596eeg.6.1338856112526; Mon, 04 Jun 2012 17:28:32 -0700 (PDT) Received: from hpza9.eem.corp.google.com ([74.125.121.33]) by gmr-mx.google.com with ESMTPS id d52si12802441eei.1.2012.06.04.17.28.32 (version=TLSv1/SSLv3 cipher=AES128-SHA); Mon, 04 Jun 2012 17:28:32 -0700 (PDT) Received: from torture.tor.corp.google.com (torture.tor.corp.google.com [172.29.41.4]) by hpza9.eem.corp.google.com (Postfix) with ESMTP id 4B0045C0050; Mon, 4 Jun 2012 17:28:32 -0700 (PDT) Received: by torture.tor.corp.google.com (Postfix, from userid 54752) id 8B1121019D3; Mon, 4 Jun 2012 20:28:31 -0400 (EDT) To: reply@codereview.appspotmail.com,gcc-patches@gcc.gnu.org Subject: [contrib] Extend validate_failures.py to run outside the build directory (issue6268051) Message-Id: <20120605002831.8B1121019D3@torture.tor.corp.google.com> Date: Mon, 4 Jun 2012 20:28:31 -0400 (EDT) From: dnovillo@google.com (Diego Novillo) X-Gm-Message-State: ALoCoQkYDYO45Bm87mSeLlUxLtilOZhIY/BACG+kz1q0rCSa0ef4L5GCwErrhHR29d9Wqf0s0BGy8bexwDPsO/baPQQqocoQRtpRgc5Fjk11L6tuYFxIJisrJ+UmOaCEwPZ2GmRUcM1CgJky9iLN2vrsC9TMRfLA0w== X-IsSubscribed: yes Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org This patch adds three new arguments to validate_failures.py so it can be used outside the build directory: --ignore_missing_failures When a failure is expected in the manifest but it is not found in the actual results, the script produces a note alerting to this fact. This means that the expected failure has been fixed, or it did not run, or it may simply be flaky. With this option, one can ask the script not to show the missing failures. --manifest Name of the manifest file to use. By default, the script will look for the manifest file in the source directory associated with this build. With this option, one can point to any arbitrary manifest file. I renamed the old --manifest flag to --produce_manifest. --results Space-separated list of .sum files with the testing results to check. The only content needed from these files are the lines starting with FAIL, XPASS or UNRESOLVED. 2012-06-04 Diego Novillo * testsuite-management/validate_failures.py (GetResults): Change argument to accept list of summary files to analyze. Update callers. (CheckExpectedResults): Handle options.manifest, options.results and options.ignore_missing_failures. (Main): Add options --manifest, --results and --ignore_missing_failures. --- This patch is available for review at http://codereview.appspot.com/6268051 diff --git a/contrib/testsuite-management/validate_failures.py b/contrib/testsuite-management/validate_failures.py index c0436e5..c07be84 100755 --- a/contrib/testsuite-management/validate_failures.py +++ b/contrib/testsuite-management/validate_failures.py @@ -1,4 +1,4 @@ -#!/usr/bin/python +#!/usr/bin/python2.6 # Script to compare testsuite failures against a list of known-to-fail # tests. @@ -206,10 +206,8 @@ def GetSumFiles(builddir): return sum_files -def GetResults(builddir): - """Collect all the test results from .sum files under the given build - directory.""" - sum_files = GetSumFiles(builddir) +def GetResults(sum_files): + """Collect all the test results from the given .sum files.""" build_results = set() for sum_fname in sum_files: print '\t%s' % sum_fname @@ -258,16 +256,26 @@ def PrintSummary(msg, summary): def CheckExpectedResults(options): - (srcdir, target, valid_build) = GetBuildData(options) - if not valid_build: - return False + if not options.manifest: + (srcdir, target, valid_build) = GetBuildData(options) + if not valid_build: + return False + manifest_name = _MANIFEST_PATH_PATTERN % (srcdir, target) + else: + manifest_name = options.manifest + if not os.path.exists(manifest_name): + Error('Manifest file %s does not exist.' % manifest_name) - manifest_name = _MANIFEST_PATH_PATTERN % (srcdir, target) print 'Manifest: %s' % manifest_name manifest = GetManifest(manifest_name) - print 'Getting actual results from build' - actual = GetResults(options.build_dir) + if not options.results: + print 'Getting actual results from build' + sum_files = GetSumFiles(options.build_dir) + else: + print 'Getting actual results from user-provided results' + sum_files = options.results.split() + actual = GetResults(sum_files) if options.verbosity >= 1: PrintSummary('Tests expected to fail', manifest) @@ -280,7 +288,7 @@ def CheckExpectedResults(options): PrintSummary('Build results not in the manifest', actual_vs_manifest) tests_ok = False - if len(manifest_vs_actual) > 0: + if not options.ignore_missing_failures and len(manifest_vs_actual) > 0: PrintSummary('Manifest results not present in the build' '\n\nNOTE: This is not a failure. It just means that the ' 'manifest expected\nthese tests to fail, ' @@ -314,20 +322,42 @@ def ProduceManifest(options): def Main(argv): parser = optparse.OptionParser(usage=__doc__) + + # Keep the following list sorted by option name. parser.add_option('--build_dir', action='store', type='string', dest='build_dir', default='.', help='Build directory to check (default = .)') - parser.add_option('--manifest', action='store_true', dest='manifest', - default=False, help='Produce the manifest for the current ' - 'build (default = False)') parser.add_option('--force', action='store_true', dest='force', - default=False, help='When used with --manifest, it will ' - 'overwrite an existing manifest file (default = False)') + default=False, help='When used with --produce_manifest, ' + 'it will overwrite an existing manifest file ' + '(default = False)') + parser.add_option('--ignore_missing_failures', action='store_true', + dest='ignore_missing_failures', default=False, + help='When a failure is expected in the manifest but ' + 'it is not found in the actual results, the script ' + 'produces a note alerting to this fact. This means ' + 'that the expected failure has been fixed, or ' + 'it did not run, or it may simply be flaky ' + '(default = False)') + parser.add_option('--manifest', action='store', type='string', + dest='manifest', default=None, + help='Name of the manifest file to use (default = ' + 'taken from contrib/testsuite-managment/.xfail)') + parser.add_option('--produce_manifest', action='store_true', + dest='produce_manifest', default=False, + help='Produce the manifest for the current ' + 'build (default = False)') + parser.add_option('--results', action='store', type='string', + dest='results', default=None, help='Space-separated list ' + 'of .sum files with the testing results to check. The ' + 'only content needed from these files are the lines ' + 'starting with FAIL, XPASS or UNRESOLVED (default = ' + '.sum files collected from the build directory).') parser.add_option('--verbosity', action='store', dest='verbosity', type='int', default=0, help='Verbosity level (default = 0)') (options, _) = parser.parse_args(argv[1:]) - if options.manifest: + if options.produce_manifest: retval = ProduceManifest(options) else: retval = CheckExpectedResults(options)