From patchwork Fri Oct 30 03:46:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1390757 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=ah/ycVFW; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CMpJx4f61z9sPB for ; Fri, 30 Oct 2020 14:51:45 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 6A4DB826B9; Fri, 30 Oct 2020 04:48:15 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="ah/ycVFW"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id E3D848266C; Fri, 30 Oct 2020 04:47:14 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-io1-xd41.google.com (mail-io1-xd41.google.com [IPv6:2607:f8b0:4864:20::d41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 4ACE382656 for ; Fri, 30 Oct 2020 04:47:11 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-io1-xd41.google.com with SMTP id q25so6157078ioh.4 for ; Thu, 29 Oct 2020 20:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IEeKFv/zw9nGfno4wnGRWbTr2c59JJztM4sSL/77FxU=; b=ah/ycVFWvGSCYJdVKd561NErl29MF4MutJ2o7o+d1Cbf9nQRxDd6v+fHR/6i2zNJFw b4R3f0myN82S5Ft3YnC1IT3Jv34lwxBlj9h7xdy3Se9oAckjlhtNHTT+N6/iugambNSY zl/YOicGEai6BuYRYxshmLxQ0Y6GfPC02+zDg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IEeKFv/zw9nGfno4wnGRWbTr2c59JJztM4sSL/77FxU=; b=SpGmLZZVL4plL+ptvyHyX2o+uj6G6sUrk/B4rzyaUiKZKVdsabXu+mleM0GTJ42IkB SpMIVll5rfNcfsVZYvOSI4FjHDR3u33YlD0nMu7AkaON8ZHrDgNUfxGjxBxd9Ewd5GTB Y6jZR/GWyHqCR40RN1AHTdOEoDC9B1F8gDjo2fTYaz/eYM/XkHjecAKsjbG6/eDKrQuz yxYF7ttBbA24c5bOr+2QtIdY2HijOe7ZLr1GzKRSSviepiuAdFNybwesRTnia6v5JlcK a/Fh+9Z5XfXh7KohR+yWH/BViPP+VMqxZaQVvTY87nFI9oxsSSCr0Z6ElawghI+tcsGp t98Q== X-Gm-Message-State: AOAM533rztTwC3VmVnV5XXPF+1V2oUZ2EFIY4HOwXAIQbQxxpBbh/0HB fCrAYGbBXohYAdvoNLBy3kcWn38UO3vjRCJA X-Google-Smtp-Source: ABdhPJzbZ0w+ZD0BIsz0147xbzE61SyIK5APgWZ++R5IUUZhtnnk0hmMAlxmgEOUT3OhKaquGjxwPQ== X-Received: by 2002:a05:6638:531:: with SMTP id j17mr457998jar.69.1604029629919; Thu, 29 Oct 2020 20:47:09 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id n28sm4858693ila.52.2020.10.29.20.47.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Oct 2020 20:47:09 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: patchwork@lists.ozlabs.org, Doug Anderson , Tom Rini , Simon Glass Subject: [PATCH v3 28/29] patman: Support parsing of review snippets Date: Thu, 29 Oct 2020 21:46:37 -0600 Message-Id: <20201030034638.2858999-29-sjg@chromium.org> X-Mailer: git-send-email 2.29.1.341.ge80a0c044ae-goog In-Reply-To: <20201030034638.2858999-1-sjg@chromium.org> References: <20201030034638.2858999-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Add support for parsing the contents of a patchwork 'patch' web page containing comments received from reviewers. This allows patman to show these comments in a simple 'snippets' format. A snippet is some quoted code plus some unquoted comments below it. Each review is from a unique person/email and can produce multiple snippets, one for each part of the code that attracts a comment. Show the file and line-number info at the top of each snippet if available. Signed-off-by: Simon Glass Signed-off-by: Simon Glass --- (no changes since v1) tools/patman/func_test.py | 83 ++++++++++++++++++++++++++++++++++++ tools/patman/patchstream.py | 85 +++++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+) diff --git a/tools/patman/func_test.py b/tools/patman/func_test.py index 2e1529525eb..bbee4b77d66 100644 --- a/tools/patman/func_test.py +++ b/tools/patman/func_test.py @@ -1043,3 +1043,86 @@ diff --git a/lib/efi_loader/efi_memory.c b/lib/efi_loader/efi_memory.c self.assertEqual('Reviewed-by: %s' % self.fred, next(lines)) self.assertEqual('Reviewed-by: %s' % self.mary, next(lines)) self.assertEqual('Tested-by: %s' % self.leb, next(lines)) + + def testParseSnippets(self): + """Test parsing of review snippets""" + text = '''Hi Fred, + +This is a comment from someone. + +Something else + +On some recent date, Fred wrote: +> This is why I wrote the patch +> so here it is + +Now a comment about the commit message +A little more to say + +Even more + +> diff --git a/file.c b/file.c +> Some more code +> Code line 2 +> Code line 3 +> Code line 4 +> Code line 5 +> Code line 6 +> Code line 7 +> Code line 8 +> Code line 9 + +And another comment + +> @@ -153,8 +143,13 @@ def CheckPatch(fname, show_types=False): +> further down on the file +> and more code +> +Addition here +> +Another addition here +> codey +> more codey + +and another thing in same file + +> @@ -253,8 +243,13 @@ +> with no function context + +one more thing + +> diff --git a/tools/patman/main.py b/tools/patman/main.py +> +line of code +now a very long comment in a different file +line2 +line3 +line4 +line5 +line6 +line7 +line8 +''' + pstrm = PatchStream.process_text(text, True) + self.assertEqual([], pstrm.commit.warn) + + # We expect to the filename and up to 5 lines of code context before + # each comment. The 'On xxx wrote:' bit should be removed. + self.assertEqual( + [['Hi Fred,', + 'This is a comment from someone.', + 'Something else'], + ['> This is why I wrote the patch', + '> so here it is', + 'Now a comment about the commit message', + 'A little more to say', 'Even more'], + ['> File: file.c', '> Code line 5', '> Code line 6', + '> Code line 7', '> Code line 8', '> Code line 9', + 'And another comment'], + ['> File: file.c', + '> Line: 153 / 143: def CheckPatch(fname, show_types=False):', + '> and more code', '> +Addition here', '> +Another addition here', + '> codey', '> more codey', 'and another thing in same file'], + ['> File: file.c', '> Line: 253 / 243', + '> with no function context', 'one more thing'], + ['> File: tools/patman/main.py', '> +line of code', + 'now a very long comment in a different file', + 'line2', 'line3', 'line4', 'line5', 'line6', 'line7', 'line8']], + pstrm.snippets) diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py index 7b0805e9e14..9dc3b29216d 100644 --- a/tools/patman/patchstream.py +++ b/tools/patman/patchstream.py @@ -4,11 +4,13 @@ """Handles parsing a stream of commits/emails from 'git log' or other source""" +import collections import datetime import io import math import os import re +import queue import shutil import tempfile @@ -51,6 +53,12 @@ RE_SPACE_BEFORE_TAB = re.compile('^[+].* \t') # Match indented lines for changes RE_LEADING_WHITESPACE = re.compile(r'^\s') +# Detect a 'diff' line +RE_DIFF = re.compile(r'^>.*diff --git a/(.*) b/(.*)$') + +# Detect a context line, like '> @@ -153,8 +153,13 @@ CheckPatch +RE_LINE = re.compile(r'>.*@@ \-(\d+),\d+ \+(\d+),\d+ @@ *(.*)') + # States we can be in - can we use range() and still have comments? STATE_MSG_HEADER = 0 # Still in the message header STATE_PATCH_SUBJECT = 1 # In patch subject (first line of log for a commit) @@ -81,6 +89,14 @@ class PatchStream: self.blank_count = 0 # Number of blank lines stored up self.state = STATE_MSG_HEADER # What state are we in? self.commit = None # Current commit + self.snippets = [] # List of unquoted test blocks + self.cur_diff = None # Last 'diff' line seen (str) + self.cur_line = None # Last context (@@) line seen (str) + self.recent_diff= None # 'diff' line for current snippet (str) + self.recent_line= None # '@@' line for current snippet (str) + self.recent_quoted = collections.deque([], 5) + self.recent_unquoted = queue.Queue() + self.was_quoted = None @staticmethod def process_text(text, is_comment=False): @@ -176,6 +192,10 @@ class PatchStream: self.skip_blank = True self.section = [] + self.cur_diff = None + self.recent_diff = None + self.recent_line = None + def _parse_version(self, value, line): """Parse a version from a *-changes tag @@ -209,6 +229,47 @@ class PatchStream: self.commit.AddChange(self.change_version, change) self.change_lines = [] + def _finalise_snippet(self): + """Finish off a snippet and add it to the list + + This is called when we get to the end of a snippet, i.e. the we enter + the next block of quoted text: + + This is a comment from someone. + + Something else + + > Now we have some code <----- end of snippet + > more code + + Now a comment about the above code + + This adds the snippet to our list + """ + quoted_lines = [] + while self.recent_quoted: + quoted_lines.append(self.recent_quoted.popleft()) + unquoted_lines = [] + valid = False + while not self.recent_unquoted.empty(): + text = self.recent_unquoted.get() + if not (text.startswith('On ') and text.endswith('wrote:')): + unquoted_lines.append(text) + if text: + valid = True + if valid: + lines = [] + if self.recent_diff: + lines.append('> File: %s' % self.recent_diff) + if self.recent_line: + out = '> Line: %s / %s' % self.recent_line[:2] + if self.recent_line[2]: + out += ': %s' % self.recent_line[2] + lines.append(out) + lines += quoted_lines + unquoted_lines + if lines: + self.snippets.append(lines) + def process_line(self, line): """Process a single line of a patch file or commit log @@ -254,6 +315,8 @@ class PatchStream: cover_match = RE_COVER.match(line) signoff_match = RE_SIGNOFF.match(line) leading_whitespace_match = RE_LEADING_WHITESPACE.match(line) + diff_match = RE_DIFF.match(line) + line_match = RE_LINE.match(line) tag_match = None if self.state == STATE_PATCH_HEADER: tag_match = RE_TAG.match(line) @@ -443,6 +506,27 @@ class PatchStream: out = [line] self.linenum += 1 self.skip_blank = False + + if diff_match: + self.cur_diff = diff_match.group(1) + + # If this is quoted, keep recent lines + if not diff_match and self.linenum > 1 and line: + if line.startswith('>'): + if not self.was_quoted: + self._finalise_snippet() + self.recent_line = None + if not line_match: + self.recent_quoted.append(line) + self.was_quoted = True + self.recent_diff = self.cur_diff + else: + self.recent_unquoted.put(line) + self.was_quoted = False + + if line_match: + self.recent_line = line_match.groups() + if self.state == STATE_DIFFS: pass @@ -466,6 +550,7 @@ class PatchStream: def finalise(self): """Close out processing of this patch stream""" + self._finalise_snippet() self._finalise_change() self._close_commit() if self.lines_after_test: