From patchwork Mon Oct 26 01:04:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Simon Glass X-Patchwork-Id: 1387391 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.a=rsa-sha256 header.s=google header.b=JzH5Khwx; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CKGwh6P38z9sVS for ; Mon, 26 Oct 2020 12:10:28 +1100 (AEDT) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id C10608255C; Mon, 26 Oct 2020 02:06:15 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=chromium.org header.i=@chromium.org header.b="JzH5Khwx"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 48387824EE; Mon, 26 Oct 2020 02:05:35 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-2.0 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,SPF_HELO_NONE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.2 Received: from mail-il1-x12d.google.com (mail-il1-x12d.google.com [IPv6:2607:f8b0:4864:20::12d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 664E9824A8 for ; Mon, 26 Oct 2020 02:05:29 +0100 (CET) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=chromium.org Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=sjg@chromium.org Received: by mail-il1-x12d.google.com with SMTP id q1so6752733ilt.6 for ; Sun, 25 Oct 2020 18:05:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SXZ5g81lpeVc1vCdVhsmoyEwY2ywwamflm68E4FR6F8=; b=JzH5KhwxKMfcYtX6z0yT0u03hYmWNqpF0OpO/PKbxElDu4LErVfoR+C+vQxN0GgvHY Y2i9xDpgCG86SmRSwtz+qsvmJki+nRrqzK6Zh7M84dnnasBuO3UmgIXqi/F4VZ+YuXqO aNr4NmCvKFdn2F1OhWn33pYs2LpdaGuqUdhAw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SXZ5g81lpeVc1vCdVhsmoyEwY2ywwamflm68E4FR6F8=; b=j6Ur55wxcEEc7JDI24Dzp8W///MSw8AxboP0ovd8Cx0hOmA0EPDIthuI7MlTzymK98 pBinLNTH6y7bSUin8+OH504o/78wcMpOJ3+ZcgOGnFmGhXefUKKnzIoLuZxkg5ar3mmT gMquj7xoYYcSLa37nTgQbDsuT51qttcxaFlEZZU2zQlvAs9DzMRQWp3EXIZsuRwnmcEI gCv3FFKODtV9n9VS+1stjoG/4OKU2TIb5rqrNFn0wN1rltgySi1HNNeiPW9HAhCMTaDE aniDEgBt41c+OCEB9Lfg315Aj8Z/Dzr/ohkl5GHSMVW6q0IZVSLmDrXEVxzLW4T0Y5nk f7AQ== X-Gm-Message-State: AOAM533OiaAs4ApGJbbxcnwUWAirFrPaBcpxWn8EHqYjstjt+83U87Ve 69AeEkxTQ6oWZMS2egQj0tEkepzbuYzpwNTt X-Google-Smtp-Source: ABdhPJxlpDV0fp3l6EKW1R/4RzMOu9Ffn46gefkRFNO6tOMH7gb/9kh5aiSSHeX2DqyRveu2Y8Wh0A== X-Received: by 2002:a05:6e02:c:: with SMTP id h12mr9415507ilr.177.1603674328012; Sun, 25 Oct 2020 18:05:28 -0700 (PDT) Received: from localhost.localdomain (c-73-14-175-90.hsd1.co.comcast.net. [73.14.175.90]) by smtp.gmail.com with ESMTPSA id a6sm5231162ili.11.2020.10.25.18.05.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 25 Oct 2020 18:05:27 -0700 (PDT) From: Simon Glass To: U-Boot Mailing List Cc: Tom Rini , Doug Anderson , patchwork@lists.ozlabs.org, Simon Glass Subject: [PATCH v2 29/30] patman: Support parsing of review snippets Date: Sun, 25 Oct 2020 19:04:41 -0600 Message-Id: <20201026010442.1606893-30-sjg@chromium.org> X-Mailer: git-send-email 2.29.0.rc2.309.g374f81d7ae-goog In-Reply-To: <20201026010442.1606893-1-sjg@chromium.org> References: <20201026010442.1606893-1-sjg@chromium.org> MIME-Version: 1.0 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.34 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.102.3 at phobos.denx.de X-Virus-Status: Clean Add support for parsing the contents of a patchwork 'patch' web page containing comments received from reviewers. This allows patman to show these comments in a simple 'snippets' format. A snippet is some quoted code plus some unquoted comments below it. Each review is from a unique person/email and can produce multiple snippets, one for each part of the code that attracts a comment. Show the file and line-number info at the top of each snippet if available. Signed-off-by: Simon Glass --- (no changes since v1) tools/patman/func_test.py | 83 ++++++++++++++++++++++++++++++++++++ tools/patman/patchstream.py | 84 +++++++++++++++++++++++++++++++++++++ 2 files changed, 167 insertions(+) diff --git a/tools/patman/func_test.py b/tools/patman/func_test.py index 873cb31202b..0ddd908de1f 100644 --- a/tools/patman/func_test.py +++ b/tools/patman/func_test.py @@ -1040,3 +1040,86 @@ Series-changes: 2 {'Tested-by': {self.leb}, 'Reviewed-by': {self.fred, self.mary}}, new_series.commits[1].rtags) + + def testParseSnippets(self): + """Test parsing of review snippets""" + text = '''Hi Fred, + +This is a comment from someone. + +Something else + +On some recent date, Fred wrote: +> This is why I wrote the patch +> so here it is + +Now a comment about the commit message +A little more to say + +Even more + +> diff --git a/file.c b/file.c +> Some more code +> Code line 2 +> Code line 3 +> Code line 4 +> Code line 5 +> Code line 6 +> Code line 7 +> Code line 8 +> Code line 9 + +And another comment + +> @@ -153,8 +143,13 @@ def CheckPatch(fname, show_types=False): +> further down on the file +> and more code +> +Addition here +> +Another addition here +> codey +> more codey + +and another thing in same file + +> @@ -253,8 +243,13 @@ +> with no function context + +one more thing + +> diff --git a/tools/patman/main.py b/tools/patman/main.py +> +line of code +now a very long comment in a different file +line2 +line3 +line4 +line5 +line6 +line7 +line8 +''' + pstrm = PatchStream.process_text(text, True) + self.assertEqual([], pstrm.commit.warn) + + # We expect to the filename and up to 5 lines of code context before + # each comment. The 'On xxx wrote:' bit should be removed. + self.assertEqual( + [['Hi Fred,', + 'This is a comment from someone.', + 'Something else'], + ['> This is why I wrote the patch', + '> so here it is', + 'Now a comment about the commit message', + 'A little more to say', 'Even more'], + ['> File: file.c', '> Code line 5', '> Code line 6', + '> Code line 7', '> Code line 8', '> Code line 9', + 'And another comment'], + ['> File: file.c', + '> Line: 153 / 143: def CheckPatch(fname, show_types=False):', + '> and more code', '> +Addition here', '> +Another addition here', + '> codey', '> more codey', 'and another thing in same file'], + ['> File: file.c', '> Line: 253 / 243', + '> with no function context', 'one more thing'], + ['> File: tools/patman/main.py', '> +line of code', + 'now a very long comment in a different file', + 'line2', 'line3', 'line4', 'line5', 'line6', 'line7', 'line8']], + pstrm.snippets) diff --git a/tools/patman/patchstream.py b/tools/patman/patchstream.py index b88a546b863..0b344614c70 100644 --- a/tools/patman/patchstream.py +++ b/tools/patman/patchstream.py @@ -4,11 +4,13 @@ """Handles parsing a stream of commits/emails from 'git log' or other source""" +import collections import datetime import io import math import os import re +import queue import shutil import tempfile @@ -51,6 +53,12 @@ RE_SPACE_BEFORE_TAB = re.compile('^[+].* \t') # Match indented lines for changes RE_LEADING_WHITESPACE = re.compile(r'^\s') +# Detect a 'diff' line +RE_DIFF = re.compile(r'^>.*diff --git a/(.*) b/(.*)$') + +# Detect a context line, like '> @@ -153,8 +153,13 @@ CheckPatch +RE_LINE = re.compile(r'>.*@@ \-(\d+),\d+ \+(\d+),\d+ @@ *(.*)') + # States we can be in - can we use range() and still have comments? STATE_MSG_HEADER = 0 # Still in the message header STATE_PATCH_SUBJECT = 1 # In patch subject (first line of log for a commit) @@ -81,6 +89,14 @@ class PatchStream: self.blank_count = 0 # Number of blank lines stored up self.state = STATE_MSG_HEADER # What state are we in? self.commit = None # Current commit + self.snippets = [] # List of unquoted test blocks + self.cur_diff = None # Last 'diff' line seen (str) + self.cur_line = None # Last context (@@) line seen (str) + self.recent_diff= None # 'diff' line for current snippet (str) + self.recent_line= None # '@@' line for current snippet (str) + self.recent_quoted = collections.deque([], 5) + self.recent_unquoted = queue.Queue() + self.was_quoted = None self.saw_signoff = False # Found sign-off line for this commit self.saw_change_id = False # Found Change-Id for this commit @@ -180,6 +196,9 @@ class PatchStream: self.saw_signoff = False self.saw_change_id = False + self.cur_diff = None + self.recent_diff = None + self.recent_line = None def _parse_version(self, value, line): """Parse a version from a *-changes tag @@ -214,6 +233,47 @@ class PatchStream: self.commit.AddChange(self.change_version, change) self.change_lines = [] + def _finalise_snippet(self): + """Finish off a snippet and add it to the list + + This is called when we get to the end of a snippet, i.e. the we enter + the next block of quoted text: + + This is a comment from someone. + + Something else + + > Now we have some code <----- end of snippet + > more code + + Now a comment about the above code + + This adds the snippet to our list + """ + quoted_lines = [] + while self.recent_quoted: + quoted_lines.append(self.recent_quoted.popleft()) + unquoted_lines = [] + valid = False + while not self.recent_unquoted.empty(): + text = self.recent_unquoted.get() + if not (text.startswith('On ') and text.endswith('wrote:')): + unquoted_lines.append(text) + if text: + valid = True + if valid: + lines = [] + if self.recent_diff: + lines.append('> File: %s' % self.recent_diff) + if self.recent_line: + out = '> Line: %s / %s' % self.recent_line[:2] + if self.recent_line[2]: + out += ': %s' % self.recent_line[2] + lines.append(out) + lines += quoted_lines + unquoted_lines + if lines: + self.snippets.append(lines) + def process_line(self, line): """Process a single line of a patch file or commit log @@ -259,6 +319,8 @@ class PatchStream: cover_match = RE_COVER.match(line) signoff_match = RE_SIGNOFF.match(line) leading_whitespace_match = RE_LEADING_WHITESPACE.match(line) + diff_match = RE_DIFF.match(line) + line_match = RE_LINE.match(line) tag_match = None if self.state == STATE_PATCH_HEADER: tag_match = RE_TAG.match(line) @@ -462,6 +524,27 @@ class PatchStream: out = [line] self.linenum += 1 self.skip_blank = False + + if diff_match: + self.cur_diff = diff_match.group(1) + + # If this is quoted, keep recent lines + if not diff_match and self.linenum > 1 and line: + if line.startswith('>'): + if not self.was_quoted: + self._finalise_snippet() + self.recent_line = None + if not line_match: + self.recent_quoted.append(line) + self.was_quoted = True + self.recent_diff = self.cur_diff + else: + self.recent_unquoted.put(line) + self.was_quoted = False + + if line_match: + self.recent_line = line_match.groups() + if self.state == STATE_DIFFS: pass @@ -485,6 +568,7 @@ class PatchStream: def finalise(self): """Close out processing of this patch stream""" + self._finalise_snippet() self._finalise_change() self._close_commit() if self.lines_after_test: