From patchwork Wed Feb 21 14:17:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876136 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgNR4h9Tz9rxp for ; Thu, 22 Feb 2018 01:48:55 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="QCIkT1xN"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgNQ4Y0NzDr6q for ; Thu, 22 Feb 2018 01:48:54 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="QCIkT1xN"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c00::22f; helo=mail-pf0-x22f.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="QCIkT1xN"; dkim-atps=neutral Received: from mail-pf0-x22f.google.com (mail-pf0-x22f.google.com [IPv6:2607:f8b0:400e:c00::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhL4C8dzDrLQ for ; Thu, 22 Feb 2018 01:17:38 +1100 (AEDT) Received: by mail-pf0-x22f.google.com with SMTP id c143so697978pfb.6 for ; Wed, 21 Feb 2018 06:17:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YYNoqlrXxF8Y24qeh+FVvpnyYnNo8opVT4NnFPW4IvM=; b=QCIkT1xNZmXnnvd032NU8u97t2GbFoCvT22L0UM5mdRhkpx0L8B/B94uG//3Z1BDnW hJDeqHq9Co7txj2MIk4m/BPixGdVHjoIEeqjJDcqcU+W27XuWfTMW0YaaKing/QSBH7N KbGtKdNCXS60jWAxiwohvkTQMO611EJg+vcAo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YYNoqlrXxF8Y24qeh+FVvpnyYnNo8opVT4NnFPW4IvM=; b=K3RItbUWgmsh49K9pUdhMGPcTdR9Rkc0WeLe9bSNIqbLdZtXVIWWitvNvVBDid5/OF aOb4fxkm0ux4UmqYIWheuKlqvT7A4XhJ+JlmqPL66fAulWcabeqsOqfAU3Uw+T9i5hiy JkVdckYksAy/OoJ6h0RJLZb4vUvgm6ojJkoqPceBoTnUvGQwPwuRXNUhniBE6qhEQosJ LIeB05JUmG0Urr5+utzSLnDIWcSEG4BhDhGvqzDnFuQXzdvn3F9XUItkEI6nLFz38RUv ppE1Ec7uKCkAthTumv17ajvs3kWUAPrwH24yRsJdN7y78HZ/BAqQE8rcanSpilHBGRse UlWg== X-Gm-Message-State: APf1xPAXYoWLXIRm9D8LBLR69d9uaWGPZT/Zf67Ctb4CSU+aThUM1Cgp 0CfBdfEnXNrL5lpmNTVltZVFr5klkqQ= X-Google-Smtp-Source: AH8x227cYfCl+1W3DQzVLNvgiBCg4FCdhTIGvUFgKD3nug0kGtVDobVlsugBWG2kPjDcV+z5BdrO9w== X-Received: by 10.101.101.83 with SMTP id a19mr2912372pgw.192.1519222656606; Wed, 21 Feb 2018 06:17:36 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:35 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 1/9] tools/docker: assume terminal supports utf-8 Date: Thu, 22 Feb 2018 01:17:08 +1100 Message-Id: <20180221141716.10908-2-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Set PYTHONIOENCODING to UTF-8, which allows Python3 to print UTF-8 directly to the terminal (to a pipe or shell-redirected file) rather than throwing an error. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- tools/docker/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index eb6d35d82528..2154ca7e67bf 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -10,6 +10,7 @@ ENV db_pass password ENV DJANGO_SETTINGS_MODULE patchwork.settings.dev ENV DEBIAN_FRONTEND noninteractive ENV PYTHONUNBUFFERED 1 +ENV PYTHONIOENCODING UTF-8 # System # trusty and findutils is for python3.4 From patchwork Wed Feb 21 14:17:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876137 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgPK6Q0rz9s5R for ; Thu, 22 Feb 2018 01:49:41 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="JiVfqqhH"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgPK3sjLzF0nD for ; Thu, 22 Feb 2018 01:49:41 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="JiVfqqhH"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c00::242; helo=mail-pf0-x242.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="JiVfqqhH"; dkim-atps=neutral Received: from mail-pf0-x242.google.com (mail-pf0-x242.google.com [IPv6:2607:f8b0:400e:c00::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhP6CpxzDrLQ for ; Thu, 22 Feb 2018 01:17:41 +1100 (AEDT) Received: by mail-pf0-x242.google.com with SMTP id c143so698030pfb.6 for ; Wed, 21 Feb 2018 06:17:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=B2iaaMfivhSpOWBmhYFQ+aAxgl2l6bMdGTZPxHAz3i4=; b=JiVfqqhHP1mL0R1SRxrSRqCCet46avBlsmxyb8rqcrkG9WZ/pNKVscky6c2QAbXs4E /3FfKO1qfOu39Cc1otMVsGZ9YQrvcXlGfvY33xsDi+UHpOx6q+kv3SaqPqHSBAJVntIF zMYDPSAkjeuS60+khsHaOgmWF5XMRR63CD3vs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=B2iaaMfivhSpOWBmhYFQ+aAxgl2l6bMdGTZPxHAz3i4=; b=thnCNKAMS5UN2r7uBqF1HmaYZX79NI9fs+cR9GkBgRVnTvG8tb7d4adKvZDRWT9JWz vMy+h3et5mswkNN8Yh3vBCDU6Mn8y3283Nrfeb6f2J9OxCqLkVZT4JGNjzIHYCq3Zoae 540dqsfoABZKuMZLGrZNwK4aMnDUsd8tfhID5yDARodNwgJ/RZXTBi8htOqXjUEDbe0e 3GHyp6pwR1tsbJqKCVTJMwBLXtFjKS+MlgnOKy9CpmZhb1n+9Il/V64SYKzd6nkNBmP2 ui/+K3/94oCikZzizt6QCI5j8HmjE6Gc5g0vLtMch2NnOYhgEvok7V0GreH6NskBUs6m gAEQ== X-Gm-Message-State: APf1xPAEZ5aAQoV/CjCcUg6InbYZvEpyy2xPrNIeHCT1ITraZm0uKgNU crDvKZn9Zw83g3i5j39iSDCpockZeyk= X-Google-Smtp-Source: AH8x226bUFHbrFBDh53jD8eijuDNgOrKSjfHXrw3MwVR7DbdO8aReHnOK1Xe7llZX92ZjPBJzFwLrQ== X-Received: by 10.101.93.82 with SMTP id e18mr2818960pgt.371.1519222659497; Wed, 21 Feb 2018 06:17:39 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:38 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 2/9] debugging: add command to dump patches and series Date: Thu, 22 Feb 2018 01:17:09 +1100 Message-Id: <20180221141716.10908-3-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" I don't want a full dump, just enough to know if the same patches and series have been created with roughly the same properties. This seemed like the easiest way to do it. Usage: python3 manage.py debug_dump > file ... make changes, reset db, reload files ... python3 manage.py debug_dump > file2 diff -u file file2 Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/management/commands/debug_dump.py | 46 +++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 patchwork/management/commands/debug_dump.py diff --git a/patchwork/management/commands/debug_dump.py b/patchwork/management/commands/debug_dump.py new file mode 100644 index 000000000000..f668dd14f1d2 --- /dev/null +++ b/patchwork/management/commands/debug_dump.py @@ -0,0 +1,46 @@ +# Patchwork - automated patch tracking system +# Copyright (C) 2018 Daniel Axtens +# +# This file is part of the Patchwork package. +# +# Patchwork is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# Patchwork is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +from django.core.management import base + +from patchwork.models import Patch +from patchwork.models import Series + + +class Command(base.BaseCommand): + help = 'DEBUG COMMAND: Return a minimal robust representation of the db.' + + def handle(self, *args, **options): + """This is to check the invariance of parsing as messages are + reordered or received in parallel.""" + + series = [] + for s in Series.objects.all(): + series += ['%s :: v%d :: %d patches :: %s' + % (s.name, s.version, s.total, s.submitter.email)] + + series.sort() + print('=== %d series ===' % len(series)) + for s in series: + print(s) + + patches = [] + for p in Patch.objects.all(): + patches += ['%s :: ID %s' % (p.name, p.msgid)] + + patches.sort() + print('=== %d patches ===' % len(patches)) + for p in patches: + print(p) From patchwork Wed Feb 21 14:17:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876138 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgPz44wwz9rxp for ; Thu, 22 Feb 2018 01:50:15 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="jiNMI5Bn"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgPz2jLSzF0q1 for ; Thu, 22 Feb 2018 01:50:15 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="jiNMI5Bn"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c05::243; helo=mail-pg0-x243.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="jiNMI5Bn"; dkim-atps=neutral Received: from mail-pg0-x243.google.com (mail-pg0-x243.google.com [IPv6:2607:f8b0:400e:c05::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhS4l0JzDrTR for ; Thu, 22 Feb 2018 01:17:44 +1100 (AEDT) Received: by mail-pg0-x243.google.com with SMTP id l4so673506pgp.11 for ; Wed, 21 Feb 2018 06:17:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EOJrm/jntT59FNNM17g783NVq9eejsbmOk6LvdOF5gg=; b=jiNMI5BnxMbbIcX4kOvTPGAYMRVHOLPM1iAb/wnTCR5QtccUKWvs46dzx3f/ER85Yf HnrCUdLdH87vKPBSL8kBWsdaMhnHpWSO5cACvuJQQtWRhCQW1a9E/wh1VL24nnDaLA3P 0vzu2B/LEsH4qnHCI6OYAE04Ytg/b7MdjgQkQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EOJrm/jntT59FNNM17g783NVq9eejsbmOk6LvdOF5gg=; b=MsBfCW/UrZfWaaBlD16e32DGi8S9Rj89E8/qrabpVdTR9oACLDbjY9QiwjNsIKihGW 4ItRiKwOenSJWHea6cLLWEguJzdG7E8RB6JVAhgO+e09tEFpbTTjdr06QM9xU/kiqHtO 9tBmtxPkHPgCWEbbLVlJoAP5XLPx/7Lp+/ljqFC1stNdThynzoZmc+PdFvU/Q4FsvNpZ 06ZLfGdoQVjib7fsrUsHXbzBgK7EXinePnY0MpbYNBT9iXimF6362epfULGaKDjoCpwb mAIfW2qo0vIGCnoHWM7eMAk3lwFcWILqcAf2zgwkVwz9XwwJvxhk7VndE4ZMnyN8pqwn N7Rw== X-Gm-Message-State: APf1xPBP3vm8umuSH0tg0Oatl9HqUGf0zmUBPqN/aqmA5Qa668I6zaLo AImE4sFJ3wKJHIIg2clrkD1/TKgsoYY= X-Google-Smtp-Source: AH8x226fmq9zXNdi9kp3gynIuwmCuSuUlZwB1TgnLVNgX71hyi6LFSAQuWDEhfJWNustCQxgmnRegg== X-Received: by 10.98.161.7 with SMTP id b7mr3495040pff.68.1519222662558; Wed, 21 Feb 2018 06:17:42 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:41 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 3/9] tools/scripts: split a mbox N ways Date: Thu, 22 Feb 2018 01:17:10 +1100 Message-Id: <20180221141716.10908-4-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" To test parallel loading of mail, it's handy to be able to split an existing mbox file into N mbox files in an alternating pattern (e.g. 1 2 1 2 or 1 2 3 4 1 2 3 4 etc) Introduce tools/scripts as a place to put things like this. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- tools/scripts/split_mail.py | 76 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 tools/scripts/split_mail.py diff --git a/tools/scripts/split_mail.py b/tools/scripts/split_mail.py new file mode 100755 index 000000000000..ce71fe16c362 --- /dev/null +++ b/tools/scripts/split_mail.py @@ -0,0 +1,76 @@ +#!/usr/bin/python3 +# Patchwork - automated patch tracking system +# Copyright (C) 2018 Daniel Axtens +# +# This file is part of the Patchwork package. +# +# Patchwork is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# Patchwork is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +import sys +import os +import mailbox + +usage = """Split a maildir or mbox into N mboxes +in an alternating pattern + +Usage: ./split_mail.py + + : input mbox file or Maildir + : output mbox + -1... must not exist + N-way split""" + + +in_name = sys.argv[1] +out_name = sys.argv[2] + +try: + n = int(sys.argv[3]) +except: + print("N must be an integer.") + print(" ") + print(usage) + exit(1) + +if n < 2: + print("N must be be at least 2") + print(" ") + print(usage) + exit(1) + +if not os.path.exists(in_name): + print("No input at ", in_name) + print(" ") + print(usage) + exit(1) + +print("Opening", in_name) +if os.path.isdir(in_name): + inmail = mailbox.Maildir(in_name) +else: + inmail = mailbox.mbox(in_name) + +out=[] +for i in range(n): + if os.path.exists(out_name+"-"+str(i+1)): + print("mbox already exists at ", out_name+"-"+str(i+1)) + print(" ") + print(usage) + exit(1) + + out += [mailbox.mbox(out_name+'-'+str(i+1))] + +print("Copying messages") + +for (i, msg) in enumerate(inmail): + out[i % n].add(msg) + +print("Done") From patchwork Wed Feb 21 14:17:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876139 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgQT6prDz9s5R for ; Thu, 22 Feb 2018 01:50:41 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="TeXrtOos"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgQT5MGRzDqZY for ; Thu, 22 Feb 2018 01:50:41 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="TeXrtOos"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c00::241; helo=mail-pf0-x241.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="TeXrtOos"; dkim-atps=neutral Received: from mail-pf0-x241.google.com (mail-pf0-x241.google.com [IPv6:2607:f8b0:400e:c00::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhW5R7pzDqgR for ; Thu, 22 Feb 2018 01:17:47 +1100 (AEDT) Received: by mail-pf0-x241.google.com with SMTP id 17so692117pfw.11 for ; Wed, 21 Feb 2018 06:17:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=SYs24C87l44sFl0aHgB1fvJv35BYeV82RR9VEisFsoo=; b=TeXrtOos8hNW5ik5K46k8Men69QZRqw5UaXkZoHZLsCdZiL+mn6ZaM9cFrQbKX0sEO nmHwPccZE5ofnLA7ntR/5sE6d07KOI+zJu2KxWG+H1w1dKXmM77hy1vF/f52zCZjFYrM Nvzpj752n3I7Ack6AiWYgXMMZW16gBx49LThU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=SYs24C87l44sFl0aHgB1fvJv35BYeV82RR9VEisFsoo=; b=SDJwyYXpmeM13YTMJAU1nBxbD+WsAwYD9YcS7T2iBW5b1wLnP+/7C0nYZo719zmIg1 OfPlpRXg9uk0lfVCNsLZY+o2kWCGF5hEI2IyCbJPM76pU2UbHTrMqE4o8U5q/m5iVcJ0 a/u69gCGASrr8bG7nS7QKA49Ri01dkq3uMw9VokHjki1eyGuSsfxAVogoW2ap7De8Ou7 aXXNvhIiZZp0sLiPV2nELIffIpQf8lh4mpEAcxSl7zABW5Tj0dZAe+4Bm2AL9lE5g2Vg OMleCeBLjXaDnMTn+hsGcuxvJWVQHw6vXLlxsq5vVv5kNsF6+RWbnHxwAbI+obXON938 or3g== X-Gm-Message-State: APf1xPBzHJj0UC18v/ZQ19WDRcxrBygZacgTr8PaOAK4adaQwNUFEv06 erWBCo8X8bzpbNJB/k7z7pqEvqXL+OA= X-Google-Smtp-Source: AH8x225pN8GqKABhZ8mkiyVwJRYPsqhiK95X2c7AEsd7EVI+gqode0+t9j+8nxc2c25SFVgRCDUDiQ== X-Received: by 10.98.63.147 with SMTP id z19mr3410260pfj.221.1519222665582; Wed, 21 Feb 2018 06:17:45 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:45 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 4/9] tools/scripts: parallel_parsearchive - load archives in parallel Date: Thu, 22 Feb 2018 01:17:11 +1100 Message-Id: <20180221141716.10908-5-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" If you have multiple archives, you quickly tire of typing stuff like python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-1 & python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-2 & python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-3 & python3 manage.py parsearchive --list-id=patchwork.ozlabs.org foo-4 & and having to copy and paste it - or retype it! - each time you reset the database. Instead, this patch allows you to do tools/scripts/parallel_parsearchive.sh --list-id=patchwork.ozlabs.org -- foo-* Much easier, especially when you are doing it a dozen times. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- tools/scripts/parallel_parsearchive.sh | 55 ++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100755 tools/scripts/parallel_parsearchive.sh diff --git a/tools/scripts/parallel_parsearchive.sh b/tools/scripts/parallel_parsearchive.sh new file mode 100755 index 000000000000..a5197f6c2c5f --- /dev/null +++ b/tools/scripts/parallel_parsearchive.sh @@ -0,0 +1,55 @@ +#!/bin/bash +# Patchwork - automated patch tracking system +# Copyright (C) 2018 Daniel Axtens +# +# This file is part of the Patchwork package. +# +# Patchwork is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# Patchwork is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +set -euo pipefail + +usage() { + cat < + The -- is mandatory. + As many processes as there are archives will be spun up. + +Example: +EOF + exit 1 +} + +if [ $# -eq 0 ] || [[ $1 == "-h" ]]; then + usage; +fi + +PARSEARCHIVE_OPTIONS="" +while [[ $1 != "--" ]]; do + PARSEARCHIVE_OPTIONS="$PARSEARCHIVE_OPTIONS $1" + shift + if [ $# -eq 0 ]; then + usage; + fi +done +shift + +if [ $# -eq 0 ]; then + usage; +fi + + +for x in "$@"; do + echo "Starting $x" + python3 manage.py parsearchive $PARSEARCHIVE_OPTIONS "$x" & +done +echo "Processes started in the background." From patchwork Wed Feb 21 14:17:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876140 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgRH6nNbz9rxp for ; Thu, 22 Feb 2018 01:51:23 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="d+z9t+XM"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgRH4yNBzF0fc for ; Thu, 22 Feb 2018 01:51:23 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="d+z9t+XM"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c00::231; helo=mail-pf0-x231.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="d+z9t+XM"; dkim-atps=neutral Received: from mail-pf0-x231.google.com (mail-pf0-x231.google.com [IPv6:2607:f8b0:400e:c00::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhZ6HZMzDrTR for ; Thu, 22 Feb 2018 01:17:50 +1100 (AEDT) Received: by mail-pf0-x231.google.com with SMTP id z24so691882pfh.13 for ; Wed, 21 Feb 2018 06:17:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=rTVYXRniVdj5fveVJgevT4+oIRKwrc4JPm6npETQISU=; b=d+z9t+XM3tNPXU84mqbb8K7p64pxaDBgwynqzzXB5cZb3cjABMgbrpPTcyVdGGxmdZ Syg9VOVap4bYhO7SPsJQRxO6qP2kDFa+DNTD3/iznaYnS/KPYrHd0ixxuiZD6xwySOxm 7kJ563m2g4Cek3zZQcQQoEGdqiItPS0Os/ZJM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=rTVYXRniVdj5fveVJgevT4+oIRKwrc4JPm6npETQISU=; b=pcxiQczv/h+Pp6bBZfpIpBiuGDWeTeVk40Kd379S81HNoxGuiM37zlpq4dXLbSQomJ 8IjwM0k+SI8QYTdvuE5zahYivXGWsdwNMxQjg3P78kbMUeoHOx4t8gtTXQAxGt9jJa9K dEfor0z42341U5iRCuF/x4sE8ekOgVnY3JJ94mWfnTDiQOupORY+QaIE4X+ZFpe+kdYi dNKSAu0tF+bKag/WMbXnfCQwn6U3UtLRrlJTsoSu+EBdMWQw5GOqY+fpMBdvizA06+Mz tLwNaL2HDcdSu8LBqjy8/PkLhob/DOMXJwXAybng/W7WuuWETnbbdvsHrtcVfFYyCtcM fOhw== X-Gm-Message-State: APf1xPDSk/zHj1vhIhW+4dwzq6lDWtheUAAD47BUpIe61WrTUjq/AiWr GXkVRrxZDPafViEDDm0whDEAeuuFdNY= X-Google-Smtp-Source: AH8x227cvQ6kv8qn9JEwX0w/ZFfQGtjdz/egtFE0bsfQCRPimgQAGLsHNSMXyR3Wins4elQdZGFsJg== X-Received: by 10.98.16.147 with SMTP id 19mr2800237pfq.84.1519222668765; Wed, 21 Feb 2018 06:17:48 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:48 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 5/9] parser: Handle even more exotically broken headers Date: Thu, 22 Feb 2018 01:17:12 +1100 Message-Id: <20180221141716.10908-6-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" An archive of the Ubuntu kernel team mailing list contains a fascinating email that causes the following parse error: email.errors.HeaderParseError: header value appears to contain an embedded header: '4Mf^tnii7k\\_EnR5aobBm6Di[DZ9@AX1wJ"okBdX-UoJ>:SRn]c6DDU"qUIwfs98vF>... The broken bit seem related to a UTF-8 quoted-printable encoded section and to be from an internal attempt to break it over multiple lines: here's a snippet from the error message: '\n\t=?utf-8?q?Tnf?=\n' but interesting the header itself does not contain the new lines, so clearly something quite weird is happening behind the scenes! This only throws on header.encode(): it actually makes it through sanitise_header and into find_headers before throwing the assertion. So, try to encode in sanitize_header as a final step. Also, fix a hilarious* python bug that this exposes: whitespace-only headers cause an index error! Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/parser.py | 8 +++++ patchwork/tests/fuzztests/x-face.mbox | 58 +++++++++++++++++++++++++++++++++++ patchwork/tests/test_parser.py | 7 +++-- 3 files changed, 71 insertions(+), 2 deletions(-) create mode 100644 patchwork/tests/fuzztests/x-face.mbox diff --git a/patchwork/parser.py b/patchwork/parser.py index 2cabb3cbc299..cbf88fe4e464 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -129,6 +129,14 @@ def sanitise_header(header_contents, header_name=None): header_name=header_name, continuation_ws='\t') + try: + header.encode() + except (HeaderParseError, IndexError): + # despite our best efforts, the header is stuffed + # HeaderParseError: some very weird multi-line headers + # IndexError: bug, thrown by make_header(decode_header(' ')).encode() + return None + return header diff --git a/patchwork/tests/fuzztests/x-face.mbox b/patchwork/tests/fuzztests/x-face.mbox new file mode 100644 index 000000000000..98019cff8250 --- /dev/null +++ b/patchwork/tests/fuzztests/x-face.mbox @@ -0,0 +1,58 @@ +From laurent.pinchart@skynet.be Thu Nov 13 15:54:10 2008 +Received: from mailrelay005.isp.belgacom.be ([195.238.6.171]) + by chlorine.canonical.com with esmtp (Exim 4.60) + (envelope-from ) id 1L0eWI-0007oB-7K + for kernel-team@lists.ubuntu.com; Thu, 13 Nov 2008 15:54:10 +0000 +X-IronPort-Anti-Spam-Filtered: true +X-IronPort-Anti-Spam-Result: ApsEAP/aG0nCTsYx/2dsb2JhbACBds9Hg1c +Received: from 49.198-78-194.adsl-static.isp.belgacom.be (HELO + laptop-laurent.belgium.cse-semaphore.com) ([194.78.198.49]) + by relay.skynet.be with ESMTP; 13 Nov 2008 16:54:09 +0100 +From: Laurent Pinchart +To: kernel-team@lists.ubuntu.com +Subject: uvcvideo (webcam) support for COMPAL JHL90 based laptops +Date: Thu, 13 Nov 2008 16:54:22 +0100 +User-Agent: KMail/1.9.9 +X-Face: 4Mf^tnii7k\_EnR5aobBm6Di[DZ9@AX1wJ"okBdX-UoJ>:SRn]c6DDU"qUIwfs98vF>=?utf-8?q?Tnf=0A=09SacR=7B?=(0Du"N%_.#X]"TXx)A'gKB1i7SK$CTLuy{h})c=g:'w3 +MIME-Version: 1.0 +Content-Type: text/plain; + charset="us-ascii" +Content-Transfer-Encoding: 7bit +Content-Disposition: inline +Message-Id: <200811131654.22389.laurent.pinchart@skynet.be> +X-Mailman-Approved-At: Fri, 14 Nov 2008 14:54:06 +0000 +Cc: amit@ubuntu.com +X-BeenThere: kernel-team@lists.ubuntu.com +X-Mailman-Version: 2.1.8 +Precedence: list +List-Id: Kernel team discussions +List-Unsubscribe: , + +List-Archive: +List-Post: +List-Help: +List-Subscribe: , + +X-List-Received-Date: Thu, 13 Nov 2008 15:54:10 -0000 + +Hi Amit, + +I've noticed by sheer luck that the Ubuntu 8.10 linux-image-2.6.27-7 packages +include a patch to the uvcvideo driver to support webcam modules integrated +into Compal JHL90 laptops. + +Is there a reason why the patch hasn't been pushed upstream ? Being the +uvcvideo author and maintainer, I'd appreciate if you could forward patches +related to the driver in the future. + +On a pure technical note, the patch might not be required with the current +uvcvideo driver version. There is no way to confirm this without testing the +driver with that particular webcam model, so I'd appreciate if you could +contact the patch author and check with him if his camera works with the +latest uvcvideo driver (available from http://linux-uvc.berlios.de/). + +Best regards, + +Laurent Pinchart + + diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py index 20d70af12120..68bcb937b273 100644 --- a/patchwork/tests/test_parser.py +++ b/patchwork/tests/test_parser.py @@ -896,8 +896,8 @@ class SubjectTest(TestCase): self.assertEqual(parse_version('Hello, world (V6)', []), 6) -class FuzzTest(TransactionTestCase): - """Test fuzzed patches.""" +class WeirdMailTest(TransactionTestCase): + """Test fuzzed or otherwise weird patches.""" def setUp(self): create_project(listid='patchwork.ozlabs.org') @@ -940,3 +940,6 @@ class FuzzTest(TransactionTestCase): self._test_patch('refshdr.mbox') self._test_patch('dateheader.mbox') self._test_patch('msgidheader.mbox') + + def test_x_face(self): + self._test_patch('x-face.mbox') From patchwork Wed Feb 21 14:17:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876141 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgRk1zTzz9rxp for ; Thu, 22 Feb 2018 01:51:46 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="TdZ9KQhx"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgRk0YDqzF0ph for ; Thu, 22 Feb 2018 01:51:46 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="TdZ9KQhx"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c00::241; helo=mail-pf0-x241.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="TdZ9KQhx"; dkim-atps=neutral Received: from mail-pf0-x241.google.com (mail-pf0-x241.google.com [IPv6:2607:f8b0:400e:c00::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhd559DzDqpP for ; Thu, 22 Feb 2018 01:17:53 +1100 (AEDT) Received: by mail-pf0-x241.google.com with SMTP id y186so17591pfb.2 for ; Wed, 21 Feb 2018 06:17:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Deeq+CzHuzIxlzxFE/MuIy6bEps1YnkfNahe2blc8Jg=; b=TdZ9KQhxXHe9JRQRNf6MBTErtDCV7fe9T3NmO7JAd973sSf+BRmjD/v8hSnt7YzixQ bUFmw8rQPmet/38Jb3S2P49GIG+3Hwl81a3LzcOmUwzVpA2nKV16nayBbU6w9YQf9E2o MDtc3r26r9Hhkgg3p1cfHB0BPpCFi7NHR/aAs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Deeq+CzHuzIxlzxFE/MuIy6bEps1YnkfNahe2blc8Jg=; b=oAx0de+RUa4ow6MY0uNkKg8FkCnIHdvEJNb58aZ2AhdCX4zN1ZCABBO4v21a3YLl04 CPfdNuA00oKcd/AJUvdmSZtt8sSmwMRqFqTtP/GsAjYe+EpdUHztb+OdSVcuokwcbOiH yb6cX+ACyon1DDIk2FrPYIPXpsZcl9araC1yYZVjotIhtk6HX9sBR6BqxRJ4DqoTFVVu m8zvUokGGHbvGZdgIZAkVHr9dhzuChD3BoEuMkHRm99NhflEb9MAoyo9G/pVFoi81839 D7BwGU4XXu9sQdBTZx+BPz1CY9ry5ebBFdGB7lkuQf8n881Qv7cJwsMfCZkfrinNLlw5 y4cg== X-Gm-Message-State: APf1xPCiqSCHRsW8S08DUPo8oP1N4KwOC1+wnMow7COX9bdb60yXdKYV aEDuEe6iBXUgT4t6qNA9MPCX764rlw8= X-Google-Smtp-Source: AH8x224OO+P+JKoXqyvUuptapUR5B7TvOSKS0dtm0buWEZ47X5xEzNQJtrtzC4tvGhgkr+z2G95wfA== X-Received: by 10.98.31.79 with SMTP id f76mr3514268pff.60.1519222671695; Wed, 21 Feb 2018 06:17:51 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:51 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 6/9] parser: close a TOCTTOU bug on Person creation Date: Thu, 22 Feb 2018 01:17:13 +1100 Message-Id: <20180221141716.10908-7-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" find_author looks up a person by email, and if they do not exist, creates a Person model, which may be saved later if the message contains something valuable. Multiple simultaneous processes can race here: both can do the SELECT, find there is no Person, and create the model. One will succeed in saving, the other will get an IntegrityError. Reduce the window by making find_author into get_or_create_author, and plumb that through. (Remove a test that specifically required find_author to *not* create). More importantly, cover the case where we lose the race, by catching the IntegrityError and fetching the winning Person model. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/parser.py | 32 ++++++++++++++++++-------------- patchwork/tests/test_parser.py | 37 +++++++++++++++---------------------- 2 files changed, 33 insertions(+), 36 deletions(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index cbf88fe4e464..1f3b3dd8901f 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -29,6 +29,7 @@ import logging import re from django.contrib.auth.models import User +from django.db import IntegrityError from django.utils import six from patchwork.models import Comment @@ -241,7 +242,7 @@ def _find_series_by_references(project, mail): continue -def _find_series_by_markers(project, mail): +def _find_series_by_markers(project, mail, author): """Find a patch's series using series markers and sender. Identify suitable series for a patch using a combination of the @@ -262,7 +263,6 @@ def _find_series_by_markers(project, mail): still won't help us if someone spams the mailing list with duplicate series but that's a tricky situation for anyone to parse. """ - author = find_author(mail) subject = mail.get('Subject') name, prefixes = clean_subject(subject, [project.linkname]) @@ -282,7 +282,7 @@ def _find_series_by_markers(project, mail): return -def find_series(project, mail): +def find_series(project, mail, author): """Find a series, if any, for a given patch. Args: @@ -297,10 +297,10 @@ def find_series(project, mail): if series: return series - return _find_series_by_markers(project, mail) + return _find_series_by_markers(project, mail, author) -def find_author(mail): +def get_or_create_author(mail): from_header = clean_header(mail.get('From')) if not from_header: @@ -342,11 +342,16 @@ def find_author(mail): name = name.strip()[:255] try: + person = Person.objects.get_or_create(email__iexact=email, + defaults={'name': name, + 'email': email})[0] + except IntegrityError: + # we lost the race to create the person person = Person.objects.get(email__iexact=email) - if name: # use the latest provided name - person.name = name - except Person.DoesNotExist: - person = Person(name=name, email=email) + + if name: # use the latest provided name + person.name = name + person.save() return person @@ -947,7 +952,6 @@ def parse_mail(mail, list_id=None): raise ValueError("Broken 'Message-Id' header") msgid = msgid[:255] - author = find_author(mail) subject = mail.get('Subject') name, prefixes = clean_subject(subject, [project.linkname]) is_comment = subject_check(subject) @@ -973,7 +977,7 @@ def parse_mail(mail, list_id=None): if not is_comment and (diff or pull_url): # patches or pull requests # we delay the saving until we know we have a patch. - author.save() + author = get_or_create_author(mail) delegate = find_delegate_by_header(mail) if not delegate and diff: @@ -984,7 +988,7 @@ def parse_mail(mail, list_id=None): # series to match against. series = None if n: - series = find_series(project, mail) + series = find_series(project, mail, author) else: x = n = 1 @@ -1061,7 +1065,7 @@ def parse_mail(mail, list_id=None): is_cover_letter = True if is_cover_letter: - author.save() + author = get_or_create_author(mail) # we don't use 'find_series' here as a cover letter will # always be the first item in a thread, thus the references @@ -1109,7 +1113,7 @@ def parse_mail(mail, list_id=None): if not submission: return - author.save() + author = get_or_create_author(mail) comment = Comment( submission=submission, diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py index 68bcb937b273..6cfe7a484443 100644 --- a/patchwork/tests/test_parser.py +++ b/patchwork/tests/test_parser.py @@ -34,7 +34,7 @@ from patchwork.models import Patch from patchwork.models import Person from patchwork.models import State from patchwork.parser import clean_subject -from patchwork.parser import find_author +from patchwork.parser import get_or_create_author from patchwork.parser import find_patch_content as find_content from patchwork.parser import find_project_by_header from patchwork.parser import find_series @@ -225,7 +225,7 @@ class SenderEncodingTest(TestCase): def _test_encoding(self, from_header, sender_name, sender_email): email = self._create_email(from_header) - person = find_author(email) + person = get_or_create_author(email) person.save() # ensure it was parsed correctly @@ -241,7 +241,7 @@ class SenderEncodingTest(TestCase): def test_empty(self): email = self._create_email('') with self.assertRaises(ValueError): - find_author(email) + get_or_create_author(email) def test_ascii_encoding(self): from_header = 'example user ' @@ -269,7 +269,7 @@ class SenderEncodingTest(TestCase): class SenderCorrelationTest(TestCase): - """Validate correct behavior of the find_author case. + """Validate correct behavior of the get_or_create_author case. Relies of checking the internal state of a Django model object. @@ -284,25 +284,16 @@ class SenderCorrelationTest(TestCase): 'test\n' return message_from_string(mail) - def test_non_existing_sender(self): - sender = 'Non-existing Sender ' - mail = self._create_email(sender) - - # don't create the person - attempt to find immediately - person = find_author(mail) - self.assertEqual(person._state.adding, True) - self.assertEqual(person.id, None) - def test_existing_sender(self): sender = 'Existing Sender ' mail = self._create_email(sender) # create the person first - person_a = find_author(mail) + person_a = get_or_create_author(mail) person_a.save() # then attempt to parse email with the same 'From' line - person_b = find_author(mail) + person_b = get_or_create_author(mail) self.assertEqual(person_b._state.adding, False) self.assertEqual(person_b.id, person_a.id) @@ -311,12 +302,12 @@ class SenderCorrelationTest(TestCase): mail = self._create_email(sender) # create the person first - person_a = find_author(mail) + person_a = get_or_create_author(mail) person_a.save() # then attempt to parse email with a new 'From' line mail = self._create_email('existing@example.com') - person_b = find_author(mail) + person_b = get_or_create_author(mail) self.assertEqual(person_b._state.adding, False) self.assertEqual(person_b.id, person_a.id) @@ -324,11 +315,11 @@ class SenderCorrelationTest(TestCase): sender = 'Existing Sender ' mail = self._create_email(sender) - person_a = find_author(mail) + person_a = get_or_create_author(mail) person_a.save() mail = self._create_email(sender.upper()) - person_b = find_author(mail) + person_b = get_or_create_author(mail) self.assertEqual(person_b._state.adding, False) self.assertEqual(person_b.id, person_a.id) @@ -361,7 +352,8 @@ class SeriesCorrelationTest(TestCase): email = self._create_email(msgid) project = create_project() - self.assertIsNone(find_series(project, email)) + self.assertIsNone(find_series(project, email, + get_or_create_author(email))) def test_first_reply(self): msgid_a = make_msgid() @@ -371,7 +363,8 @@ class SeriesCorrelationTest(TestCase): # assume msgid_a was already handled ref = create_series_reference(msgid=msgid_a) - series = find_series(ref.series.project, email) + series = find_series(ref.series.project, email, + get_or_create_author(email)) self.assertEqual(series, ref.series) def test_nested_series(self): @@ -395,7 +388,7 @@ class SeriesCorrelationTest(TestCase): # ...and the "first patch" of this new series msgid = make_msgid() email = self._create_email(msgid, msgids) - series = find_series(project, email) + series = find_series(project, email, get_or_create_author(email)) # this should link to the second series - not the first self.assertEqual(len(msgids), 4 + 1) # old series + new cover From patchwork Wed Feb 21 14:17:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876142 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgS74JpPz9s5R for ; Thu, 22 Feb 2018 01:52:07 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="YInMVR+W"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgS72TSZzF0m1 for ; Thu, 22 Feb 2018 01:52:07 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="YInMVR+W"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c01::242; helo=mail-pl0-x242.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="YInMVR+W"; dkim-atps=neutral Received: from mail-pl0-x242.google.com (mail-pl0-x242.google.com [IPv6:2607:f8b0:400e:c01::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhj58gZzDqbZ for ; Thu, 22 Feb 2018 01:17:57 +1100 (AEDT) Received: by mail-pl0-x242.google.com with SMTP id s13so968057plq.6 for ; Wed, 21 Feb 2018 06:17:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vHp3z7QQSWCCl/hkFyODWLDRT172DROFYETE7VC+lTU=; b=YInMVR+WDg8FAMnE5Q9B5CGNTEZDpmgT8logSK4YQRl/iBjftoTzZd40agt8slIpxd CYfD7yhA1iYUSppPhzl60jrl0TeZeUL4UdTF0J35iNTQbVfg4ZSMZszi3SbJhx93XG4p 3Ygckrvzz9X8pcDE4GDxlyZBrVAYuNqayH4OM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vHp3z7QQSWCCl/hkFyODWLDRT172DROFYETE7VC+lTU=; b=MURpJz23LZpNNK9AX6A+wnszfkZSXfJL2a+Ypb3C21slVGB0/1tP6vFIG/bD862YZp Nb/1poJWrmJeLiSwkF/bEwi+3fnc08UoqjBsLQLooD3JGtumx49r/2y2jwTsgB9l/fFw g9HOidadgOhSkTgH5kVeMOc1FVHojwiOhjGMKGw0/6mXluHpSyFc7ajvLqvLGh10niTv mReFKkKCPqXlVDiOPWU0TtowCXAewLzmbbi8cdGh5Edbiq2IT4xBnQTK9i8drK8CiF75 dN0LReEbUqJO3V0HOEbxhm5Ofv3ewEhfVePahJCZt0d+/tu4dtqTnaQPdnH07k9lhso3 EHYw== X-Gm-Message-State: APf1xPAFjcg8rTl5dw5NzNwru/qLl/uuK/6fhIOk9iaFTmErTA1D1z0S izz6xGDNNDt0FzWKF/JgRJsmkD8Ya1k= X-Google-Smtp-Source: AH8x227ybaxj+mTgqPUhMEvOM4+8T6+PtG/jLmKbJpSu1r5xEEXPksvI3VIexKBf6jTUHsMGq+r11Q== X-Received: by 2002:a17:902:43e4:: with SMTP id j91-v6mr3239250pld.153.1519222675561; Wed, 21 Feb 2018 06:17:55 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:54 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 7/9] parser: avoid an unnecessary UPDATE of Person Date: Thu, 22 Feb 2018 01:17:14 +1100 Message-Id: <20180221141716.10908-8-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Analysis of SQL statements showed that when parsing an email, the row for the Person who sent the email was always getting updated. This is because the test for updating it only checks if the incoming mail has *a* name attached to the email address, and not if it has a new name. Django is not smart enough to figure that out, and so unconditionally UPDATEs the model when asked to save. Give it a hand - only update the model and save it if the new name is in fact different. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/parser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 1f3b3dd8901f..3d40b74375e0 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -349,7 +349,7 @@ def get_or_create_author(mail): # we lost the race to create the person person = Person.objects.get(email__iexact=email) - if name: # use the latest provided name + if name and name != person.name: # use the latest provided name person.name = name person.save() From patchwork Wed Feb 21 14:17:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876143 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgSV6t3rz9s5R for ; Thu, 22 Feb 2018 01:52:26 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GXojF290"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgSV4nPwzDrDd for ; Thu, 22 Feb 2018 01:52:26 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GXojF290"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c01::234; helo=mail-pl0-x234.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GXojF290"; dkim-atps=neutral Received: from mail-pl0-x234.google.com (mail-pl0-x234.google.com [IPv6:2607:f8b0:400e:c01::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhn696pzF0kQ for ; Thu, 22 Feb 2018 01:18:00 +1100 (AEDT) Received: by mail-pl0-x234.google.com with SMTP id d4so969955pll.5 for ; Wed, 21 Feb 2018 06:18:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jXWUMKGM6kNcTGsEeArWik48piOkgN2/VXIsqRHlkms=; b=GXojF290QPsNmf3QvQmRB8hy+1GqE6DMuQ54pH8ccHCArtmtVvlDHbcIpCZ7Dezh4q XSzHj/7jU6FKlP5UdDL5c6zXy3l7lTp2GDshaljq7EaG88jH2mlvi9Q3Y24Ld3GtHBFi JSAoUZBICP/Zj/S/cGCViVGGnbIqAC+5tQKcg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jXWUMKGM6kNcTGsEeArWik48piOkgN2/VXIsqRHlkms=; b=nYJXjgjarACsx6bl8NIXA+OPBm3Miy8JEl59XxAN1xZCy/GOPRAaUKOzwW8L7IfxEd Cotn5g9wK8voe6eBVtPbJLbYTsXFeIGeaDa7tsbChMcYRLIT/NYskLOlPAZWhfsx1Okd tPNm2+RPcl/b9c+ApnshVUqxU0X7IsXFBuEgbVrmPBm1oTvfcoIlg+zNI2iTpaFNP6Nd T1wYrJiekhXWHjT1T9R/tvJNmzl3lwx5BYKxwS+nBybZUV8Mna+NMSJdEtbZFkLZzSh8 bhzKRZimPpWIr1f9xvMgYLkzqqGVvS+y7w72l0oIm5kJf+S8nl7Gx9tBszvXLnyd/HYh UB3Q== X-Gm-Message-State: APf1xPA6by5E6B46VzcatO+qVjHsVXEKrVjaSetgyh+PPO74HNLbGXDC 7ryMQyeHPr89f6zWjh6LyP+TOAewddA= X-Google-Smtp-Source: AH8x227OMhsKTBATpYwPk2hRYSULojPiTRgJtlUoJ4lmZ4SAGGe2afFfF/Caw7bNcViNCwLOJqK1vA== X-Received: by 2002:a17:902:b081:: with SMTP id p1-v6mr3259734plr.314.1519222678798; Wed, 21 Feb 2018 06:17:58 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:58 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 8/9] parser: use Patch.objects.create instead of save() Date: Thu, 22 Feb 2018 01:17:15 +1100 Message-Id: <20180221141716.10908-9-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Attempts to do parallel parsing with MySQL threw the following errors: _mysql_exceptions.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction') Looking at the code, it was thrown when we created a patch like this: patch = Patch(...) patch.save() The SQL statements that were being generated were weird: UPDATE "patchwork_patch" SET ... INSERT INTO "patchwork_patch" (...) VALUES (...) As far as I can tell, the update could never work, because it was trying to update a patch that didn't exist yet. My hypothesis is that Django somehow didn't quite 'get' that because of the backend complexity of the Patch model, so it tried to do an update, failed, and then tried an insert. Change the code to use Patch.objects.create, which makes the UPDATEs and the weird MySQL errors go away. Also move it up a bit earlier in the process so that if things go wrong later at least we've committed the patch to the db. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/parser.py | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 3d40b74375e0..0e53e6b9a3af 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -984,6 +984,20 @@ def parse_mail(mail, list_id=None): filenames = find_filenames(diff) delegate = find_delegate_by_filename(project, filenames) + patch = Patch.objects.create( + msgid=msgid, + project=project, + name=name[:255], + date=date, + headers=headers, + submitter=author, + content=message, + diff=diff, + pull_url=pull_url, + delegate=delegate, + state=find_state(mail)) + logger.debug('Patch saved') + # if we don't have a series marker, we will never have an existing # series to match against. series = None @@ -1024,21 +1038,6 @@ def parse_mail(mail, list_id=None): except SeriesReference.DoesNotExist: SeriesReference.objects.create(series=series, msgid=ref) - patch = Patch( - msgid=msgid, - project=project, - name=name[:255], - date=date, - headers=headers, - submitter=author, - content=message, - diff=diff, - pull_url=pull_url, - delegate=delegate, - state=find_state(mail)) - patch.save() - logger.debug('Patch saved') - # add to a series if we have found one, and we have a numbered # patch. Don't add unnumbered patches (for example diffs sent # in reply, or just messages with random refs/in-reply-tos) From patchwork Wed Feb 21 14:17:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876144 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgSx44wmz9rxp for ; Thu, 22 Feb 2018 01:52:49 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="kf/bBFdl"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgSx23dPzDrDd for ; Thu, 22 Feb 2018 01:52:49 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="kf/bBFdl"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c01::22f; helo=mail-pl0-x22f.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="kf/bBFdl"; dkim-atps=neutral Received: from mail-pl0-x22f.google.com (mail-pl0-x22f.google.com [IPv6:2607:f8b0:400e:c01::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhr2LKZzDr25 for ; Thu, 22 Feb 2018 01:18:04 +1100 (AEDT) Received: by mail-pl0-x22f.google.com with SMTP id s13so968201plq.6 for ; Wed, 21 Feb 2018 06:18:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=H4V1kE4GbicpBBhwC24WtcoBdCc70s0369CtsYhreDc=; b=kf/bBFdl/TF3pVXsTFRob/3GO6aNmZP/GO9Ro4kiUbeiZQUErQCwe37qHSkLySgjSj Szcwc+RNAPZlgmYyk2EKAHrGvGlsWTpj8k9bZvx/uzwOUp93ORrc1oUdsVP59+QR0ddH y+zRHGOWX+jqheegQCYBDWMMb660rvrbug8MM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=H4V1kE4GbicpBBhwC24WtcoBdCc70s0369CtsYhreDc=; b=BF9L2EDJALvQOSukWOez0L1gjpdVpaZuc6XuXjSMk4kaIJDVNZPE3pcelcIGKdpGvm m7AZCPr+/ufUDOnFV1YswLUBL2ppZ4YMSU3XUtt4v6rtrGUQIfnS1o6OHvvRYFMGL+lp 9kQqPTmzrm3xNfRma/XKeH6PSmvezrYulHio23vQ3T+orXuBo5PTfmCXoL0wXqb54e4B OcKHyJv2aTuuMr2jMe3dclC7FG94lMsPKpne96EAWFZtodXQywMk2QD9xN5ZZBEwsMAZ csWYzDUQmgnI9mokzll4ikSWswRfMHvAhfuVZkMv0MUfiGGxnxL8hcUu8K/NGQcpbdOo 5ArQ== X-Gm-Message-State: APf1xPBBwejlkr4qlnhUFkNg/s9arV08IfGQUJcqFxlLPyp0JFXUCQ92 kkhvQ6X6vuiMprTUnRDefD3SpjnwhFk= X-Google-Smtp-Source: AH8x224Tp5WohZE3A2upbqcnUm/yqvPuf/uMUL+9pVx0WL3nHcI9PlKTmBLdZ3QE15pcwYWpHoGgoQ== X-Received: by 2002:a17:902:1a2:: with SMTP id b31-v6mr3176558plb.28.1519222681919; Wed, 21 Feb 2018 06:18:01 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:18:01 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 9/9] parser: don't fail on multiple SeriesReferences Date: Thu, 22 Feb 2018 01:17:16 +1100 Message-Id: <20180221141716.10908-10-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Parallel parsing would occasonally fail with: patchwork.models.MultipleObjectsReturned: get() returned more than one SeriesReference -- it returned 2! I think these are happening if you have different processes parsing e.g. 1/3 and 2/3 simultaneously: both will have a reference to 1/3, in the case of 1 it will be the msgid, in the case of 2 it will be in References. So when we come to parse 3/3, .get() finds 2 and throws the exception. This does not fix the creation of multiple series references; it just causes them to be ignored. We still have serious race conditions with series creation, but I don't yet have clear answers for them. With this patch, they will at least not stop patches from being processed - they'll just lead to wonky series, which we already have. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/parser.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 0e53e6b9a3af..4c9e636336d9 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -240,6 +240,13 @@ def _find_series_by_references(project, mail): msgid=ref[:255], series__project=project).series except SeriesReference.DoesNotExist: continue + except SeriesReference.MultipleObjectsReturned: + # Open bug: this can happen when we're processing messages + # in parallel. Pick the first and log. + logger.error("Multiple SeriesReferences for %s in project %s!" % + (ref[:255], project.name)) + return SeriesReference.objects.filter( + msgid=ref[:255], series__project=project).first().series def _find_series_by_markers(project, mail, author): @@ -1037,6 +1044,9 @@ def parse_mail(mail, list_id=None): series__project=project) except SeriesReference.DoesNotExist: SeriesReference.objects.create(series=series, msgid=ref) + except SeriesReference.MultipleObjectsReturned: + logger.error("Multiple SeriesReferences for %s" + " in project %s!" % (ref, project.name)) # add to a series if we have found one, and we have a numbered # patch. Don't add unnumbered patches (for example diffs sent @@ -1075,6 +1085,11 @@ def parse_mail(mail, list_id=None): msgid=msgid, series__project=project).series except SeriesReference.DoesNotExist: series = None + except SeriesReference.MultipleObjectsReturned: + logger.error("Multiple SeriesReferences for %s" + " in project %s!" % (msgid, project.name)) + series = SeriesReference.objects.filter( + msgid=msgid, series__project=project).first().series if not series: series = Series(project=project, @@ -1087,8 +1102,12 @@ def parse_mail(mail, list_id=None): # we don't save the in-reply-to or references fields # for a cover letter, as they can't refer to the same # series - SeriesReference.objects.get_or_create(series=series, - msgid=msgid) + try: + SeriesReference.objects.get_or_create(series=series, + msgid=msgid) + except SeriesReference.MultipleObjectsReturned: + logger.error("Multiple SeriesReferences for %s" + " in project %s!" % (msgid, project.name)) cover_letter = CoverLetter( msgid=msgid,