From patchwork Sat Feb 24 14:50:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 877430 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zpWQG0T60z9sWD for ; Sun, 25 Feb 2018 01:56:54 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="IhwM7wwh"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zpWQF3rZszF0xM for ; Sun, 25 Feb 2018 01:56:53 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="IhwM7wwh"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c01::235; helo=mail-pl0-x235.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="IhwM7wwh"; dkim-atps=neutral Received: from mail-pl0-x235.google.com (mail-pl0-x235.google.com [IPv6:2607:f8b0:400e:c01::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zpWHZ0GXfzF1lP for ; Sun, 25 Feb 2018 01:51:05 +1100 (AEDT) Received: by mail-pl0-x235.google.com with SMTP id u13so6668035plq.1 for ; Sat, 24 Feb 2018 06:51:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=H8+zA/pUA00ZG42+H3caZjjSaZvu8FA5pjUhQ1W3gcY=; b=IhwM7wwh/kLb/CLrE+pafF7DGhlK1GaF0q+KAkMJhsU87k+WlVjgH+I2BKJ+X6nMHE HKXHU2kN2ssiIhNov1XLmawmYuceTRfRbzoOHw38ZMCP1d+YZeQT5+LTim41840+uBm4 /bO+5KZt2WGnQRZRRbbtgo/715kayXpDeb5qc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=H8+zA/pUA00ZG42+H3caZjjSaZvu8FA5pjUhQ1W3gcY=; b=YUJETH3HyLjYBMY26HNVnynhqidJc0DMq8pi35WPfQPolHZRZc0+kwNv8dAM5GtJYT WADhoWzSPWwlUk8AURB8hBKM4r3XjtAbmrv1Aw8OlVqkfox0CCmB0f9Wd7k8Gk7aWapj aVmX10/QdU0xSNBLASrrkMkWYK8hxL9+cHoCxhdoO4Hn12QkRTWifbzOzs9oBSVpR7ue B76rjGPSpojBsGPbnyoCz4ZrOaWErXxm5B8eCAcqFVdkiiN2vUKdv5ENSoQf3Kb4vbrS 3AV4R829sTKT2TdW4kCPhX2VYRCZh3mfnhqo58p1/EhXx9hsILbvFMj9hYxM4+swwXzS 7gew== X-Gm-Message-State: APf1xPCe+Q1isL0vNyhQ0+/iMzEQAlPK25Gk5vlFLKveo2sDWGTPSX+l UV5GRNeLnK2opXyQSxKSCn9TT+145vc= X-Google-Smtp-Source: AH8x224iyZwmviU6MOUeAHiS2e6CND2Xo1TjFHpudfHu+vg/tDMmxmw/OEhzL7TvCd0keNz/R8+Vyw== X-Received: by 2002:a17:902:518d:: with SMTP id y13-v6mr5161600plh.121.1519483863093; Sat, 24 Feb 2018 06:51:03 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id p63sm10106360pfk.74.2018.02.24.06.51.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 24 Feb 2018 06:51:02 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH v2 9/9] parser: don't fail on multiple SeriesReferences Date: Sun, 25 Feb 2018 01:50:20 +1100 Message-Id: <20180224145020.15181-10-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180224145020.15181-1-dja@axtens.net> References: <20180224145020.15181-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Parallel parsing would occasonally fail with: patchwork.models.MultipleObjectsReturned: get() returned more than one SeriesReference -- it returned 2! I think these are happening if you have different processes parsing e.g. 1/3 and 2/3 simultaneously: both will have a reference to 1/3, in the case of 1 it will be the msgid, in the case of 2 it will be in References. So when we come to parse 3/3, .get() finds 2 and throws the exception. This does not fix the creation of multiple series references; it just causes them to be ignored. We still have serious race conditions with series creation, but I don't yet have clear answers for them. With this patch, they will at least not stop patches from being processed - they'll just lead to wonky series, which we already have. Reviewed-by: Andrew Donnellan Signed-off-by: Daniel Axtens --- patchwork/parser.py | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 56dc7006c811..5a7344cee93c 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -240,6 +240,13 @@ def _find_series_by_references(project, mail): msgid=ref[:255], series__project=project).series except SeriesReference.DoesNotExist: continue + except SeriesReference.MultipleObjectsReturned: + # FIXME: Open bug: this can happen when we're processing + # messages in parallel. Pick the first and log. + logger.error("Multiple SeriesReferences for %s in project %s!" % + (ref[:255], project.name)) + return SeriesReference.objects.filter( + msgid=ref[:255], series__project=project).first().series def _find_series_by_markers(project, mail, author): @@ -1037,6 +1044,9 @@ def parse_mail(mail, list_id=None): series__project=project) except SeriesReference.DoesNotExist: SeriesReference.objects.create(series=series, msgid=ref) + except SeriesReference.MultipleObjectsReturned: + logger.error("Multiple SeriesReferences for %s" + " in project %s!" % (ref, project.name)) # add to a series if we have found one, and we have a numbered # patch. Don't add unnumbered patches (for example diffs sent @@ -1075,6 +1085,11 @@ def parse_mail(mail, list_id=None): msgid=msgid, series__project=project).series except SeriesReference.DoesNotExist: series = None + except SeriesReference.MultipleObjectsReturned: + logger.error("Multiple SeriesReferences for %s" + " in project %s!" % (msgid, project.name)) + series = SeriesReference.objects.filter( + msgid=msgid, series__project=project).first().series if not series: series = Series(project=project, @@ -1087,8 +1102,12 @@ def parse_mail(mail, list_id=None): # we don't save the in-reply-to or references fields # for a cover letter, as they can't refer to the same # series - SeriesReference.objects.get_or_create(series=series, - msgid=msgid) + try: + SeriesReference.objects.get_or_create(series=series, + msgid=msgid) + except SeriesReference.MultipleObjectsReturned: + logger.error("Multiple SeriesReferences for %s" + " in project %s!" % (msgid, project.name)) cover_letter = CoverLetter( msgid=msgid,