From patchwork Wed Feb 21 14:17:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 876143 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zmgSV6t3rz9s5R for ; Thu, 22 Feb 2018 01:52:26 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GXojF290"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zmgSV4nPwzDrDd for ; Thu, 22 Feb 2018 01:52:26 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GXojF290"; dkim-atps=neutral X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:400e:c01::234; helo=mail-pl0-x234.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GXojF290"; dkim-atps=neutral Received: from mail-pl0-x234.google.com (mail-pl0-x234.google.com [IPv6:2607:f8b0:400e:c01::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zmfhn696pzF0kQ for ; Thu, 22 Feb 2018 01:18:00 +1100 (AEDT) Received: by mail-pl0-x234.google.com with SMTP id d4so969955pll.5 for ; Wed, 21 Feb 2018 06:18:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=jXWUMKGM6kNcTGsEeArWik48piOkgN2/VXIsqRHlkms=; b=GXojF290QPsNmf3QvQmRB8hy+1GqE6DMuQ54pH8ccHCArtmtVvlDHbcIpCZ7Dezh4q XSzHj/7jU6FKlP5UdDL5c6zXy3l7lTp2GDshaljq7EaG88jH2mlvi9Q3Y24Ld3GtHBFi JSAoUZBICP/Zj/S/cGCViVGGnbIqAC+5tQKcg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=jXWUMKGM6kNcTGsEeArWik48piOkgN2/VXIsqRHlkms=; b=nYJXjgjarACsx6bl8NIXA+OPBm3Miy8JEl59XxAN1xZCy/GOPRAaUKOzwW8L7IfxEd Cotn5g9wK8voe6eBVtPbJLbYTsXFeIGeaDa7tsbChMcYRLIT/NYskLOlPAZWhfsx1Okd tPNm2+RPcl/b9c+ApnshVUqxU0X7IsXFBuEgbVrmPBm1oTvfcoIlg+zNI2iTpaFNP6Nd T1wYrJiekhXWHjT1T9R/tvJNmzl3lwx5BYKxwS+nBybZUV8Mna+NMSJdEtbZFkLZzSh8 bhzKRZimPpWIr1f9xvMgYLkzqqGVvS+y7w72l0oIm5kJf+S8nl7Gx9tBszvXLnyd/HYh UB3Q== X-Gm-Message-State: APf1xPA6by5E6B46VzcatO+qVjHsVXEKrVjaSetgyh+PPO74HNLbGXDC 7ryMQyeHPr89f6zWjh6LyP+TOAewddA= X-Google-Smtp-Source: AH8x227OMhsKTBATpYwPk2hRYSULojPiTRgJtlUoJ4lmZ4SAGGe2afFfF/Caw7bNcViNCwLOJqK1vA== X-Received: by 2002:a17:902:b081:: with SMTP id p1-v6mr3259734plr.314.1519222678798; Wed, 21 Feb 2018 06:17:58 -0800 (PST) Received: from linkitivity.iinet.net.au (124-171-212-101.dyn.iinet.net.au. [124.171.212.101]) by smtp.gmail.com with ESMTPSA id w63sm14102667pgb.80.2018.02.21.06.17.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 21 Feb 2018 06:17:58 -0800 (PST) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 8/9] parser: use Patch.objects.create instead of save() Date: Thu, 22 Feb 2018 01:17:15 +1100 Message-Id: <20180221141716.10908-9-dja@axtens.net> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180221141716.10908-1-dja@axtens.net> References: <20180221141716.10908-1-dja@axtens.net> X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Andrew Donnellan MIME-Version: 1.0 Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Attempts to do parallel parsing with MySQL threw the following errors: _mysql_exceptions.OperationalError: (1213, 'Deadlock found when trying to get lock; try restarting transaction') Looking at the code, it was thrown when we created a patch like this: patch = Patch(...) patch.save() The SQL statements that were being generated were weird: UPDATE "patchwork_patch" SET ... INSERT INTO "patchwork_patch" (...) VALUES (...) As far as I can tell, the update could never work, because it was trying to update a patch that didn't exist yet. My hypothesis is that Django somehow didn't quite 'get' that because of the backend complexity of the Patch model, so it tried to do an update, failed, and then tried an insert. Change the code to use Patch.objects.create, which makes the UPDATEs and the weird MySQL errors go away. Also move it up a bit earlier in the process so that if things go wrong later at least we've committed the patch to the db. Signed-off-by: Daniel Axtens Reviewed-by: Andrew Donnellan --- patchwork/parser.py | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 3d40b74375e0..0e53e6b9a3af 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -984,6 +984,20 @@ def parse_mail(mail, list_id=None): filenames = find_filenames(diff) delegate = find_delegate_by_filename(project, filenames) + patch = Patch.objects.create( + msgid=msgid, + project=project, + name=name[:255], + date=date, + headers=headers, + submitter=author, + content=message, + diff=diff, + pull_url=pull_url, + delegate=delegate, + state=find_state(mail)) + logger.debug('Patch saved') + # if we don't have a series marker, we will never have an existing # series to match against. series = None @@ -1024,21 +1038,6 @@ def parse_mail(mail, list_id=None): except SeriesReference.DoesNotExist: SeriesReference.objects.create(series=series, msgid=ref) - patch = Patch( - msgid=msgid, - project=project, - name=name[:255], - date=date, - headers=headers, - submitter=author, - content=message, - diff=diff, - pull_url=pull_url, - delegate=delegate, - state=find_state(mail)) - patch.save() - logger.debug('Patch saved') - # add to a series if we have found one, and we have a numbered # patch. Don't add unnumbered patches (for example diffs sent # in reply, or just messages with random refs/in-reply-tos)