From patchwork Fri Feb 10 00:57:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dustin Lundquist X-Patchwork-Id: 1740226 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.openwrt.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=d0fkW1+w; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=null-ptr-net.20210112.gappssmtp.com header.i=@null-ptr-net.20210112.gappssmtp.com header.a=rsa-sha256 header.s=20210112 header.b=O40v97oP; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4PCb7c419rz23hX for ; Fri, 10 Feb 2023 12:02:49 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=PVpCdbBwRxMtjVZjsgJpTCM1ncC56ouotePPKz4ORLs=; b=d0fkW1+we5hYad 7s+LL19DiJgsDDYof7gBwj4FAH9rUxMYyn8DaEH7lOx6QgWjP8FIMeBHok30mqvW1xOAdxI3N7frt odZlNlRMmAC6BPDtWgsRrzLThN46nV3YFegDxWAoAox7+dTI6Z0AxkZrMziEND3Cq+I5AZhAJr6xx Cm5++EzarFW9HzIBzP9RTXLYudhCaYFEIJyQ8l8dtlLUrD7MeplJhJhjwK/P+5afnPW1OwcCj2pOr 1oEIawRBzK/EmYs1ifo6gyymgWQRTYwKXOkQ/h1nKYMSfd0f2ONB6VBZ7qpvHE5UQRqgQJ+r1Agq8 QrtdIhR63yBcrQeAgYEw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pQHkC-003mAh-I3; Fri, 10 Feb 2023 00:58:28 +0000 Received: from mail-pj1-x1029.google.com ([2607:f8b0:4864:20::1029]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pQHk1-003m9I-5Z for openwrt-devel@lists.openwrt.org; Fri, 10 Feb 2023 00:58:24 +0000 Received: by mail-pj1-x1029.google.com with SMTP id s89-20020a17090a2f6200b0023125ebb4b1so4014776pjd.3 for ; Thu, 09 Feb 2023 16:58:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=null-ptr-net.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ujI+Ny2dcAZ4OiJ6ayMytgFnGxWV5QjBdZufABNkU8U=; b=O40v97oP9M35vWN7WvAqTS+iiOMOnBZpRWwsgYrJHT8sUw5bBsEjKgptjMa3/BX+Zc e23rdjeacRs6klCIfhZMwQyW4yJA21+2tJcKmXsqwJGQwUFxLSorHAP7hHcMSE0cWnVl uj63JitJD+6vsbn522b2eA3uskwFCN+ME+AVO3/zWw5dnZLq0NRI3Qwj4xz6AU7reuBQ aFAWv93vywb4Z/WSSzMKVrwSZeotnQiXMk85Cpsr5ZGxrK1mMxT4CjS4BGRCdY1igJme TR4fhUI8s1ivoiLFFMz7b5iMcKC5shDsX+JtmXxIM3fKVJNLcE2RU+xPwC9H3wEf6bw7 6nIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ujI+Ny2dcAZ4OiJ6ayMytgFnGxWV5QjBdZufABNkU8U=; b=EUlN1kJFDCUGidNnDWExm6T8asMYP/bUpWgcahxbsfc0CkZbpdjS6wtMt/4W8BfquO qs9APEJyMinyw25xmoTxr2dD1PTn332BxvAGA5OI9eVX4ZqtQCZGbUCi5lzwvlB6nQH9 hfxTVrEG5cVuAiGdolvygXwMIi40MhFIotr2XthCxoOHILfhhd0Z8f/K7ifnywYlXqEG mVqw6zhg9h/d3rpz4bv/GdUx7/Lsuk6qwpQ2haeYQ7QA0mZROuhn0KO2ElSwU+aoXSre MNyju9MBB8wC+TDJTPElP3hStWPGPJtjPmxUEcYhobA2zZkVw5idkXztSxlGkd4+lGu5 z8RQ== X-Gm-Message-State: AO0yUKXBQSUz+vCW4M7E+2seagtYTGWEP1Puov/nGyBiz7yTc8cHYF01 8CtaoB4FY8iBaOf3Y7f9liYnAp2TWg3IfVKWQdg= X-Google-Smtp-Source: AK7set/hHUTWrx2gfHMz5NQfT1fl7wP2nBhUq9Eo/Bdn4+S0lg1a5/I18P+sjfsHz5U5oS+cCG2/eQ== X-Received: by 2002:a17:902:e484:b0:199:4ea:866e with SMTP id i4-20020a170902e48400b0019904ea866emr9731175ple.40.1675990691557; Thu, 09 Feb 2023 16:58:11 -0800 (PST) Received: from rigel.overthere.org (c-71-197-159-127.hsd1.wa.comcast.net. [71.197.159.127]) by smtp.gmail.com with ESMTPSA id d23-20020a170902b71700b001933b4b1a49sm1062596pls.183.2023.02.09.16.58.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Feb 2023 16:58:10 -0800 (PST) From: Dustin Lundquist To: openwrt-devel@lists.openwrt.org Cc: Dustin Lundquist Subject: [PATCH] scripts: avoid simultanious downloads from interfering Date: Thu, 9 Feb 2023 16:57:59 -0800 Message-Id: <20230210005759.1739135-1-dustin@null-ptr.net> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230209_165817_459132_CAD96F1F X-CRM114-Status: GOOD ( 10.69 ) X-Spam-Score: 0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: When you have multiple build roots sharing the download cache, simultaneous downloads of the same file can result download.pl failing with: Could not generate file hash Use File::Temp for the download and hash output files, and then move the download into place after the file hash is verified. Content analysis details: (0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:1029 listed in] [list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.0 SPF_NONE SPF: sender does not publish an SPF Record -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: OpenWrt Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org When you have multiple build roots sharing the download cache, simultaneous downloads of the same file can result download.pl failing with: Could not generate file hash Use File::Temp for the download and hash output files, and then move the download into place after the file hash is verified. Signed-off-by: Dustin Lundquist --- scripts/download.pl | 40 +++++++++++++++------------------------- 1 file changed, 15 insertions(+), 25 deletions(-) diff --git a/scripts/download.pl b/scripts/download.pl index 676c6e9e6b..f735ecc6d5 100755 --- a/scripts/download.pl +++ b/scripts/download.pl @@ -1,5 +1,5 @@ #!/usr/bin/env perl -# +# # Copyright (C) 2006 OpenWrt.org # Copyright (C) 2016 LEDE project # @@ -11,6 +11,7 @@ use strict; use warnings; use File::Basename; use File::Copy; +use File::Temp; use Text::ParseWords; @ARGV > 2 or die "Syntax: $0 [ ...]\n"; @@ -149,10 +150,12 @@ sub download $mirror =~ s!/$!!; + my $hash_tmp = File::Temp->new(); + my $dl_tmp = File::Temp->new(); + if ($mirror =~ s!^file://!!) { if (! -d "$mirror") { print STDERR "Wrong local cache directory -$mirror-.\n"; - cleanup(); return; } @@ -183,10 +186,10 @@ sub download } print("Copying $filename from $link\n"); - copy($link, "$target/$filename.dl"); + copy($link, $dl_tmp); $hash_cmd and do { - if (system("cat '$target/$filename.dl' | $hash_cmd > '$target/$filename.hash'")) { + if (system("cat '$dl_tmp' | $hash_cmd > '$hash_tmp'")) { print("Failed to generate hash for $filename\n"); return; } @@ -196,46 +199,34 @@ sub download print STDERR "+ ".join(" ",@cmd)."\n"; open(FETCH_FD, '-|', @cmd) or die "Cannot launch aria2c, curl or wget.\n"; $hash_cmd and do { - open MD5SUM, "| $hash_cmd > '$target/$filename.hash'" or die "Cannot launch $hash_cmd.\n"; + open MD5SUM, "| $hash_cmd > '$hash_tmp'" or die "Cannot launch $hash_cmd.\n"; }; - open OUTPUT, "> $target/$filename.dl" or die "Cannot create file $target/$filename.dl: $!\n"; my $buffer; while (read FETCH_FD, $buffer, 1048576) { $hash_cmd and print MD5SUM $buffer; - print OUTPUT $buffer; + print $dl_tmp $buffer; } $hash_cmd and close MD5SUM; close FETCH_FD; - close OUTPUT; if ($? >> 8) { print STDERR "Download failed.\n"; - cleanup(); return; } } $hash_cmd and do { - my $sum = `cat "$target/$filename.hash"`; + my $sum = `cat "$hash_tmp"`; $sum =~ /^(\w+)\s*/ or die "Could not generate file hash\n"; $sum = $1; if ($sum ne $file_hash) { print STDERR "Hash of the downloaded file does not match (file: $sum, requested: $file_hash) - deleting download.\n"; - cleanup(); return; } }; - unlink "$target/$filename"; - system("mv", "$target/$filename.dl", "$target/$filename"); - cleanup(); -} - -sub cleanup -{ - unlink "$target/$filename.dl"; - unlink "$target/$filename.hash"; + system("mv", "-f", "$dl_tmp", "$target/$filename"); } @mirrors = localmirrors(); @@ -332,19 +323,19 @@ push @mirrors, 'https://mirror2.openwrt.org/sources'; if (-f "$target/$filename") { $hash_cmd and do { - if (system("cat '$target/$filename' | $hash_cmd > '$target/$filename.hash'")) { + my $hash_tmp = File::Temp->new(); + + if (system("cat '$target/$filename' | $hash_cmd > '$hash_tmp'")) { die "Failed to generate hash for $filename\n"; } - my $sum = `cat "$target/$filename.hash"`; + my $sum = `cat "$hash_tmp"`; $sum =~ /^(\w+)\s*/ or die "Could not generate file hash\n"; $sum = $1; - cleanup(); exit 0 if $sum eq $file_hash; die "Hash of the local file $filename does not match (file: $sum, requested: $file_hash) - deleting download.\n"; - unlink "$target/$filename"; }; } @@ -360,4 +351,3 @@ while (!-f "$target/$filename") { } } -$SIG{INT} = \&cleanup;