From patchwork Fri Nov 20 22:00:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Petr_=C5=A0tetiar?= X-Patchwork-Id: 1404155 X-Patchwork-Delegate: ynezz@true.cz Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.openwrt.org (client-ip=2001:8b0:10b:1231::1; helo=merlin.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=true.cz Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=merlin.20170209 header.b=eSZrD1ob; dkim-atps=neutral Received: from merlin.infradead.org (merlin.infradead.org [IPv6:2001:8b0:10b:1231::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Cd9XD1kBsz9sVH for ; Sat, 21 Nov 2020 09:02:49 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Message-Id:Date:Subject:To: From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:References:List-Owner; bh=AOKYS3TwiMvk6zpW+EyJEPen/Yh1khuqkIegVij1JHs=; b=eSZrD1obhxkB/6NnY8hDNTAca 4nJyqtwKpzuhDU13p+Q/GpvbKIqqD0LEwo7d5j/+MA8rV0RCOMmHPZhjZz1NOh1KPP53/SlBRSQTO b8jEkuZRFFgfkPIR70Iz7y8DPc0WVj9IChJwxagDN2x6EHa5AmZsRWp4Hkih+Vf2CFTe3HjfT/b6H h1Vrb6kHzkdowv7Vyggt3KZ3MyO8AenPdGTQuwv2pHVJntZjcJkfW8VTcvxLex3aWiUPrFCO28WXN j4/Wa7cNeccg3pb3WDpsHmUp5Hzm1wia4ycioqUqcVAyyDqX/qWdlvqzeMbnM1OVG/AMzg7PsQgHP 3P6g1tBLw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kgESF-0001vr-T6; Fri, 20 Nov 2020 22:00:32 +0000 Received: from smtp-out.xnet.cz ([178.217.244.18]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kgESC-0001vW-Hp for openwrt-devel@lists.openwrt.org; Fri, 20 Nov 2020 22:00:30 +0000 Received: from meh.true.cz (meh.true.cz [108.61.167.218]) (Authenticated sender: petr@true.cz) by smtp-out.xnet.cz (Postfix) with ESMTPSA id AFE96186DE; Fri, 20 Nov 2020 23:00:24 +0100 (CET) Received: by meh.true.cz (OpenSMTPD) with ESMTP id fd9d2f78; Fri, 20 Nov 2020 23:00:05 +0100 (CET) From: =?utf-8?q?Petr_=C5=A0tetiar?= To: openwrt-devel@lists.openwrt.org Subject: [PATCH v2] download: handle possibly invalid local tarballs Date: Fri, 20 Nov 2020 23:00:18 +0100 Message-Id: <20201120220018.25061-1-ynezz@true.cz> MIME-Version: 1.0 In-Reply-To: <20201119212050.14005-1-ynezz@true.cz> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201120_170028_707717_9729067D X-CRM114-Status: GOOD ( 18.08 ) X-Spam-Score: 0.0 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 SPF_NONE SPF: sender does not publish an SPF Record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: OpenWrt Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: =?utf-8?q?Petr_=C5=A0tetiar?= Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org Currently it's assumed, that already downloaded tarballs are always fine, so no checksum checking is performed and the tarball is used even if it might be corrupted. From now on, we're going to always check the downloaded tarballs before considering them valid. Steps to reproduce: 1. Remove cached tarball rm dl/libubox-2020-08-06-9e52171d.tar.xz 2. Download valid tarball again make package/libubox/download 3. Invalidate the tarball sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile 4. Now compile with corrupt tarball source make package/libubox/{clean,compile} Signed-off-by: Petr Štetiar --- Changes since v1: * fixed infinite re-downloading of the source tarball when using KERNEL_GIT_LOCAL_REPOSITORY include/host-build.mk | 2 ++ include/package.mk | 2 ++ scripts/download.pl | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+) diff --git a/include/host-build.mk b/include/host-build.mk index 7d84ab0f5fc4..4ac140518113 100644 --- a/include/host-build.mk +++ b/include/host-build.mk @@ -186,6 +186,8 @@ ifndef DUMP clean-build: host-clean-build endif + $(DL_DIR)/$(FILE): FORCE + $(_host_target)host-prepare: $(HOST_STAMP_PREPARED) $(_host_target)host-configure: $(HOST_STAMP_CONFIGURED) $(_host_target)host-compile: $(HOST_STAMP_BUILT) $(HOST_STAMP_INSTALLED) diff --git a/include/package.mk b/include/package.mk index 50bd838180d8..5eb4460db86c 100644 --- a/include/package.mk +++ b/include/package.mk @@ -189,6 +189,8 @@ define Build/CoreTargets $(call Build/Autoclean) $(call DefaultTargets) + $(DL_DIR)/$(FILE): FORCE + download: $(foreach hook,$(Hooks/Download), $(call $(hook))$(sep) diff --git a/scripts/download.pl b/scripts/download.pl index 351b06a08b2f..2d87f47f842b 100755 --- a/scripts/download.pl +++ b/scripts/download.pl @@ -262,6 +262,24 @@ foreach my $mirror (@ARGV) { push @mirrors, 'https://sources.openwrt.org'; push @mirrors, 'https://mirror2.openwrt.org/sources'; +if (-f "$target/$filename") { + $hash_cmd and do { + if (system("cat '$target/$filename' | $hash_cmd > '$target/$filename.hash'")) { + die "Failed to generate hash for $filename\n"; + } + + my $sum = `cat "$target/$filename.hash"`; + $sum =~ /^(\w+)\s*/ or die "Could not generate file hash\n"; + $sum = $1; + + exit 0 if $sum eq $file_hash; + + die "Hash of the local file $filename does not match (file: $sum, requested: $file_hash) - deleting download.\n"; + unlink "$target/$filename"; + cleanup(); + }; +} + while (!-f "$target/$filename") { my $mirror = shift @mirrors; $mirror or die "No more mirrors to try - giving up.\n";