From patchwork Thu Dec 11 22:52:08 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 420277 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from hemlock.osuosl.org (hemlock.osuosl.org [140.211.166.133]) by ozlabs.org (Postfix) with ESMTP id 8BC901400EA for ; Fri, 12 Dec 2014 09:52:25 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 9005096110; Thu, 11 Dec 2014 22:52:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eeACZqdbJ6R0; Thu, 11 Dec 2014 22:52:24 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id DA8A6960DC; Thu, 11 Dec 2014 22:52:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (silver.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id E3ED41BFA34 for ; Thu, 11 Dec 2014 22:52:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C5A1C3400E for ; Thu, 11 Dec 2014 22:52:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rmeJ5IVkwsJ9 for ; Thu, 11 Dec 2014 22:52:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wg0-f42.google.com (mail-wg0-f42.google.com [74.125.82.42]) by silver.osuosl.org (Postfix) with ESMTPS id 4CAB533FE9 for ; Thu, 11 Dec 2014 22:52:17 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id z12so7718703wgg.1 for ; Thu, 11 Dec 2014 14:52:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=fXFTRtFiuxQd220FY586TNgfimwuV+Hke9jccJz9eSk=; b=c4yS3J1IGeAvr4mTLoMTCVtcjhrWSeRzagmXWXtzBxUxZJR5F8ReI5vxCTRgZ88eAW 5xD5+cz0XESrT5ovukh5GT/8eOVbrbX6VJtQzcrWz+byx96vklVUMhmebCKhBm2Z1gxH zbwfKOiVbzhNTvglQ7XYl8EpzkHnt8NtRXZA7IWI8A/E0JXAOYSyBJ3I/mXACNndP0Ti vNBi8Uu2Tjxs/4VSKB90aKrCejOm0TzcKSx07zNsfCDxU77FhpgnAWRkQua4IvO6v8hF IqHe1l26UqFlPLsPhWHeIHW68rdzSQZk4Jm2q9miTpr5uGjBwchgI0kddep9usmwDeJ8 Tj+g== X-Received: by 10.180.109.130 with SMTP id hs2mr2204712wib.54.1418338335854; Thu, 11 Dec 2014 14:52:15 -0800 (PST) Received: from gourin.bzh.lan (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPSA id l3sm3387299wje.12.2014.12.11.14.52.14 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 11 Dec 2014 14:52:15 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Thu, 11 Dec 2014 23:52:08 +0100 Message-Id: <06b4eb57a7671f63c52d7a71bfc141682fdb0e38.1418337531.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 1.9.1 In-Reply-To: References: Cc: Thomas Petazzoni , "Yann E. MORIN" , Peter Korsgaard Subject: [Buildroot] [PATCH 4/4 v5] pkg-download: check hashes for locally cached files X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In some cases, upstream just update their releases in-place, without renaming them. When that package is updated in Buildroot, a new hash to match the new upstream release is included in the corresponding .hash file. As a consequence, users who previously downloaded that package's tarball with an older version of Buildroot, will get stuck with an old archive for that package, and after updating their Buildroot copy, will be greeted with a failed download, due to the local file not matching the new hashes. Also, an upstream would sometime serve us HTML garbage instead of the actual tarball we requested, like SourceForge does from time for as-yet unknown reasons. So, to avoid this situation, check the hashes prior to doing the download. If the hashes match, consider the locally cached file genuine, and do not download it. However, if the locally cached file does not match the known hashes we have for it, it is promptly removed, and a download is re-attempted. Note: this does not add any overhead compared to the previous situation, because we were already checking hashes of locally cached files. It just changes the order in which we do the checks. For the records, here is the overhead of hashing a 231MiB file (qt-everywhere-opensource-src-4.8.6.tar.gz) on a core-i5 @2.5GHz: cache-cold cache-hot sha1 1.914s 0.762s sha256 2.109s 1.270s But again, this overhead already existed before this patch. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni Cc: Peter Korsgaard Cc: Gustavo Zacarias Reviewed-by: Thomas Petazzoni --- support/download/dl-wrapper | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index f0cdd73..cced8f6 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -49,7 +49,11 @@ main() { # If the output file already exists, do not download it again if [ -e "${output}" ]; then - exit 0 + if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then + exit 0 + fi + rm -f "${output}" + printf "Re-downloading '%s'...\n" "${output##*/}" fi # tmpd is a temporary directory in which backends may store intermediate