From patchwork Thu Dec 11 18:24:48 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 420216 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from hemlock.osuosl.org (hemlock.osuosl.org [140.211.166.133]) by ozlabs.org (Postfix) with ESMTP id A4EFA1400B7 for ; Fri, 12 Dec 2014 05:25:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id DB3399609E; Thu, 11 Dec 2014 18:25:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id A+TRaqi8jL0F; Thu, 11 Dec 2014 18:25:03 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 9AA4E96082; Thu, 11 Dec 2014 18:25:02 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (fraxinus.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 0F1171C2854 for ; Thu, 11 Dec 2014 18:24:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0B8EAA1510 for ; Thu, 11 Dec 2014 18:24:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EKZs88pmf5jY for ; Thu, 11 Dec 2014 18:24:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wi0-f179.google.com (mail-wi0-f179.google.com [209.85.212.179]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 5029DA1502 for ; Thu, 11 Dec 2014 18:24:58 +0000 (UTC) Received: by mail-wi0-f179.google.com with SMTP id ex7so213700wid.0 for ; Thu, 11 Dec 2014 10:24:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=9QB1lYB0puG7kvqrtp3RsLTDz5cbIH9WTqdP1SjoruU=; b=Dif7KqPqsLoD3nKmBMwAiyxeyKkz37El6oO9yoOZ4aqhd4GWhWMQOv88RlqP/wamVe SA9Q3HBgck9yDAkcyBI77yoQPOpFLWXUe7ijXJt1V3lDyPHAEf9MXOUUjfVNb6vPxzGq 8dl8oGMPP68x6EfxQLfAf325I6/8W3uE3SLUD5myz8xetR22/jhjXrVSM/rrim7pQdOt KtTrXo1gdMWcg4JNm5YMZv8KSK/RnpQCGWJGPJdwyDNBVrH6ZfP4MK0rLWIB0zGZaivf d/umHzu4LkOYTh0eBJjye8VX+xE2s3ELvCrAChxfCHWNfD2h7D6ahf7Lj/UMOauh29XC Cwjw== X-Received: by 10.180.109.130 with SMTP id hs2mr460298wib.54.1418322295720; Thu, 11 Dec 2014 10:24:55 -0800 (PST) Received: from gourin.bzh.lan (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPSA id l3sm2668776wje.12.2014.12.11.10.24.54 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 11 Dec 2014 10:24:54 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Thu, 11 Dec 2014 19:24:48 +0100 Message-Id: <0301f2d3c762f8815c7c6dc25ed5ea8307275305.1418322200.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 1.9.1 In-Reply-To: References: Cc: Thomas Petazzoni , "Yann E. MORIN" , Peter Korsgaard Subject: [Buildroot] [PATCH 4/4 v4] pkg-download: check hashes for locally cached files X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In some cases, upstream just update their releases in-place, without renaming them. When that package is updated in Buildroot, a new hash to match the new upstream release is included in the corresponding .hash file. As a consequence, users who previously downloaded that package's tarball with an older version of Buildroot, will get stuck with an old archive for that package, and after updating their Buildroot copy, will be greeted with a failed download, due to the local file not matching the new hashes. So, to avoid this situation, check the hashes prior to doing the download. If the hashes match, consider the locally cached file genuine, and do not download it. However, if the locally cached file does not match the known hashes we have for it, it is promptly removed, and a download is re-attempted. Note: this does not add any overhead compared to the previous situation, because we were already checking hashes of localy cached files. It just changes the order in which we do the checks. For the records, here is the overhead of hashing a 231MiB file (qt-everywhere-opensource-src-4.8.6.tar.gz) on a core-i5 @2.5GHz: cache-cold cache-hot sha1 1.914s 0.762s sha256 2.109s 1.270s But again, this overhead already existed before this patch. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni Cc: Peter Korsgaard Cc: Gustavo Zacarias Reviewed-by: Thomas Petazzoni --- support/download/dl-wrapper | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index fdb49db..d66715f 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -56,7 +56,11 @@ main() { # If the output file already exists, do not download it again if [ -e "${output}" ]; then - exit 0 + if support/download/check-hash "${hfile}" "${output}" "${output##*/}"; then + exit 0 + fi + rm -f "${output}" + printf "Re-downloading '%s'...\n" "${output##*/}" fi # tmpd is a temporary directory in which backends may store intermediate