From patchwork Sun May 4 12:01:47 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 345470 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from fraxinus.osuosl.org (fraxinus.osuosl.org [140.211.166.137]) by ozlabs.org (Postfix) with ESMTP id 7613B140194 for ; Sun, 4 May 2014 22:02:33 +1000 (EST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8DB9A8B986; Sun, 4 May 2014 12:02:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id D-gzzZxu5pMK; Sun, 4 May 2014 12:02:18 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3AC0D8BD93; Sun, 4 May 2014 12:02:13 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (fraxinus.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 1E7ED1C2226 for ; Sun, 4 May 2014 12:02:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0CE088BD8F for ; Sun, 4 May 2014 12:02:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kvJ9hgIkbYWp for ; Sun, 4 May 2014 12:02:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wg0-f42.google.com (mail-wg0-f42.google.com [74.125.82.42]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 706628B192 for ; Sun, 4 May 2014 12:02:04 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id y10so253822wgg.13 for ; Sun, 04 May 2014 05:02:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=4NF9JxwG0JIoBdx0xgt7Wn/6SMcuD8p44x3qVL2IoQ4=; b=eaN2AtWOUjKyVLEKIDt8prxp3/g0m7xJXdOWdoxhYM45DA2aPCKCk72b1VMl93IXw9 RKeJIxR7lserHtmDFD46fendB4xqJA0a1HrlBYF0MRmYwdCwdCh9NeQMhHeus5iOeLzP x5QEwZgo2g3RTEztS8ND7yxYW+gw3KrqM3nZfQvlfhAz9GkegErtm+dir7IcKRQrmLAe 7Tr9WCH7wV4kocKvNj3xU2KJkyG9Uyj9k8l4euXRjNS2YbvsU/lgFeMMP0G6IbDH9/Oi xzmmfTtQUksLI+r0Rtw1qeU8AXDAwGs4ADImMBw+g6s7xGSkva4qEWaBfPgVYknrdi/C tZHQ== X-Received: by 10.194.78.77 with SMTP id z13mr160270wjw.64.1399204922989; Sun, 04 May 2014 05:02:02 -0700 (PDT) Received: from gourin.bzh.lan (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPSA id iy13sm10455501wic.1.2014.05.04.05.02.01 for (version=TLSv1.1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 04 May 2014 05:02:02 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Sun, 4 May 2014 14:01:47 +0200 Message-Id: X-Mailer: git-send-email 1.8.3.2 In-Reply-To: References: Cc: "Yann E. MORIN" Subject: [Buildroot] [PATCH 08/11] pkg-infra: don't use DL_DIR as scratchpad for temporary VCS checkouts X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: buildroot-bounces@busybox.net From: "Yann E. MORIN" DL_DIR can be a very precious place for some users: they use it to store all the downloaded archives to share across all their Buildroot (and maybe non-Buildroot) builds. We do not want to trash this location with our temporary VCS (git, Hg, svn, cvs) repository clones/checkouts. Turns out that we already have some kind of scratchpad, the BUILD_DIR. Although it is not really a disposable location, that's the best we have so far. Also, we create the temporary tarballs by appending '.tmp' to the final tarball, since we want the temporary to be on the same filesystem as the final location, so the 'mv' is just a plain rename(2), and we are not left with a half-copied file as the final location. Note: we're using neither ${TMP} nor ${TMPDIR} since they are shared locations, sometime with little place (eg. tmpfs), and some of the repositories we clone/checkout can be very big. Signed-off-by: "Yann E. MORIN" Cc: Samuel Martin Cc: Arnout Vandecappelle --- Changes v4 -> v5 - explain why we create the temp files where we create them (Arnout) Changes v3 -> v4: - remove spurious bump in package/fis (Samuel) --- support/download/cvs | 18 ++++++++++++------ support/download/git | 16 ++++++++++------ support/download/hg | 20 +++++++++++++------- support/download/svn | 19 ++++++++++++------- support/download/wget | 7 +++++-- 5 files changed, 52 insertions(+), 28 deletions(-) diff --git a/support/download/cvs b/support/download/cvs index 06b8647..e4c1f32 100755 --- a/support/download/cvs +++ b/support/download/cvs @@ -11,8 +11,8 @@ set -e # $4: package's basename (eg. foobar-1.2.3) # $5: output file # And this environment: -# CVS : the cvs command to call -# BR2_DL_DIR: path to Buildroot's download dir +# CVS : the cvs command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" rev="${2}" @@ -20,8 +20,14 @@ rawname="${3}" basename="${4}" output="${5}" -cd "${BR2_DL_DIR}" +repodir="${basename}.tmp-cvs-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + ${CVS} -z3 -d":pserver:anonymous@${repo}" \ - co -d "${basename}" -r ":${rev}" -P "${rawname}" -tar czf "${output}" "${basename}" -rm -rf "${basename}" + co -d "${repodir}" -r ":${rev}" -P "${rawname}" +tar czf "${output}.tmp" "${repodir}" +mv "${output}.tmp" "${output}" +rm -rf "${repodir}" diff --git a/support/download/git b/support/download/git index 96db3a9..5ba26fb 100755 --- a/support/download/git +++ b/support/download/git @@ -10,15 +10,19 @@ set -e # $3: package's basename (eg. foobar-1.2.3) # $4: output file # And this environment: -# BR2_DL_DIR: path to Buildroot's download dir -# GIT : the git command to call +# GIT : the git command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" cset="${2}" basename="${3}" output="${4}" -repodir="${BR2_DL_DIR}/${basename}" +repodir="${basename}.tmp-git-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" "${output}.tmp2" if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then printf "Doing shallow clone\n" @@ -30,8 +34,8 @@ fi pushd "${repodir}" ${GIT} archive --prefix="${basename}/" -o "${output}.tmp" --format=tar "${cset}" -gzip -c "${output}.tmp" >"${output}" -rm -f "${output}.tmp" +gzip -c "${output}.tmp" >"${output}.tmp2" +mv "${output}.tmp2" "${output}" popd -rm -rf "${repodir}" +rm -rf "${repodir}" "${output}.tmp" diff --git a/support/download/hg b/support/download/hg index 70b49cf..d96a0c5 100755 --- a/support/download/hg +++ b/support/download/hg @@ -10,16 +10,22 @@ set -e # $3: package's basename (eg. foobar-1.2.3) # $4: output file # And this environment: -# HG : the hg command to call -# BR2_DL_DIR: path to Buildroot's download dir +# HG : the hg command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" cset="${2}" basename="${3}" output="${4}" -cd "${BR2_DL_DIR}" -${HG} clone --noupdate --rev "${cset}" "${repo}" "${basename}" -${HG} archive --repository "${basename}" --type tgz --prefix "${basename}" \ - --rev "${cset}" "${output}" -rm -rf "${basename}" +repodir="${basename}.tmp-hg-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + +${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}" +${HG} archive --repository "${repodir}" --type tgz --prefix "${basename}" \ + --rev "${cset}" "${output}.tmp" +mv "${output}.tmp" "${output}" +rm -rf "${repodir}" diff --git a/support/download/svn b/support/download/svn index c3ab32c..073709f 100755 --- a/support/download/svn +++ b/support/download/svn @@ -10,16 +10,21 @@ set -e # $3: package's basename (eg. foobar-1.2.3) # $4: output file # And this environment: -# SVN : the svn command to call -# BR2_DL_DIR: path to Buildroot's download dir +# SVN : the svn command to call +# BUILD_DIR: path to Buildroot's build dir repo="${1}" rev="${2}" basename="${3}" output="${4}" -pushd "${BR2_DL_DIR}" -${SVN} export -r "${rev}" "${repo}" "${basename}" -tar czf "${output}" "${basename}" -rm -rf "${basename}" -popd +repodir="${basename}.tmp-svn-checkout" + +cd "${BUILD_DIR}" +# Remove leftovers from a previous failed run +rm -rf "${repodir}" "${output}.tmp" + +${SVN} export -r "${rev}" "${repo}" "${repodir}" +tar czf "${output}.tmp" "${repodir}" +mv "${output}.tmp" "${output}" +rm -rf "${repodir}" diff --git a/support/download/wget b/support/download/wget index 7865517..7c941e4 100755 --- a/support/download/wget +++ b/support/download/wget @@ -8,14 +8,17 @@ set -e # $1: URL # $2: output file # And this environment: -# WGET : the wget command to call +# WGET : the wget command to call url="${1}" output="${2}" +# Remove leftovers from a previous failed run +rm -rf "${output}.tmp" + if ${WGET} -O "${output}.tmp" "${url}"; then mv "${output}.tmp" "${output}" else - rm -f "${output}.tmp" + rm -f "${tmpfile}" exit 1 fi