From patchwork Mon Jul 7 21:44:37 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 367738 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from silver.osuosl.org (silver.osuosl.org [140.211.166.136]) by ozlabs.org (Postfix) with ESMTP id 99D461400BB for ; Tue, 8 Jul 2014 07:45:17 +1000 (EST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9973B31BF3; Mon, 7 Jul 2014 21:45:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id taeKbCWLkbGL; Mon, 7 Jul 2014 21:45:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id B481131C76; Mon, 7 Jul 2014 21:44:58 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (silver.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 186001BF96A for ; Mon, 7 Jul 2014 21:44:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0CA0631792 for ; Mon, 7 Jul 2014 21:44:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LFowFaxSQFNr for ; Mon, 7 Jul 2014 21:44:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-we0-f169.google.com (mail-we0-f169.google.com [74.125.82.169]) by silver.osuosl.org (Postfix) with ESMTPS id 76D7C31963 for ; Mon, 7 Jul 2014 21:44:48 +0000 (UTC) Received: by mail-we0-f169.google.com with SMTP id t60so5082243wes.28 for ; Mon, 07 Jul 2014 14:44:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=OrpQgEhxANSXUTFBPsg3ghHAQs3yGRK3Et1LkokG7w8=; b=FSgtr+3OdOA/1uuDLv9NMOhfwsM936qVD60O23B3xR7xpRqx+8K0SoeYZyaDf820xP MAyQVGb3bRFvtMFis9gAgSRv29IxsjcO5Jha39RqMz8+TPxOE3hNKAne8FTRVd96ghFB RMvJsqU3bfFXPOylOPJm8OJV252zrlMoy+7HGzPV+FFQ2N6HJZzb1w8PBXn9JmBqCFbg nnZN3/xXEYMiwNcC8K8ts0Blat5vC+2FIIgbsH7uYP5a+A08lhLe4ETCVjYjr0ubbWhV 2kuMIl/egcMUqzF67m0CIlByR+T26anXXdpMCTrTqIUDrmdAL/fRO4qUC+RmzgRh/qRc HU7A== X-Received: by 10.194.238.6 with SMTP id vg6mr34552287wjc.24.1404769487004; Mon, 07 Jul 2014 14:44:47 -0700 (PDT) Received: from gourin.bzh.lan (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPSA id cj8sm91949118wjb.5.2014.07.07.14.44.45 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 07 Jul 2014 14:44:46 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Mon, 7 Jul 2014 23:44:37 +0200 Message-Id: X-Mailer: git-send-email 1.9.1 In-Reply-To: References: Cc: "Yann E. MORIN" Subject: [Buildroot] [PATCH 6/6 v2] support/download: rationalise naming and use of the temporary files X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: buildroot-bounces@busybox.net Currently, we have temporary files to store 'downloaded' files or repositories, and they all are created using 'mktemp .XXXXXX'. Also, temporary repositories are named in a inconsistent manner. This poses a problem in case something goes wrong: it is not possible to easily see what temp file corresponds to what attempted download. Make all this a bit more homogeneous: - name the temporary files and directories after the final file, with this mktemp pattern: ${BUILD_DIR}/.${output##*/}.XXXXXX This ensures it is easy to correlate a temp file or dir to the associated download attempt. Signed-off-by: "Yann E. MORIN" --- Notes: - I was not able to test those helpers, because we have no package using them: cp, cvs, scp; - I tested bzr with our sole package using bazaar, so corner cases might spring to our attention at unexpected times; - the cvs helper uses a little trick, due to the fact that cvs is so ancient it has never learned the ability to generate archives on its own. Tested with this defconfig, which exercises all possible helpers: BR2_x86_i686=y BR2_TOOLCHAIN_EXTERNAL=y # wget BR2_PACKAGE_DVB_APPS=y # hg BR2_PACKAGE_DTV_SCAN_TABLES=y # git BR2_PACKAGE_FIS=y # svn BR2_PACKAGE_PYTHON=y # wget, needed for pynfc: BR2_PACKAGE_PYTHON_NFC=y # bzr --- Changes v1 -> v2: - remove left-over hunk in cvs helper - fix double-dot in scp helper - use the tar transform trick in svn helper, too --- support/download/bzr | 3 +-- support/download/cvs | 17 ++++++++++------- support/download/git | 7 +------ support/download/hg | 7 +------ support/download/scp | 4 +++- support/download/svn | 19 +++++++++++-------- support/download/wget | 3 +-- 7 files changed, 28 insertions(+), 32 deletions(-) diff --git a/support/download/bzr b/support/download/bzr index a2cb440..ff01bff 100755 --- a/support/download/bzr +++ b/support/download/bzr @@ -16,8 +16,6 @@ repo="${1}" rev="${2}" output="${3}" -tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )" - # Play tic-tac-toe with temp files # - first, we download to a trashable location (the build-dir) # - the we move to a temp file in the final location, so it is @@ -25,6 +23,7 @@ tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )" # - finally, we atomically rename to the final file ret=1 +tmp_dl="$( mktemp "${BUILD_DIR}/.${output##*/}.XXXXXX" )" if ${BZR} export --format=tgz "${tmp_dl}" "${repo}" -r "${rev}"; then tmp_output="$( mktemp "${output}.XXXXXX" )" if cat "${tmp_dl}" >"${tmp_output}"; then diff --git a/support/download/cvs b/support/download/cvs index 22863d8..b580eda 100755 --- a/support/download/cvs +++ b/support/download/cvs @@ -20,23 +20,26 @@ rawname="${3}" basename="${4}" output="${5}" -repodir="${basename}.tmp-cvs-checkout" - -cd "${BUILD_DIR}" -# Remove leftovers from a previous failed run -rm -rf "${repodir}" - # Play tic-tac-toe with temp files # - first, we download to a trashable location (the build-dir) # - then we create a temporary tarball in the final location, so it is # on the same filesystem as the final file # - finally, we atomically rename to the final file +# Since cvs by itself is not capable of generating archives, we use +# a little trick of tar, to transform the filenames as they are being +# added to the archive, by replacing the leadin '.' with the basename +# of the package. Note: basename is just that, a basename, so it does +# not contain any '/', so we need not escape them for the transform +# expression. + ret=1 +repodir="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )" if ${CVS} -z3 -d":pserver:anonymous@${repo}" \ co -d "${repodir}" -r ":${rev}" -P "${rawname}"; then tmp_output="$( mktemp "${output}.XXXXXX" )" - if tar czf - "${repodir}" >"${tmp_output}"; then + if tar czf - -C "${repodir}" --transform "s/^\./${basename}/;" . \ + >"${tmp_output}"; then mv "${tmp_output}" "${output}" ret=0 fi diff --git a/support/download/git b/support/download/git index e8e9055..c53d87f 100755 --- a/support/download/git +++ b/support/download/git @@ -18,8 +18,6 @@ cset="${2}" basename="${3}" output="${4}" -repodir="${basename}.tmp-git-checkout" - # Play tic-tac-toe with temp files # - first, we download to a trashable location (the build-dir) # - then we create the uncomporessed tarball in tht same trashable location @@ -27,14 +25,11 @@ repodir="${basename}.tmp-git-checkout" # it is on the same filesystem as the final file # - finally, we atomically rename to the final file -cd "${BUILD_DIR}" -# Remove leftovers from a previous failed run -rm -rf "${repodir}" - # Upon failure, git cleans behind itself, so no need to catch failures # here. The only case when git would not clean up, is if it gets killed # with SIGKILL, which is user-initiated, so we let the user handle that. git_done=0 +repodir="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )" if [ -n "$(${GIT} ls-remote "${repo}" "${cset}" 2>&1)" ]; then printf "Doing shallow clone\n" if ${GIT} clone --depth 1 -b "${cset}" --bare "${repo}" "${repodir}"; then diff --git a/support/download/hg b/support/download/hg index 7e5c9eb..104d034 100755 --- a/support/download/hg +++ b/support/download/hg @@ -18,12 +18,6 @@ cset="${2}" basename="${3}" output="${4}" -repodir="${basename}.tmp-hg-checkout" - -cd "${BUILD_DIR}" -# Remove leftovers from a previous failed run -rm -rf "${repodir}" - # Play tic-tac-toe with temp files # - first, we download to a trashable location (the build-dir) # - then we create a temporary tarball in the final location, so it is @@ -31,6 +25,7 @@ rm -rf "${repodir}" # - finally, we atomically rename to the final file ret=1 +repodir="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )" if ${HG} clone --noupdate --rev "${cset}" "${repo}" "${repodir}"; then tmp_output="$( mktemp "${output}.XXXXXX" )" if ${HG} archive --repository "${repodir}" --type tgz \ diff --git a/support/download/scp b/support/download/scp index 1cc18de..6aa3a23 100755 --- a/support/download/scp +++ b/support/download/scp @@ -12,9 +12,11 @@ set -e url="${1}" output="${2}" -tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )" +tmp_dl="$( mktemp "${BUILD_DIR}/.${output##*/}.XXXXXX" )" ret=1 +# Note: it looks like scp does not unlink the destination, +# which is what we want (observed by stracing scp.) if ${SCP} "${url}" "${tmp_dl}"; then tmp_output="$( mktemp "${output}.XXXXXX" )" if cat "${tmp_dl}" >"${tmp_output}"; then diff --git a/support/download/svn b/support/download/svn index 232d887..7a46d0f 100755 --- a/support/download/svn +++ b/support/download/svn @@ -18,22 +18,25 @@ rev="${2}" basename="${3}" output="${4}" -repodir="${basename}.tmp-svn-checkout" - -cd "${BUILD_DIR}" -# Remove leftovers from a previous failed run -rm -rf "${repodir}" - # Play tic-tac-toe with temp files # - first, we download to a trashable location (the build-dir) # - then we create a temporary tarball in the final location, so it is # on the same filesystem as the final file # - finally, we atomically rename to the final file +# Since svn by itself is not capable of generating archives, we use +# a little trick of tar, to transform the filenames as they are being +# added to the archive, by replacing the leadin '.' with the basename +# of the package. Note: basename is just that, a basename, so it does +# not contain any '/', so we need not escape them for the transform +# expression. + ret=1 -if ${SVN} export "${repo}@${rev}" "${repodir}"; then +repodir="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )" +if ${SVN} export --force "${repo}@${rev}" "${repodir}"; then tmp_output="$( mktemp "${output}.XXXXXX" )" - if tar czf - "${repodir}" >"${tmp_output}"; then + if tar czf - -C "${repodir}" --transform "s/^\./${basename}/;" . \ + >"${tmp_output}"; then mv "${tmp_output}" "${output}" ret=0 fi diff --git a/support/download/wget b/support/download/wget index e961d71..9bb6700 100755 --- a/support/download/wget +++ b/support/download/wget @@ -14,8 +14,6 @@ set -e url="${1}" output="${2}" -tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )" - # Play tic-tac-toe with temp files # - first, we download to a trashable location (the build-dir) # - then we copy to a temporary tarball in the final location, so it is @@ -23,6 +21,7 @@ tmp_dl="$( mktemp "${BUILD_DIR}/.XXXXXX" )" # - finally, we atomically rename to the final file ret=1 +tmp_dl="$( mktemp "${BUILD_DIR}/.${output##*/}.XXXXXX" )" if ${WGET} -O "${tmp_dl}" "${url}"; then tmp_output="$( mktemp "${output}.XXXXXX" )" if cat "${tmp_dl}" >"${tmp_output}"; then