From patchwork Thu Dec 11 18:24:45 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 420213 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from hemlock.osuosl.org (hemlock.osuosl.org [140.211.166.133]) by ozlabs.org (Postfix) with ESMTP id 550F11400B7 for ; Fri, 12 Dec 2014 05:24:58 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 68C109606D; Thu, 11 Dec 2014 18:24:57 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V83P6kcwUdZX; Thu, 11 Dec 2014 18:24:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id E9C928A551; Thu, 11 Dec 2014 18:24:55 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (hemlock.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id BBD391C2854 for ; Thu, 11 Dec 2014 18:24:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id B87C789E61 for ; Thu, 11 Dec 2014 18:24:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pHg7pKtNduQk for ; Thu, 11 Dec 2014 18:24:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wi0-f170.google.com (mail-wi0-f170.google.com [209.85.212.170]) by hemlock.osuosl.org (Postfix) with ESMTPS id B9FCF89EAB for ; Thu, 11 Dec 2014 18:24:52 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id bs8so2342010wib.1 for ; Thu, 11 Dec 2014 10:24:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=vxoIJLJErRstpD50n+f9OFGtTmIbeNl/xIMyotq+NfA=; b=pW/9uFiNmiCkKCa5gfs/M8Nw7CEApVH26lWKm9EDelllvxuucESw3Klhd6ixtLtoP2 yTRaWnfOEI3jGwYylQHv9Sw8SEALPKRJr4yOJKa9y8ddE3xCYrahPAKEs/F9zmg3WMJ6 w+kLrlWuoUXGW6j84Tx5XbxWI+uEu+gkoS5GPLNXkMqXt4XuPRvsAiNjBrZOqUXE+P5d +BbVD2wAFZSMTfnOF/nOxmhERbDEzTwfChVRkc+ut3jkOoOKN69/XmB+/ZiyTZClTadw 6Y49KfsR4bqDysem6oZjmTNUS0IXOerMpK7mfPE+PSvHrCRL9+gmyb23KfIAhnosOiCH m7JQ== X-Received: by 10.194.157.4 with SMTP id wi4mr5055102wjb.54.1418322291319; Thu, 11 Dec 2014 10:24:51 -0800 (PST) Received: from gourin.bzh.lan (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPSA id l3sm2668776wje.12.2014.12.11.10.24.49 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 11 Dec 2014 10:24:50 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Thu, 11 Dec 2014 19:24:45 +0100 Message-Id: <204467c4c3722faed45db924c489ad768ed1ae33.1418322200.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 1.9.1 In-Reply-To: References: Cc: Thomas Petazzoni , "Yann E. MORIN" Subject: [Buildroot] [PATCH 1/4 v4] suppot/download: add option parsing to the download wrapper X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Instead of relying on argument ordering, use actual options in the download wrapper. Download backends (bzr, cp, hg...) are left as-is, because it does not make sense to complexifies them, since they are almost very trivial shell scripts, and adding option parsing would be really overkill. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni --- package/pkg-download.mk | 42 ++++++----- support/download/dl-wrapper | 166 ++++++++++++++++++++++++++++++++++++++++++++ support/download/wrapper | 99 -------------------------- 3 files changed, 192 insertions(+), 115 deletions(-) create mode 100755 support/download/dl-wrapper delete mode 100755 support/download/wrapper diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 190b5b7..8424eca 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -18,6 +18,8 @@ export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET) SSH := $(call qstrip,$(BR2_SSH)) $(QUIET) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) +DL_WRAPPER = support/download/dl-wrapper + # Default spider mode is 'DOWNLOAD'. Other possible values are 'SOURCE_CHECK' # used by the _source-check target and 'SHOW_EXTERNAL_DEPS', used by the # external-deps target. @@ -95,8 +97,9 @@ endef # problems define DOWNLOAD_GIT test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper git \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b git \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -115,8 +118,9 @@ endef define DOWNLOAD_BZR test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper bzr \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b bzr \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -132,8 +136,9 @@ endef define DOWNLOAD_CVS test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper cvs \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \ $($(PKG)_DL_VERSION) \ $($(PKG)_RAWNAME) \ @@ -151,8 +156,9 @@ endef define DOWNLOAD_SVN test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper svn \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b svn \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -171,8 +177,9 @@ endef # to prepend the path with a slash: scp://[user@]host:/absolutepath define DOWNLOAD_SCP test -e $(DL_DIR)/$(2) || \ - $(EXTRA_ENV) support/download/wrapper scp \ - $(DL_DIR)/$(2) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b scp \ + -o $(DL_DIR)/$(2) \ + -- \ '$(call stripurischeme,$(call qstrip,$(1)))' && \ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_RAWNAME).hash,$(DL_DIR)/$(2)) endef @@ -188,8 +195,9 @@ endef define DOWNLOAD_HG test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper hg \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b hg \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -208,8 +216,9 @@ endef define DOWNLOAD_WGET test -e $(DL_DIR)/$(2) || \ - $(EXTRA_ENV) support/download/wrapper wget \ - $(DL_DIR)/$(2) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b wget \ + -o $(DL_DIR)/$(2) \ + -- \ '$(call qstrip,$(1))' && \ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_RAWNAME).hash,$(DL_DIR)/$(2)) endef @@ -224,8 +233,9 @@ endef define DOWNLOAD_LOCALFILES test -e $(DL_DIR)/$(2) || \ - $(EXTRA_ENV) support/download/wrapper cp \ - $(DL_DIR)/$(2) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b cp \ + -o $(DL_DIR)/$(2) \ + -- \ $(call stripurischeme,$(call qstrip,$(1))) && \ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_RAWNAME).hash,$(DL_DIR)/$(2)) endef diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper new file mode 100755 index 0000000..b1b9158 --- /dev/null +++ b/support/download/dl-wrapper @@ -0,0 +1,166 @@ +#!/usr/bin/env bash + +# This script is a wrapper to the other download backends. +# Its role is to ensure atomicity when saving downloaded files +# back to BR2_DL_DIR, and not clutter BR2_DL_DIR with partial, +# failed downloads. +# +# Call it with: +# -b BACKEND name of the backend (eg. cvs, git, cp...) +# -o FILE full path to the file in which to save the download +# -- everything after '--' are options for the backend +# Environment: +# BUILD_DIR: the path to Buildroot's build dir + +# To avoid cluttering BR2_DL_DIR, we download to a trashable +# location, namely in $(BUILD_DIR). +# Then, we move the downloaded file to a temporary file in the +# same directory as the final output file. +# This allows us to finally atomically rename it to its final +# name. +# If anything goes wrong, we just remove all the temporaries +# created so far. + +# We want to catch any unexpected failure, and exit immediately. +set -e + +main() { + local OPT OPTARG + local backend output + + # Parse our options; anythong after '--' is for the backend + # Sop we only care to -b (backend) and -o (output file) + while getopts :hb:o: OPT; do + case "${OPT}" in + h) help; exit 0;; + b) backend="${OPTARG}";; + o) output="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac + done + # Forget our options, and keep only those for the backend + shift $((OPTIND-1)) + + if [ -z "${backend}" ]; then + error "no backend specified, use -b\n" + fi + if [ -z "${output}" ]; then + error "no output specified, use -o\n" + fi + + # tmpd is a temporary directory in which backends may store intermediate + # by-products of the download. + # tmpf is the file in which the backends should put the downloaded content. + # tmpd is located in $(BUILD_DIR), so as not to clutter the (precious) + # $(BR2_DL_DIR) + # We let the backends create tmpf, so they are able to set whatever + # permission bits they want (although we're only really interested in + # the executable bit.) + tmpd="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )" + tmpf="${tmpd}/output" + + # Helpers expect to run in a directory that is *really* trashable, so + # they are free to create whatever files and/or sub-dirs they might need. + # Doing the 'cd' here rather than in all backends is easier. + cd "${tmpd}" + + # If the backend fails, we can just remove the temporary directory to + # remove all the cruft it may have left behind. Then we just exit in + # error too. + if ! "${OLDPWD}/support/download/${backend}" "${tmpf}" "${@}"; then + rm -rf "${tmpd}" + exit 1 + fi + + # cd back to free the temp-dir, so we can remove it later + cd "${OLDPWD}" + + # tmp_output is in the same directory as the final output, so we can + # later move it atomically. + tmp_output="$( mktemp "${output}.XXXXXX" )" + + # 'mktemp' creates files with 'go=-rwx', so the files are not accessible + # to users other than the one doing the download (and root, of course). + # This can be problematic when a shared BR2_DL_DIR is used by different + # users (e.g. on a build server), where all users may write to the shared + # location, since other users would not be allowed to read the files + # another user downloaded. + # So, we restore the 'go' access rights to a more sensible value, while + # still abiding by the current user's umask. We must do that before the + # final 'mv', so just do it now. + # Some backends (cp and scp) may create executable files, so we need to + # carry the executable bit if needed. + [ -x "${tmpf}" ] && new_mode=755 || new_mode=644 + new_mode=$( printf "%04o" $((0${new_mode} & ~0$(umask))) ) + chmod ${new_mode} "${tmp_output}" + + # We must *not* unlink tmp_output, otherwise there is a small window + # during which another download process may create the same tmp_output + # name (very, very unlikely; but not impossible.) + # Using 'cp' is not reliable, since 'cp' may unlink the destination file + # if it is unable to open it with O_WRONLY|O_TRUNC; see: + # http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html + # Since the destination filesystem can be anything, it might not support + # O_TRUNC, so 'cp' would unlink it first. + # Use 'cat' and append-redirection '>>' to save to the final location, + # since that is the only way we can be 100% sure of the behaviour. + if ! cat "${tmpf}" >>"${tmp_output}"; then + rm -rf "${tmpd}" "${tmp_output}" + exit 1 + fi + rm -rf "${tmpd}" + + # tmp_output and output are on the same filesystem, so POSIX guarantees + # that 'mv' is atomic, because it then uses rename() that POSIX mandates + # to be atomic, see: + # http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html + if ! mv -f "${tmp_output}" "${output}"; then + rm -f "${tmp_output}" + exit 1 + fi +} + +help() { + cat <<_EOF_ +NAME + ${my_name} - downlaod wrapper for Buildroot + +SYNOPSIS + ${my_name} [OPTION]... -- [BACKEND OPTION]... + +DESCRIPTION + Wrapper script around different download mechanisms. Ensure that + concurrent downloads do not conflict, that partial downloads are + properly evicted without leaving temporary files, and that access + rights are maintained. + + -h This help text. + + -b BACKEND + Wrap the specified BACKEND. Known backends are: + bzr Bazaar + cp local files via simple copy + cvs Concurrent Versions System + git git + hg Mercurial + scp remote files via Secure copy + svn Subversion + wget remote files via HTTP download + + -o FILE + Store the downloaded archive in FILE. + + Exit status: + 0 if OK + !0 in case of error +_EOF_ +} + +trace() { local msg="${1}"; shift; printf "%s: ${msg}" "${my_name}" "${@}"; } +warn() { trace "${@}" >&2; } +errorN() { local ret="${1}"; shift; warn "${@}"; exit ${ret}; } +error() { errorN 1 "${@}"; } + +my_name="${0##*/}" +main "${@}" diff --git a/support/download/wrapper b/support/download/wrapper deleted file mode 100755 index 320a37e..0000000 --- a/support/download/wrapper +++ /dev/null @@ -1,99 +0,0 @@ -#!/usr/bin/env bash - -# This script is a wrapper to the other download helpers. -# Its role is to ensure atomicity when saving downloaded files -# back to BR2_DL_DIR, and not clutter BR2_DL_DIR with partial, -# failed downloads. -# -# Call it with: -# $1: name of the helper (eg. cvs, git, cp...) -# $2: full path to the file in which to save the download -# $*: additional arguments to the helper in $1 -# Environment: -# BUILD_DIR: the path to Buildroot's build dir - -# To avoid cluttering BR2_DL_DIR, we download to a trashable -# location, namely in $(BUILD_DIR). -# Then, we move the downloaded file to a temporary file in the -# same directory as the final output file. -# This allows us to finally atomically rename it to its final -# name. -# If anything goes wrong, we just remove all the temporaries -# created so far. - -# We want to catch any unexpected failure, and exit immediately. -set -e - -helper="${1}" -output="${2}" -shift 2 - -# tmpd is a temporary directory in which helpers may store intermediate -# by-products of the download. -# tmpf is the file in which the helpers should put the downloaded content. -# tmpd is located in $(BUILD_DIR), so as not to clutter the (precious) -# $(BR2_DL_DIR) -# We let the helpers create tmpf, so they are able to set whatever -# permission bits they want (although we're only really interested in -# the executable bit.) -tmpd="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )" -tmpf="${tmpd}/output" - -# Helpers expect to run in a directory that is *really* trashable, so -# they are free to create whatever files and/or sub-dirs they might need. -# Doing the 'cd' here rather than in all helpers is easier. -cd "${tmpd}" - -# If the helper fails, we can just remove the temporary directory to -# remove all the cruft it may have left behind. Then we just exit in -# error too. -if ! "${OLDPWD}/support/download/${helper}" "${tmpf}" "${@}"; then - rm -rf "${tmpd}" - exit 1 -fi - -# cd back to free the temp-dir, so we can remove it later -cd "${OLDPWD}" - -# tmp_output is in the same directory as the final output, so we can -# later move it atomically. -tmp_output="$( mktemp "${output}.XXXXXX" )" - -# 'mktemp' creates files with 'go=-rwx', so the files are not accessible -# to users other than the one doing the download (and root, of course). -# This can be problematic when a shared BR2_DL_DIR is used by different -# users (e.g. on a build server), where all users may write to the shared -# location, since other users would not be allowed to read the files -# another user downloaded. -# So, we restore the 'go' access rights to a more sensible value, while -# still abiding by the current user's umask. We must do that before the -# final 'mv', so just do it now. -# Some helpers (cp and scp) may create executable files, so we need to -# carry the executable bit if needed. -[ -x "${tmpf}" ] && new_mode=755 || new_mode=644 -new_mode=$( printf "%04o" $((0${new_mode} & ~0$(umask))) ) -chmod ${new_mode} "${tmp_output}" - -# We must *not* unlink tmp_output, otherwise there is a small window -# during which another download process may create the same tmp_output -# name (very, very unlikely; but not impossible.) -# Using 'cp' is not reliable, since 'cp' may unlink the destination file -# if it is unable to open it with O_WRONLY|O_TRUNC; see: -# http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html -# Since the destination filesystem can be anything, it might not support -# O_TRUNC, so 'cp' would unlink it first. -# Use 'cat' and append-redirection '>>' to save to the final location, -# since that is the only way we can be 100% sure of the behaviour. -if ! cat "${tmpf}" >>"${tmp_output}"; then - rm -rf "${tmpd}" "${tmp_output}" - exit 1 -fi -rm -rf "${tmpd}" -# tmp_output and output are on the same filesystem, so POSIX guarantees -# that 'mv' is atomic, because it then uses rename() that POSIX mandates -# to be atomic, see: -# http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html -if ! mv "${tmp_output}" "${output}"; then - rm -f "${tmp_output}" - exit 1 -fi