From patchwork Thu Dec 11 22:52:05 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 420278 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from silver.osuosl.org (silver.osuosl.org [140.211.166.136]) by ozlabs.org (Postfix) with ESMTP id 4ADB314010F for ; Fri, 12 Dec 2014 09:52:29 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8F50D33F7C; Thu, 11 Dec 2014 22:52:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n6489cn2SjDT; Thu, 11 Dec 2014 22:52:20 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id F3E4A34007; Thu, 11 Dec 2014 22:52:18 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (silver.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 2F5871BFA34 for ; Thu, 11 Dec 2014 22:52:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 287D833FFC for ; Thu, 11 Dec 2014 22:52:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fABOzg+6TSbs for ; Thu, 11 Dec 2014 22:52:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wg0-f41.google.com (mail-wg0-f41.google.com [74.125.82.41]) by silver.osuosl.org (Postfix) with ESMTPS id 7B03E33FF9 for ; Thu, 11 Dec 2014 22:52:12 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id y19so7744912wgg.0 for ; Thu, 11 Dec 2014 14:52:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=4TR3d+qQHmoMlNux0fGwNtMWKRymZYCKbpfK+iBiYgw=; b=HDoxxcgOivoxSP408zIKUHJ+dqNRXdNtnb3IIHeBxIb3rmCGey4O0i+18CU3vNJnvB 9RUFOlT3e7MsG2CE0aJ2junyKZr3imqsOCWo0NGIOOXwDLK7j1KMZFTYN45VKiskhZQe tB32xsO/LoZs9YabCKD8rDcPeGkymM00F+U2mClFkfXeFACwidkd5+JNN+AVGaVTzdGN mKplEerKE4p4NLppUvNtrQyHEBeafFqSPxsPHXjPxYP91plgfaLFLCg4yGuII6VnDQ88 gpqVE8CGgjgQ4v9FEfA2A5aDe5GDZe9u9LmIjd0bv5Owx8PgdTkDZMRsq5oUSzQ/rPlp KhSg== X-Received: by 10.194.246.130 with SMTP id xw2mr20466458wjc.33.1418338331045; Thu, 11 Dec 2014 14:52:11 -0800 (PST) Received: from gourin.bzh.lan (ks3095497.kimsufi.com. [94.23.60.27]) by mx.google.com with ESMTPSA id l3sm3387299wje.12.2014.12.11.14.52.09 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 11 Dec 2014 14:52:10 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Thu, 11 Dec 2014 23:52:05 +0100 Message-Id: <24f42401feff49628acda6ec948ee2a486e73d91.1418337531.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 1.9.1 In-Reply-To: References: Cc: Thomas Petazzoni , "Yann E. MORIN" Subject: [Buildroot] [PATCH 1/4 v5] support/download: add option parsing to the download wrapper X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Instead of relying on argument ordering, use actual options in the download wrapper. Download backends (bzr, cp, hg...) are left as-is, because it does not make sense to complexify them, since they are almost very trivial shell scripts, and adding option parsing would be really overkill. This commit also renames the script to dl-wrapper so it looks better in the traces, and it is not confused with another wrapper. Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni --- package/pkg-download.mk | 42 ++++++----- support/download/dl-wrapper | 165 ++++++++++++++++++++++++++++++++++++++++++++ support/download/wrapper | 99 -------------------------- 3 files changed, 191 insertions(+), 115 deletions(-) create mode 100755 support/download/dl-wrapper delete mode 100755 support/download/wrapper diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 190b5b7..8424eca 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -18,6 +18,8 @@ export SCP := $(call qstrip,$(BR2_SCP)) $(QUIET) SSH := $(call qstrip,$(BR2_SSH)) $(QUIET) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) +DL_WRAPPER = support/download/dl-wrapper + # Default spider mode is 'DOWNLOAD'. Other possible values are 'SOURCE_CHECK' # used by the _source-check target and 'SHOW_EXTERNAL_DEPS', used by the # external-deps target. @@ -95,8 +97,9 @@ endef # problems define DOWNLOAD_GIT test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper git \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b git \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -115,8 +118,9 @@ endef define DOWNLOAD_BZR test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper bzr \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b bzr \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -132,8 +136,9 @@ endef define DOWNLOAD_CVS test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper cvs \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b cvs \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \ $($(PKG)_DL_VERSION) \ $($(PKG)_RAWNAME) \ @@ -151,8 +156,9 @@ endef define DOWNLOAD_SVN test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper svn \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b svn \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -171,8 +177,9 @@ endef # to prepend the path with a slash: scp://[user@]host:/absolutepath define DOWNLOAD_SCP test -e $(DL_DIR)/$(2) || \ - $(EXTRA_ENV) support/download/wrapper scp \ - $(DL_DIR)/$(2) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b scp \ + -o $(DL_DIR)/$(2) \ + -- \ '$(call stripurischeme,$(call qstrip,$(1)))' && \ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_RAWNAME).hash,$(DL_DIR)/$(2)) endef @@ -188,8 +195,9 @@ endef define DOWNLOAD_HG test -e $(DL_DIR)/$($(PKG)_SOURCE) || \ - $(EXTRA_ENV) support/download/wrapper hg \ - $(DL_DIR)/$($(PKG)_SOURCE) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b hg \ + -o $(DL_DIR)/$($(PKG)_SOURCE) \ + -- \ $($(PKG)_SITE) \ $($(PKG)_DL_VERSION) \ $($(PKG)_BASE_NAME) @@ -208,8 +216,9 @@ endef define DOWNLOAD_WGET test -e $(DL_DIR)/$(2) || \ - $(EXTRA_ENV) support/download/wrapper wget \ - $(DL_DIR)/$(2) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b wget \ + -o $(DL_DIR)/$(2) \ + -- \ '$(call qstrip,$(1))' && \ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_RAWNAME).hash,$(DL_DIR)/$(2)) endef @@ -224,8 +233,9 @@ endef define DOWNLOAD_LOCALFILES test -e $(DL_DIR)/$(2) || \ - $(EXTRA_ENV) support/download/wrapper cp \ - $(DL_DIR)/$(2) \ + $(EXTRA_ENV) $(DL_WRAPPER) -b cp \ + -o $(DL_DIR)/$(2) \ + -- \ $(call stripurischeme,$(call qstrip,$(1))) && \ $(call VERIFY_HASH,$(PKGDIR)/$($(PKG)_RAWNAME).hash,$(DL_DIR)/$(2)) endef diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper new file mode 100755 index 0000000..f1bb73a --- /dev/null +++ b/support/download/dl-wrapper @@ -0,0 +1,165 @@ +#!/usr/bin/env bash + +# This script is a wrapper to the other download backends. +# Its role is to ensure atomicity when saving downloaded files +# back to BR2_DL_DIR, and not clutter BR2_DL_DIR with partial, +# failed downloads. +# +# Call it with -h to see some help. + +# To avoid cluttering BR2_DL_DIR, we download to a trashable +# location, namely in $(BUILD_DIR). +# Then, we move the downloaded file to a temporary file in the +# same directory as the final output file. +# This allows us to finally atomically rename it to its final +# name. +# If anything goes wrong, we just remove all the temporaries +# created so far. + +# We want to catch any unexpected failure, and exit immediately. +set -e + +main() { + local OPT OPTARG + local backend output + + # Parse our options; anything after '--' is for the backend + while getopts :hb:o: OPT; do + case "${OPT}" in + h) help; exit 0;; + b) backend="${OPTARG}";; + o) output="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac + done + # Forget our options, and keep only those for the backend + shift $((OPTIND-1)) + + if [ -z "${backend}" ]; then + error "no backend specified, use -b\n" + fi + if [ -z "${output}" ]; then + error "no output specified, use -o\n" + fi + + # tmpd is a temporary directory in which backends may store intermediate + # by-products of the download. + # tmpf is the file in which the backends should put the downloaded content. + # tmpd is located in $(BUILD_DIR), so as not to clutter the (precious) + # $(BR2_DL_DIR) + # We let the backends create tmpf, so they are able to set whatever + # permission bits they want (although we're only really interested in + # the executable bit.) + tmpd="$(mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX")" + tmpf="${tmpd}/output" + + # Helpers expect to run in a directory that is *really* trashable, so + # they are free to create whatever files and/or sub-dirs they might need. + # Doing the 'cd' here rather than in all backends is easier. + cd "${tmpd}" + + # If the backend fails, we can just remove the temporary directory to + # remove all the cruft it may have left behind. Then we just exit in + # error too. + if ! "${OLDPWD}/support/download/${backend}" "${tmpf}" "${@}"; then + rm -rf "${tmpd}" + exit 1 + fi + + # cd back to free the temp-dir, so we can remove it later + cd "${OLDPWD}" + + # tmp_output is in the same directory as the final output, so we can + # later move it atomically. + tmp_output="$(mktemp "${output}.XXXXXX")" + + # 'mktemp' creates files with 'go=-rwx', so the files are not accessible + # to users other than the one doing the download (and root, of course). + # This can be problematic when a shared BR2_DL_DIR is used by different + # users (e.g. on a build server), where all users may write to the shared + # location, since other users would not be allowed to read the files + # another user downloaded. + # So, we restore the 'go' access rights to a more sensible value, while + # still abiding by the current user's umask. We must do that before the + # final 'mv', so just do it now. + # Some backends (cp and scp) may create executable files, so we need to + # carry the executable bit if needed. + [ -x "${tmpf}" ] && new_mode=755 || new_mode=644 + new_mode=$(printf "%04o" $((0${new_mode} & ~0$(umask)))) + chmod ${new_mode} "${tmp_output}" + + # We must *not* unlink tmp_output, otherwise there is a small window + # during which another download process may create the same tmp_output + # name (very, very unlikely; but not impossible.) + # Using 'cp' is not reliable, since 'cp' may unlink the destination file + # if it is unable to open it with O_WRONLY|O_TRUNC; see: + # http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html + # Since the destination filesystem can be anything, it might not support + # O_TRUNC, so 'cp' would unlink it first. + # Use 'cat' and append-redirection '>>' to save to the final location, + # since that is the only way we can be 100% sure of the behaviour. + if ! cat "${tmpf}" >>"${tmp_output}"; then + rm -rf "${tmpd}" "${tmp_output}" + exit 1 + fi + rm -rf "${tmpd}" + + # tmp_output and output are on the same filesystem, so POSIX guarantees + # that 'mv' is atomic, because it then uses rename() that POSIX mandates + # to be atomic, see: + # http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html + if ! mv -f "${tmp_output}" "${output}"; then + rm -f "${tmp_output}" + exit 1 + fi +} + +help() { + cat <<_EOF_ +NAME + ${my_name} - download wrapper for Buildroot + +SYNOPSIS + ${my_name} [OPTION]... -- [BACKEND OPTION]... + +DESCRIPTION + Wrapper script around different download mechanisms. Ensures that + concurrent downloads do not conflict, that partial downloads are + properly evicted without leaving temporary files, and that access + rights are maintained. + + -h This help text. + + -b BACKEND + Wrap the specified BACKEND. Known backends are: + bzr Bazaar + cp Local files + cvs Concurrent Versions System + git Git + hg Mercurial + scp Secure copy + svn Subversion + wget HTTP download + + -o FILE + Store the downloaded archive in FILE. + + Exit status: + 0 if OK + !0 in case of error + +ENVIRONMENT + + BUILD_DIR + The path to Buildroot's build dir +_EOF_ +} + +trace() { local msg="${1}"; shift; printf "%s: ${msg}" "${my_name}" "${@}"; } +warn() { trace "${@}" >&2; } +errorN() { local ret="${1}"; shift; warn "${@}"; exit ${ret}; } +error() { errorN 1 "${@}"; } + +my_name="${0##*/}" +main "${@}" diff --git a/support/download/wrapper b/support/download/wrapper deleted file mode 100755 index 320a37e..0000000 --- a/support/download/wrapper +++ /dev/null @@ -1,99 +0,0 @@ -#!/usr/bin/env bash - -# This script is a wrapper to the other download helpers. -# Its role is to ensure atomicity when saving downloaded files -# back to BR2_DL_DIR, and not clutter BR2_DL_DIR with partial, -# failed downloads. -# -# Call it with: -# $1: name of the helper (eg. cvs, git, cp...) -# $2: full path to the file in which to save the download -# $*: additional arguments to the helper in $1 -# Environment: -# BUILD_DIR: the path to Buildroot's build dir - -# To avoid cluttering BR2_DL_DIR, we download to a trashable -# location, namely in $(BUILD_DIR). -# Then, we move the downloaded file to a temporary file in the -# same directory as the final output file. -# This allows us to finally atomically rename it to its final -# name. -# If anything goes wrong, we just remove all the temporaries -# created so far. - -# We want to catch any unexpected failure, and exit immediately. -set -e - -helper="${1}" -output="${2}" -shift 2 - -# tmpd is a temporary directory in which helpers may store intermediate -# by-products of the download. -# tmpf is the file in which the helpers should put the downloaded content. -# tmpd is located in $(BUILD_DIR), so as not to clutter the (precious) -# $(BR2_DL_DIR) -# We let the helpers create tmpf, so they are able to set whatever -# permission bits they want (although we're only really interested in -# the executable bit.) -tmpd="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )" -tmpf="${tmpd}/output" - -# Helpers expect to run in a directory that is *really* trashable, so -# they are free to create whatever files and/or sub-dirs they might need. -# Doing the 'cd' here rather than in all helpers is easier. -cd "${tmpd}" - -# If the helper fails, we can just remove the temporary directory to -# remove all the cruft it may have left behind. Then we just exit in -# error too. -if ! "${OLDPWD}/support/download/${helper}" "${tmpf}" "${@}"; then - rm -rf "${tmpd}" - exit 1 -fi - -# cd back to free the temp-dir, so we can remove it later -cd "${OLDPWD}" - -# tmp_output is in the same directory as the final output, so we can -# later move it atomically. -tmp_output="$( mktemp "${output}.XXXXXX" )" - -# 'mktemp' creates files with 'go=-rwx', so the files are not accessible -# to users other than the one doing the download (and root, of course). -# This can be problematic when a shared BR2_DL_DIR is used by different -# users (e.g. on a build server), where all users may write to the shared -# location, since other users would not be allowed to read the files -# another user downloaded. -# So, we restore the 'go' access rights to a more sensible value, while -# still abiding by the current user's umask. We must do that before the -# final 'mv', so just do it now. -# Some helpers (cp and scp) may create executable files, so we need to -# carry the executable bit if needed. -[ -x "${tmpf}" ] && new_mode=755 || new_mode=644 -new_mode=$( printf "%04o" $((0${new_mode} & ~0$(umask))) ) -chmod ${new_mode} "${tmp_output}" - -# We must *not* unlink tmp_output, otherwise there is a small window -# during which another download process may create the same tmp_output -# name (very, very unlikely; but not impossible.) -# Using 'cp' is not reliable, since 'cp' may unlink the destination file -# if it is unable to open it with O_WRONLY|O_TRUNC; see: -# http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html -# Since the destination filesystem can be anything, it might not support -# O_TRUNC, so 'cp' would unlink it first. -# Use 'cat' and append-redirection '>>' to save to the final location, -# since that is the only way we can be 100% sure of the behaviour. -if ! cat "${tmpf}" >>"${tmp_output}"; then - rm -rf "${tmpd}" "${tmp_output}" - exit 1 -fi -rm -rf "${tmpd}" -# tmp_output and output are on the same filesystem, so POSIX guarantees -# that 'mv' is atomic, because it then uses rename() that POSIX mandates -# to be atomic, see: -# http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html -if ! mv "${tmp_output}" "${output}"; then - rm -f "${tmp_output}" - exit 1 -fi