Patchwork [1/9,v3] support/download: add download wrapper

login
register
mail settings
Submitter Yann E. MORIN
Date July 20, 2014, 10:42 p.m.
Message ID <dc7cb9f1441183b56518a81a1ea45974cae07635.1405895896.git.yann.morin.1998@free.fr>
Download mbox | patch
Permalink /patch/371945/
State Changes Requested
Headers show

Comments

Yann E. MORIN - July 20, 2014, 10:42 p.m.
The download wrapper is responsible for ensuring the atomicity
of saving into $(BR2_DL_DIR).

It calls the appropriate download helper, telling it to save the
downloaded content to a temporary file in $(BUILD_DIR) (so it does
not clutter $(BR2_DL_DIR) with partial, failed downloads.

Then, only it the download helper was successful, does the wrapper
saves the downloaded content to the final location, yet still in a
teporary file, and finally atomically renames it to the final output
file.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 support/download/wrapper | 99 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 99 insertions(+)
 create mode 100755 support/download/wrapper
Thomas De Schampheleire - Aug. 3, 2014, 7:18 a.m.
Hi Yann,

"Yann E. MORIN" <yann.morin.1998@free.fr> schreef:
>The download wrapper is responsible for ensuring the atomicity
>of saving into $(BR2_DL_DIR).
>
>It calls the appropriate download helper, telling it to save the
>downloaded content to a temporary file in $(BUILD_DIR) (so it does
>not clutter $(BR2_DL_DIR) with partial, failed downloads.
>
>Then, only it the download helper was successful, does the wrapper

only if

>saves the downloaded content to the final location, yet still in a

does ... save

>teporary file, and finally atomically renames it to the final output

temporary

>file.
>
>Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>---
> support/download/wrapper | 99 ++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 99 insertions(+)
> create mode 100755 support/download/wrapper
>
>diff --git a/support/download/wrapper b/support/download/wrapper
>new file mode 100755
>index 0000000..bc0d486
>--- /dev/null
>+++ b/support/download/wrapper
>@@ -0,0 +1,99 @@
>+#!/bin/bash
>+
>+# This script is a wrapper to the other download helpers.
>+# Its role is to ensure atomicity when saving downloaded files
>+# back to BR2_DL_DIR, and not clutter BR2_DL_DIR with partial,
>+# failed downloads.
>+#
>+# Call it with:
>+#   $1: name of the helper (eg. cvs, git, cp...)
>+#   $2: full path to the file in which to save the download
>+#   $*: additional arguments to the helper in $1
>+# Environment:
>+#   BUILD_DIR: the path to Buildroot's build dir
>+
>+# To avoid cluttering BR2_DL_DIR, we download to a trashable
>+# location, namely in $(BUILD_DIR).
>+# Then, we move the downloaed file to a temporary file in the

downloaded

>+# same directory as the final output file.
>+# This allows us to finally atomically rename it to its final
>+# name.
>+# If anything goes wrong, we just remove all the temporaries
>+# created so far.
>+
>+# We want to catch any unexpected failure, and exit immediately.
>+set -e
>+
>+helper="${1}"
>+output="${2}"
>+shift 2
>+
>+# tmpd is a temporary directory in which helpers may store intermediate
>+# by-products of the download.
>+# tmpf is the file in which the helpers should put the downloaded content.
>+# tmpd located in $(BUILD_DIR), so as not to cluter the (precious)

tmpd is located
clutter

>+# $(BR2_DL_DIR)
>+# We let the helpers create tmpf, so they are able to set whatever
>+# permission bits they want (although we're only really interested in
>+# the executable bit.)
>+tmpd="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )"
>+tmpf="${tmpd}/output"
>+
>+# Helpers expect to run in a directory that is *really* trashable, so
>+# they are free to create whatever files and/or sub-dirs they might need.
>+# Doing the 'cd' here rather than in all helpers is easier.
>+cd "${tmpd}"
>+
>+# If the helper fails, we can just remove the temporary directory to
>+# remove all the cruft it may have left behind. Then we just exit in
>+# error too.
>+if ! "${OLDPWD}/support/download/${helper}" "${tmpf}" "${@}"; then
>+    rm -rf "${tmpd}"
>+    exit 1
>+fi
>+
>+# cd back to free the temp-dir, so we can remove it later
>+cd "${OLDPWD}"
>+
>+# tmp_output is in the same directory as the final output, so we can
>+# later move it atomically.
>+tmp_output="$( mktemp "${output}.XXXXXX" )"
>+
>+# 'mktemp' creates files with 'go=-rwx', so the files are not accessible
>+# to users other than the one doing the download (and root, of course).
>+# This can be problematic when a shared BR2_DL_DIR is used by different
>+# users (e.g. on a build server), where all users may write to the shared
>+# location, since other users would not be allowed to read the files
>+# another user downloaded.
>+# So, we restore the 'go' access rights to a more sensible value, while
>+# still abiding by the current user's umask. We must do that before the
>+# final 'mv', so just do it now.
>+# Some helpers (cp and scp) may create executable files, so we need to
>+# carry the executable bit if needed.
>+[ -x "${tmpf}" ] && new_mode=755 || new_mode=644
>+new_mode=$( printf "%04o" $((0${new_mode} & ~0$(umask))) )
>+chmod ${new_mode} "${tmp_output}"
>+
>+# We must *not* unlink tmp_output, otherwise there is a small window
>+# during which another download process may create the same tmp_output
>+# name (very, very unlikely; but not impossible.)
>+# Using 'cp' is not reliable, since 'cp' may unlink the destination file
>+# if it is unable to open it with O_WRONLY|O_TRUNC; see:
>+#   http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html
>+# Since the destination filesystem can be anything, it might not support
>+# O_TRUNC, so 'cp' would unlink it first.
>+# Use 'cat' and append-redirection '>>' to save to the final location,
>+# since that is the only way we can be 100% sure of the behaviour.
>+if ! cat "${tmpf}" >>"${tmp_output}"; then
>+    rm -rf "${tmpd}" "${tmp_output}"
>+    exit 1
>+fi
>+rm -rf "${tmpd}"
>+# tmp_output and output are on the same filesystem, so POSIX guarantees
>+# that 'mv' is atomic, because it then uses rename() that POSIX mandates
>+# to be atomic, see:
>+#   http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html
>+if ! mv "${tmp_output}" "${output}"; then
>+    rm -f "${tmp_output}"
>+    exit 1
>+fi


Note that I hope we can apply this series in the current release still...

Best regards,
Thomas
Yann E. MORIN - Aug. 3, 2014, 7:27 a.m.
Thomas, All,

On 2014-08-03 09:18 +0200, Thomas De Schampheleire spake thusly:
> "Yann E. MORIN" <yann.morin.1998@free.fr> schreef:
[--SNIP typoes--]

Typoes fixed, thanks!

> Note that I hope we can apply this series in the current release still...

So do I, so do I! ;-)

Regards,
Yann E. MORIN.

Patch

diff --git a/support/download/wrapper b/support/download/wrapper
new file mode 100755
index 0000000..bc0d486
--- /dev/null
+++ b/support/download/wrapper
@@ -0,0 +1,99 @@ 
+#!/bin/bash
+
+# This script is a wrapper to the other download helpers.
+# Its role is to ensure atomicity when saving downloaded files
+# back to BR2_DL_DIR, and not clutter BR2_DL_DIR with partial,
+# failed downloads.
+#
+# Call it with:
+#   $1: name of the helper (eg. cvs, git, cp...)
+#   $2: full path to the file in which to save the download
+#   $*: additional arguments to the helper in $1
+# Environment:
+#   BUILD_DIR: the path to Buildroot's build dir
+
+# To avoid cluttering BR2_DL_DIR, we download to a trashable
+# location, namely in $(BUILD_DIR).
+# Then, we move the downloaed file to a temporary file in the
+# same directory as the final output file.
+# This allows us to finally atomically rename it to its final
+# name.
+# If anything goes wrong, we just remove all the temporaries
+# created so far.
+
+# We want to catch any unexpected failure, and exit immediately.
+set -e
+
+helper="${1}"
+output="${2}"
+shift 2
+
+# tmpd is a temporary directory in which helpers may store intermediate
+# by-products of the download.
+# tmpf is the file in which the helpers should put the downloaded content.
+# tmpd located in $(BUILD_DIR), so as not to cluter the (precious)
+# $(BR2_DL_DIR)
+# We let the helpers create tmpf, so they are able to set whatever
+# permission bits they want (although we're only really interested in
+# the executable bit.)
+tmpd="$( mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX" )"
+tmpf="${tmpd}/output"
+
+# Helpers expect to run in a directory that is *really* trashable, so
+# they are free to create whatever files and/or sub-dirs they might need.
+# Doing the 'cd' here rather than in all helpers is easier.
+cd "${tmpd}"
+
+# If the helper fails, we can just remove the temporary directory to
+# remove all the cruft it may have left behind. Then we just exit in
+# error too.
+if ! "${OLDPWD}/support/download/${helper}" "${tmpf}" "${@}"; then
+    rm -rf "${tmpd}"
+    exit 1
+fi
+
+# cd back to free the temp-dir, so we can remove it later
+cd "${OLDPWD}"
+
+# tmp_output is in the same directory as the final output, so we can
+# later move it atomically.
+tmp_output="$( mktemp "${output}.XXXXXX" )"
+
+# 'mktemp' creates files with 'go=-rwx', so the files are not accessible
+# to users other than the one doing the download (and root, of course).
+# This can be problematic when a shared BR2_DL_DIR is used by different
+# users (e.g. on a build server), where all users may write to the shared
+# location, since other users would not be allowed to read the files
+# another user downloaded.
+# So, we restore the 'go' access rights to a more sensible value, while
+# still abiding by the current user's umask. We must do that before the
+# final 'mv', so just do it now.
+# Some helpers (cp and scp) may create executable files, so we need to
+# carry the executable bit if needed.
+[ -x "${tmpf}" ] && new_mode=755 || new_mode=644
+new_mode=$( printf "%04o" $((0${new_mode} & ~0$(umask))) )
+chmod ${new_mode} "${tmp_output}"
+
+# We must *not* unlink tmp_output, otherwise there is a small window
+# during which another download process may create the same tmp_output
+# name (very, very unlikely; but not impossible.)
+# Using 'cp' is not reliable, since 'cp' may unlink the destination file
+# if it is unable to open it with O_WRONLY|O_TRUNC; see:
+#   http://pubs.opengroup.org/onlinepubs/9699919799/utilities/cp.html
+# Since the destination filesystem can be anything, it might not support
+# O_TRUNC, so 'cp' would unlink it first.
+# Use 'cat' and append-redirection '>>' to save to the final location,
+# since that is the only way we can be 100% sure of the behaviour.
+if ! cat "${tmpf}" >>"${tmp_output}"; then
+    rm -rf "${tmpd}" "${tmp_output}"
+    exit 1
+fi
+rm -rf "${tmpd}"
+# tmp_output and output are on the same filesystem, so POSIX guarantees
+# that 'mv' is atomic, because it then uses rename() that POSIX mandates
+# to be atomic, see:
+#   http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html
+if ! mv "${tmp_output}" "${output}"; then
+    rm -f "${tmp_output}"
+    exit 1
+fi