[03/13] download: put most of the infra in dl-wrapper

Submitted by Maxime Hadjinlian on July 4, 2017, 4:22 p.m.

Details

Message ID 20170704162211.13238-4-maxime.hadjinlian@gmail.com
State New
Headers show

Commit Message

Maxime Hadjinlian July 4, 2017, 4:22 p.m.
The goal here is to simplify the infrastructure by putting most of the
code in the dl-wrapper as it's easier to implement and to read.

Most of the function where common already, this patch finalize it by
making the pkg-download.mk pass all the parameters needed to the
dl-wrapper which in turns will pass everything to every backend.

The backend will then cherry-pick what it needs from theses arguments
and act accordingly.

It eases the transition to the addition of a sub directory per package
in the DL_DIR, and later on, a git cache.

Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
---
 package/pkg-download.mk     | 166 ++++++++------------------------------------
 support/download/bzr        |   2 +-
 support/download/cp         |   2 +-
 support/download/cvs        |   2 +-
 support/download/dl-wrapper |  65 ++++++++++-------
 support/download/wget       |   7 +-
 6 files changed, 77 insertions(+), 167 deletions(-)

Comments

Arnout Vandecappelle July 5, 2017, 12:32 a.m.
[Thomas, question for you in the middle]

On 04-07-17 18:22, Maxime Hadjinlian wrote:
> The goal here is to simplify the infrastructure by putting most of the
> code in the dl-wrapper as it's easier to implement and to read.

 Yes! I like it!

> 
> Most of the function where common already, this patch finalize it by
> making the pkg-download.mk pass all the parameters needed to the
> dl-wrapper which in turns will pass everything to every backend.
> 
> The backend will then cherry-pick what it needs from theses arguments
> and act accordingly.
> 
> It eases the transition to the addition of a sub directory per package
> in the DL_DIR, and later on, a git cache.
> 
> Signed-off-by: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> ---
>  package/pkg-download.mk     | 166 ++++++++------------------------------------

 This huge diff could have been simplified a little by

1. Removing the third argument to DOWNLOAD_INNER (could actually be part of the
first patch of this series).
2. Replacing the calls to the DOWNLOAD_* helpers with calls to dl-wrapper - the
same patch can also replace the case with "-b $($(PKG)_SITE_METHOD)".
3. Replacing the entire DOWNLOAD_INNER with a single call to dl-wrapper.

 This can stay as a single big patch but it's quite tricky to review it...

>  support/download/bzr        |   2 +-
>  support/download/cp         |   2 +-
>  support/download/cvs        |   2 +-
>  support/download/dl-wrapper |  65 ++++++++++-------
>  support/download/wget       |   7 +-
>  6 files changed, 77 insertions(+), 167 deletions(-)
> 

[snip]
> +ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
> +DOWNLOAD_URIS += \
> +	-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))

 I was at first not so enthusiastic about this URI list and was going to suggest
passing BR2_PRIMARY_SITE into dl-wrapper instead, but the loop in dl-wrapper is
in fact quite elegant.

> +endif
>  
> -define DOWNLOAD_CVS
> -	$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
> -		-o $(DL_DIR)/$($(PKG)_SOURCE) \
> -		$(QUIET) \
> -		-- \
> -		-u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
> -		-c $($(PKG)_DL_VERSION) \
> -		-N $($(PKG)_RAWNAME) \
> -		-n $($(PKG)_RAW_BASE_NAME) \
> -		$($(PKG)_DL_OPTS)
> -endef
> +ifeq ($(BR2_PRIMARY_SITE_ONLY),)
> +DOWNLOAD_URIS += \
> +	-u $($(PKG)_SITE_METHOD)+$(dir $(1))

 Although I find this solution pretty elegant, Thomas was a bit confused by it
and I understand his reasoning. So to make things explicit, and in line with how
we do things elsewhere:

ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
DOWNLOAD_URI_PRIMARY = -u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
endif
ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
DOWNLOAD_URI_SECONDARY = -u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
endif

and in the call itself:

	$(DOWNLOAD_URI_PRIMARY) \
	$(if $(BR2_PRIMARY_SITE_ONLY),,\
		-u $($(PKG)_SITE_METHOD)+$(dir $(1)) \
		$(DOWNLOAD_URI_SECONDARY)) \

 Thomas, do you think this is clearer?

> +ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
> +DOWNLOAD_URIS += \
> +	-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
> +endif
> +endif
>  
> -define DOWNLOAD_SVN
> -	$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
> -		-o $(DL_DIR)/$($(PKG)_SOURCE) \
> -		$(QUIET) \
> -		-- \
> -		-u $($(PKG)_SITE) \
> +define DOWNLOAD
> +	$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \

 No need for the export, there is just one command.

 Not related to this patch, but what we do here is actually wrong: when a
package uses one of these download methods, *everything* related to that package
including patches etc. will not be checked... However, apparently we simply
don't support downloading anything extra for the VCS download methods, so it
doesn't matter.

> +	$(EXTRA_ENV) $(DL_WRAPPER) \
>  		-c $($(PKG)_DL_VERSION) \
[snip]
> diff --git a/support/download/cp b/support/download/cp
> index 52fe2de83d..9c64b7b70b 100755
> --- a/support/download/cp
> +++ b/support/download/cp
> @@ -23,7 +23,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
>      case "${OPT}" in
>      q)  verbose=;;
>      o)  output="${OPTARG}";;
> -    u)  source="${OPTARG}";;
> +    u)  source="${OPTARG#*//}";;
                            ^^^ *://

 This stuff would also be easier to understand if it was introduced in a
separate patch. Basically, previously it was called with
-u $(call stripurischeme,$(call qstrip,$(1)))
and now it is called with
-u '$(call qstrip,$(1))'
so we still need to strip the urischeme.


>      :)  printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
>      \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
>      esac
> diff --git a/support/download/cvs b/support/download/cvs
> index 69d5c71f28..9caffb4b82 100755
> --- a/support/download/cvs
> +++ b/support/download/cvs
> @@ -21,7 +21,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
>      case "${OPT}" in
>      q)  verbose=-Q;;
>      o)  output="${OPTARG}";;
> -    u)  uri="${OPTARG}";;
> +    u)  uri="${OPTARG#*//}";;
>      c)  rev="${OPTARG}";;
>      N)  rawname="${OPTARG}";;
>      n)  basename="${OPTARG}";;
> diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
> index a29411e0ae..50c14a2e16 100755
> --- a/support/download/dl-wrapper
> +++ b/support/download/dl-wrapper
> @@ -19,31 +19,34 @@
>  # We want to catch any unexpected failure, and exit immediately.
>  set -e
>  
> -export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
> +export BR_BACKEND_DL_GETOPTS=":hc:o:H:n:N:ru:qf:"

 Something not right here: you're not adding any options to the download
helpers, so this shouldn't change...

 Options that are only valid for the wrapper and not for the helper should not
be included here since they won't be passed down.

>  
>  main() {
>      local OPT OPTARG
>      local backend output hfile recurse quiet
> +    local -a uris
>  
>      # Parse our options; anything after '--' is for the backend
> -    while getopts :hb:o:H:rq OPT; do
> +    while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
>          case "${OPT}" in
>          h)  help; exit 0;;
> -        b)  backend="${OPTARG}";;
> +        c)  cset="${OPTARG}";;
>          o)  output="${OPTARG}";;
> +        n)  raw_base_name="${OPTARG}";;
> +        N)  base_name="${OPTARG}";;
>          H)  hfile="${OPTARG}";;
>          r)  recurse="-r";;
> +        f)  filename="${OPTARG}";;
> +        u)  uris+=( "${OPTARG}" );;
>          q)  quiet="-q";;
>          :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
>          \?) error "unknown option '%s'\n" "${OPTARG}";;
>          esac
>      done
> +
>      # Forget our options, and keep only those for the backend
>      shift $((OPTIND-1))
>  
> -    if [ -z "${backend}" ]; then
> -        error "no backend specified, use -b\n"
> -    fi
>      if [ -z "${output}" ]; then
>          error "no output specified, use -o\n"
>      fi
> @@ -82,16 +85,29 @@ main() {
>      # Doing the 'cd' here rather than in all backends is easier.
>      cd "${tmpd}"
>  
> -    # If the backend fails, we can just remove the temporary directory to
> -    # remove all the cruft it may have left behind. Then we just exit in
> -    # error too.
> -    if ! "${OLDPWD}/support/download/${backend}" \
> -            ${quiet} ${recurse} \
> -            -o "${tmpf}" "${@}"
> -    then
> -        rm -rf "${tmpd}"
> -        exit 1
> -    fi
> +    # Look through all the uris that we were given to downoad the package
> +    # source
> +    for uri in "${uris[@]}"; do
> +        backend=${uri%+*}
> +        case "${backend}" in
> +            git|svn|cvs|bzr|file|scp|hg) ;;

 The backend for "file" is called "cp", so probably the easiest is to rename
that one to "file".

> +            *) backend="wget" ;;
> +        esac
> +        uri=${uri#*+}
> +
> +        # If the backend fails, we can just remove the temporary directory to
> +        # remove all the cruft it may have left behind. Then we just exit in

 No, we don't exit any more.

> +        # error too.
> +        if "${OLDPWD}/support/download/${backend}" \
> +                -c "${cset}" -n "${raw_base_name}" \
> +                -N "${raw_name}" -f "${filename}" -u "${uri}" -o "${tmpf}" \
> +                ${quiet} ${recurse} "${@}"
> +        then
> +            break

 Wrong, see below.

> +        else
> +            rm -rf "${tmpd:?}/*"

 There should be a continue here, see below.

> +        fi

 We have to do the hash check within the loop - if it fails, we should continue
with the next URI. We should break if the hash check is successful. And we
should keep track somewhere if any download+check was successful, so we can give
the appropriate exit code.

> +    done
>  
>      # cd back to free the temp-dir, so we can remove it later
>      cd "${OLDPWD}"
> @@ -164,16 +180,13 @@ DESCRIPTION
>  
>      -h  This help text.
>  
> -    -b BACKEND
> -        Wrap the specified BACKEND. Known backends are:
> -            bzr     Bazaar
> -            cp      Local files
> -            cvs     Concurrent Versions System
> -            git     Git
> -            hg      Mercurial
> -            scp     Secure copy
> -            svn     Subversion
> -            wget    HTTP download
> +    -u URIs
> +        The URI to get the file from, the URI must respect the format given in
> +        the example.
> +        You may give as many '-u URI' as you want, the script will stop at the
> +        frist successful download.
> +
> +        Example: backend+URI; git+http://example.com or http+http://example.com
>  
>      -o FILE
>          Store the downloaded archive in FILE.
> diff --git a/support/download/wget b/support/download/wget
> index fece6663ca..3e6a6b446c 100755
> --- a/support/download/wget
> +++ b/support/download/wget
> @@ -8,6 +8,7 @@ set -e
>  # Options:
>  #   -q          Be quiet.
>  #   -o FILE     Save into file FILE.
> +#   -f FILENAME The filename of the tarball to get at URL
>  #   -u URL      Download file at URL.
>  #
>  # Environment:
> @@ -18,6 +19,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
>      case "${OPT}" in
>      q)  verbose=-q;;
>      o)  output="${OPTARG}";;
> +    f)  filename="${OPTARG}";;
>      u)  url="${OPTARG}";;
>      :)  printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
>      \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
> @@ -32,4 +34,7 @@ _wget() {
>      eval ${WGET} "${@}"
>  }
>  
> -_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
> +# Replace every '?' with '%3F' in the URI
> +url=${url//\?/%3F}

 We *don't* do that for the normal download, only for the primary/secondary.
Indeed, doing this will fail the download of musl-compat-headers. Not sure how
to solve it elegantly.

 Oh, hang on, this is completely wrong: you need to do this substitution in the
filename, not in the url. The url normally doesn't have any ?, and if it does it
should be kept. The subsitution is there to make sure something like
musl-compat-headers, which has a filename like "queue.h?rev=1.70", can still be
downloaded from primary/secondary. But again, you have to distinguish between
the primary/secondary case and the normal case.

 Regards,
 Arnout

> +
> +_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"
>

Patch hide | download patch | download mbox

diff --git a/package/pkg-download.mk b/package/pkg-download.mk
index ce069b9926..be61fb130a 100644
--- a/package/pkg-download.mk
+++ b/package/pkg-download.mk
@@ -42,6 +42,8 @@  DL_DIR := $(shell mkdir -p $(DL_DIR) && cd $(DL_DIR) >/dev/null && pwd)
 #
 # geturischeme: http
 geturischeme = $(firstword $(subst ://, ,$(call qstrip,$(1))))
+# getschemeplusuri: git+http://example.com
+getschemeplusuri = $(call geturischeme,$(1))+$(1)
 # stripurischeme: www.example.com/dir/file
 stripurischeme = $(lastword $(subst ://, ,$(call qstrip,$(1))))
 # domain: www.example.com
@@ -61,152 +63,42 @@  github = https://github.com/$(1)/$(2)/archive/$(3)
 export BR_NO_CHECK_HASH_FOR =
 
 ################################################################################
-# The DOWNLOAD_* helpers are in charge of getting a working copy
-# of the source repository for their corresponding SCM,
-# checking out the requested version / commit / tag, and create an
-# archive out of it. DOWNLOAD_SCP uses scp to obtain a remote file with
-# ssh authentication. DOWNLOAD_WGET is the normal wget-based download
-# mechanism.
+# DOWNLOAD -- Download helper. Will call DL_WRAPPER which will try to download
+# source from:
+# 1) BR2_PRIMARY_SITE if enabled
+# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
+# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
+#
+# Argument 1 is the source location
 #
 ################################################################################
 
-define DOWNLOAD_GIT
-	$(EXTRA_ENV) $(DL_WRAPPER) -b git \
-		-o $(DL_DIR)/$($(PKG)_SOURCE) \
-		$(if $($(PKG)_GIT_SUBMODULES),-r) \
-		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
-		$(QUIET) \
-		-- \
-		-u $($(PKG)_SITE) \
-		-c $($(PKG)_DL_VERSION) \
-		-n $($(PKG)_RAW_BASE_NAME) \
-		$($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_BZR
-	$(EXTRA_ENV) $(DL_WRAPPER) -b bzr \
-		-o $(DL_DIR)/$($(PKG)_SOURCE) \
-		$(QUIET) \
-		-- \
-		-u $($(PKG)_SITE) \
-		-c $($(PKG)_DL_VERSION) \
-		-n $($(PKG)_RAW_BASE_NAME) \
-		$($(PKG)_DL_OPTS)
-endef
+ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),)
+DOWNLOAD_URIS += \
+	-u $(call getschemeplusuri,$(BR2_PRIMARY_SITE))
+endif
 
-define DOWNLOAD_CVS
-	$(EXTRA_ENV) $(DL_WRAPPER) -b cvs \
-		-o $(DL_DIR)/$($(PKG)_SOURCE) \
-		$(QUIET) \
-		-- \
-		-u $(call stripurischeme,$(call qstrip,$($(PKG)_SITE))) \
-		-c $($(PKG)_DL_VERSION) \
-		-N $($(PKG)_RAWNAME) \
-		-n $($(PKG)_RAW_BASE_NAME) \
-		$($(PKG)_DL_OPTS)
-endef
+ifeq ($(BR2_PRIMARY_SITE_ONLY),)
+DOWNLOAD_URIS += \
+	-u $($(PKG)_SITE_METHOD)+$(dir $(1))
+ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
+DOWNLOAD_URIS += \
+	-u $(call getschemeplusuri,$(BR2_BACKUP_SITE))
+endif
+endif
 
-define DOWNLOAD_SVN
-	$(EXTRA_ENV) $(DL_WRAPPER) -b svn \
-		-o $(DL_DIR)/$($(PKG)_SOURCE) \
-		$(QUIET) \
-		-- \
-		-u $($(PKG)_SITE) \
+define DOWNLOAD
+	$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(notdir $(1));) \
+	$(EXTRA_ENV) $(DL_WRAPPER) \
 		-c $($(PKG)_DL_VERSION) \
-		-n $($(PKG)_RAW_BASE_NAME) \
-		$($(PKG)_DL_OPTS)
-endef
-
-# SCP URIs should be of the form scp://[user@]host:filepath
-# Note that filepath is relative to the user's home directory, so you may want
-# to prepend the path with a slash: scp://[user@]host:/absolutepath
-define DOWNLOAD_SCP
-	$(EXTRA_ENV) $(DL_WRAPPER) -b scp \
-		-o $(DL_DIR)/$(2) \
+		-f $(notdir $(1)) \
 		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
-		$(QUIET) \
-		-- \
-		-u '$(call stripurischeme,$(call qstrip,$(1)))' \
-		$($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_HG
-	$(EXTRA_ENV) $(DL_WRAPPER) -b hg \
-		-o $(DL_DIR)/$($(PKG)_SOURCE) \
-		$(QUIET) \
-		-- \
-		-u $($(PKG)_SITE) \
-		-c $($(PKG)_DL_VERSION) \
 		-n $($(PKG)_RAW_BASE_NAME) \
-		$($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_WGET
-	$(EXTRA_ENV) $(DL_WRAPPER) -b wget \
-		-o $(DL_DIR)/$(2) \
-		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
-		$(QUIET) \
-		-- \
-		-u '$(call qstrip,$(1))' \
-		$($(PKG)_DL_OPTS)
-endef
-
-define DOWNLOAD_LOCALFILES
-	$(EXTRA_ENV) $(DL_WRAPPER) -b cp \
-		-o $(DL_DIR)/$(2) \
-		-H $(PKGDIR)/$($(PKG)_RAWNAME).hash \
+		-N $($(PKG)_RAWNAME) \
+		-o $(DL_DIR)/$(notdir $(1)) \
+		$(if $($(PKG)_GIT_SUBMODULES),-r) \
+		$(DOWNLOAD_URIS) \
 		$(QUIET) \
 		-- \
-		-u $(call stripurischeme,$(call qstrip,$(1))) \
 		$($(PKG)_DL_OPTS)
 endef
-
-################################################################################
-# DOWNLOAD -- Download helper. Will try to download source from:
-# 1) BR2_PRIMARY_SITE if enabled
-# 2) Download site, unless BR2_PRIMARY_SITE_ONLY is set
-# 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set
-#
-# Argument 1 is the source location
-#
-# E.G. use like this:
-# $(call DOWNLOAD,$(FOO_SITE))
-#
-# For PRIMARY and BACKUP site, any ? in the URL is replaced by %3F. A ? in
-# the URL is used to separate query arguments, but the PRIMARY and BACKUP
-# sites serve just plain files.
-################################################################################
-
-define DOWNLOAD
-	$(call DOWNLOAD_INNER,$(1),$(notdir $(1)),DOWNLOAD)
-endef
-
-define DOWNLOAD_INNER
-	$(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)),export BR_NO_CHECK_HASH_FOR=$(2);) \
-	if test -n "$(call qstrip,$(BR2_PRIMARY_SITE))" ; then \
-		case "$(call geturischeme,$(BR2_PRIMARY_SITE))" in \
-			file) $(call $(3)_LOCALFILES,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
-			scp) $(call $(3)_SCP,$(BR2_PRIMARY_SITE)/$(2),$(2)) && exit ;; \
-			*) $(call $(3)_WGET,$(BR2_PRIMARY_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ;; \
-		esac ; \
-	fi ; \
-	if test "$(BR2_PRIMARY_SITE_ONLY)" = "y" ; then \
-		exit 1 ; \
-	fi ; \
-	if test -n "$(1)" ; then \
-		case "$($(PKG)_SITE_METHOD)" in \
-			git) $($(3)_GIT) && exit ;; \
-			svn) $($(3)_SVN) && exit ;; \
-			cvs) $($(3)_CVS) && exit ;; \
-			bzr) $($(3)_BZR) && exit ;; \
-			file) $($(3)_LOCALFILES) && exit ;; \
-			scp) $($(3)_SCP) && exit ;; \
-			hg) $($(3)_HG) && exit ;; \
-			*) $(call $(3)_WGET,$(1),$(2)) && exit ;; \
-		esac ; \
-	fi ; \
-	if test -n "$(call qstrip,$(BR2_BACKUP_SITE))" ; then \
-		$(call $(3)_WGET,$(BR2_BACKUP_SITE)/$(subst ?,%3F,$(2)),$(2)) && exit ; \
-	fi ; \
-	exit 1
-endef
diff --git a/support/download/bzr b/support/download/bzr
index a70cb19cf1..5289a421cd 100755
--- a/support/download/bzr
+++ b/support/download/bzr
@@ -8,7 +8,7 @@  set -e
 # Options:
 #   -q          Be quiet
 #   -o FILE     Generate archive in FILE.
-#   -u URI      Clone from repository URI.
+#   -u URI      Clone from repository at URI.
 #   -c CSET     Use changeset (or revision) CSET.
 #   -n NAME     Use basename NAME.
 #
diff --git a/support/download/cp b/support/download/cp
index 52fe2de83d..9c64b7b70b 100755
--- a/support/download/cp
+++ b/support/download/cp
@@ -23,7 +23,7 @@  while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=;;
     o)  output="${OPTARG}";;
-    u)  source="${OPTARG}";;
+    u)  source="${OPTARG#*//}";;
     :)  printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
     \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
     esac
diff --git a/support/download/cvs b/support/download/cvs
index 69d5c71f28..9caffb4b82 100755
--- a/support/download/cvs
+++ b/support/download/cvs
@@ -21,7 +21,7 @@  while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-Q;;
     o)  output="${OPTARG}";;
-    u)  uri="${OPTARG}";;
+    u)  uri="${OPTARG#*//}";;
     c)  rev="${OPTARG}";;
     N)  rawname="${OPTARG}";;
     n)  basename="${OPTARG}";;
diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper
index a29411e0ae..50c14a2e16 100755
--- a/support/download/dl-wrapper
+++ b/support/download/dl-wrapper
@@ -19,31 +19,34 @@ 
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
-export BR_BACKEND_DL_GETOPTS=":hb:o:H:rRq"
+export BR_BACKEND_DL_GETOPTS=":hc:o:H:n:N:ru:qf:"
 
 main() {
     local OPT OPTARG
     local backend output hfile recurse quiet
+    local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts :hb:o:H:rq OPT; do
+    while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
         case "${OPT}" in
         h)  help; exit 0;;
-        b)  backend="${OPTARG}";;
+        c)  cset="${OPTARG}";;
         o)  output="${OPTARG}";;
+        n)  raw_base_name="${OPTARG}";;
+        N)  base_name="${OPTARG}";;
         H)  hfile="${OPTARG}";;
         r)  recurse="-r";;
+        f)  filename="${OPTARG}";;
+        u)  uris+=( "${OPTARG}" );;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
         esac
     done
+
     # Forget our options, and keep only those for the backend
     shift $((OPTIND-1))
 
-    if [ -z "${backend}" ]; then
-        error "no backend specified, use -b\n"
-    fi
     if [ -z "${output}" ]; then
         error "no output specified, use -o\n"
     fi
@@ -82,16 +85,29 @@  main() {
     # Doing the 'cd' here rather than in all backends is easier.
     cd "${tmpd}"
 
-    # If the backend fails, we can just remove the temporary directory to
-    # remove all the cruft it may have left behind. Then we just exit in
-    # error too.
-    if ! "${OLDPWD}/support/download/${backend}" \
-            ${quiet} ${recurse} \
-            -o "${tmpf}" "${@}"
-    then
-        rm -rf "${tmpd}"
-        exit 1
-    fi
+    # Look through all the uris that we were given to downoad the package
+    # source
+    for uri in "${uris[@]}"; do
+        backend=${uri%+*}
+        case "${backend}" in
+            git|svn|cvs|bzr|file|scp|hg) ;;
+            *) backend="wget" ;;
+        esac
+        uri=${uri#*+}
+
+        # If the backend fails, we can just remove the temporary directory to
+        # remove all the cruft it may have left behind. Then we just exit in
+        # error too.
+        if "${OLDPWD}/support/download/${backend}" \
+                -c "${cset}" -n "${raw_base_name}" \
+                -N "${raw_name}" -f "${filename}" -u "${uri}" -o "${tmpf}" \
+                ${quiet} ${recurse} "${@}"
+        then
+            break
+        else
+            rm -rf "${tmpd:?}/*"
+        fi
+    done
 
     # cd back to free the temp-dir, so we can remove it later
     cd "${OLDPWD}"
@@ -164,16 +180,13 @@  DESCRIPTION
 
     -h  This help text.
 
-    -b BACKEND
-        Wrap the specified BACKEND. Known backends are:
-            bzr     Bazaar
-            cp      Local files
-            cvs     Concurrent Versions System
-            git     Git
-            hg      Mercurial
-            scp     Secure copy
-            svn     Subversion
-            wget    HTTP download
+    -u URIs
+        The URI to get the file from, the URI must respect the format given in
+        the example.
+        You may give as many '-u URI' as you want, the script will stop at the
+        frist successful download.
+
+        Example: backend+URI; git+http://example.com or http+http://example.com
 
     -o FILE
         Store the downloaded archive in FILE.
diff --git a/support/download/wget b/support/download/wget
index fece6663ca..3e6a6b446c 100755
--- a/support/download/wget
+++ b/support/download/wget
@@ -8,6 +8,7 @@  set -e
 # Options:
 #   -q          Be quiet.
 #   -o FILE     Save into file FILE.
+#   -f FILENAME The filename of the tarball to get at URL
 #   -u URL      Download file at URL.
 #
 # Environment:
@@ -18,6 +19,7 @@  while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do
     case "${OPT}" in
     q)  verbose=-q;;
     o)  output="${OPTARG}";;
+    f)  filename="${OPTARG}";;
     u)  url="${OPTARG}";;
     :)  printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;;
     \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;;
@@ -32,4 +34,7 @@  _wget() {
     eval ${WGET} "${@}"
 }
 
-_wget ${verbose} "${@}" -O "'${output}'" "'${url}'"
+# Replace every '?' with '%3F' in the URI
+url=${url//\?/%3F}
+
+_wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'"