From patchwork Thu Feb 20 16:01:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Havelange X-Patchwork-Id: 1241518 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=essensium.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=essensium.com header.i=@essensium.com header.a=rsa-sha256 header.s=google header.b=ArHvQt42; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48NfV93cmtz9sPK for ; Fri, 21 Feb 2020 03:01:53 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B7D6B87591; Thu, 20 Feb 2020 16:01:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ItOEXjDAh9xg; Thu, 20 Feb 2020 16:01:48 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id B0569874DA; Thu, 20 Feb 2020 16:01:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 6FF961BF9B3 for ; Thu, 20 Feb 2020 16:01:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 64215203C7 for ; Thu, 20 Feb 2020 16:01:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yVuT8n4gHYXB for ; Thu, 20 Feb 2020 16:01:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by silver.osuosl.org (Postfix) with ESMTPS id 81B4E203BA for ; Thu, 20 Feb 2020 16:01:42 +0000 (UTC) Received: by mail-wm1-f42.google.com with SMTP id a5so2584876wmb.0 for ; Thu, 20 Feb 2020 08:01:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=essensium.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4FggsvQgO2/QlDJW5GlG4PTnz8yLNcOfjExdHichItE=; b=ArHvQt42yMarfy1sikxi+UgyMVCNpAngJKiwNLcyYzgGaptLkDsAeeXKwl0TVtGldD l3NwHxxIr9/LZ//MSS3+pgdp5sAoqOveM4GWt7SKZaHit4CJJ11K2UVTxtPpLsbSKcSk Wmd6OGFOAGd22ljhnlo0VBcQDfFMfZtr7HNr+KSCvYgG+d1IAYAEJvYjnIO4onPcZ3eq MRDaBEa06ZLbC0YMFdTOrvDIUbYadrMAY6HDFkkdDasSrmfSXq3trK/HT/dh9HEvH5Zi fegRmEH1fQet+giPMjPAaj+XLxS0Wyv/xr4IjEA+hrA8QzH5RYNV0reXYAk9WGO/i1j4 l6/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4FggsvQgO2/QlDJW5GlG4PTnz8yLNcOfjExdHichItE=; b=Mu4mJ8WZPHtkXzTF0ln0rEERedW/m/MhfUqk3ZV8vwSZSSZYX5mYWA0sDKhT+3B92Y SYPTU/WQqvW0mwqh/dtG/irOpWuDkE/rhF1ww7qi2DQzwInkqMXhiY0+E0aExLD/LJ78 3EGUTD/DB11nzoG781kHZfrBsLAPeLnE1LbVZ70hggHVNiOe6vtJOm48RIUx66c89nk8 dwaVBuJ6vbDZ40HOkQkpKYWdnB0cVuhhgRXI1ScS45Dx4YCFrV++6ohU1mDMfLk/anqb Xm4ot1H/037X/UMDgtmRPImV6CO9uLyU4qZgx/ETwhc7Lj+VzeP9FTD3+D75F2COe2yt JMAg== X-Gm-Message-State: APjAAAUUSgE2xyanvmZzJxcKiFcz80VoF/lha6QKAGyc937+ncpouecH BhRpfsmnFddgKH6wyuO3E6r2egI1diiZ2Q== X-Google-Smtp-Source: APXvYqwL/ANw1RkSqySfTebJ4Sem0WgR/r9tolOPbIvnY2eLB1R4ezTA0elShsp09Y0bHFWUj6TCHQ== X-Received: by 2002:a1c:6588:: with SMTP id z130mr5222398wmb.0.1582214499746; Thu, 20 Feb 2020 08:01:39 -0800 (PST) Received: from ph-ThinkPad-E560.local.ess-mail.com (ip-188-118-3-185.reverse.destiny.be. [188.118.3.185]) by smtp.gmail.com with ESMTPSA id h5sm5483691wmf.8.2020.02.20.08.01.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2020 08:01:39 -0800 (PST) From: Patrick Havelange To: buildroot@buildroot.org Date: Thu, 20 Feb 2020 17:01:13 +0100 Message-Id: <20200220160119.3407-4-patrick.havelange@essensium.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200220160119.3407-1-patrick.havelange@essensium.com> References: <20200220160119.3407-1-patrick.havelange@essensium.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 04/10] support/download/dl-wrapper: rework backend parsing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire , Patrick Havelange , Sam Voss , "Yann E . MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The point of this modification is to be more generic in the way the backends/options are parsed from the urls. The '+' char serves as delimiter between the backends and the actual url. The '|' char serves as separator between the different options or backends. This will be useful later when more options or backends would be needed. pkg-generic.mk now passes the optional FOO_PKGMGR value in case a package needs to use a second backend. Signed-off-by: Patrick Havelange --- package/pkg-generic.mk | 2 +- support/download/dl-wrapper | 34 ++++++++++++++++++++++------------ 2 files changed, 23 insertions(+), 13 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 268d999efb..a5038ebff4 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -541,7 +541,7 @@ ifndef $(2)_PATCH endif $(2)_ALL_DOWNLOADS = \ - $$(if $$($(2)_SOURCE),$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE)) \ + $$(if $$($(2)_SOURCE),$$($(2)_PKGMGR)$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE)) \ $$(foreach p,$$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\ $$(if $$(findstring ://,$$(p)),$$(p),\ $$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$(p))) diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 3315bd410e..3f613bb622 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -21,8 +21,8 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG - local backend output hfile recurse quiet rc - local -a uris + local backend backends_str output hfile recurse quiet rc backend2 + local -a uris backends # Parse our options; anything after '--' is for the backend while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do @@ -85,18 +85,24 @@ main() { download_and_check=0 rc=1 for uri in "${uris[@]}"; do - backend_urlencode="${uri%%+*}" - backend="${backend_urlencode%|*}" - case "${backend}" in - git|svn|cvs|bzr|file|scp|hg) ;; - *) backend="wget" ;; - esac + urlencode="" + backend="wget" + backend2="" + + # Extract the backends list, separated by '+' from the actual url + backends_str="${uri%%+*}" + # make an array out of it, backends are '|' separated + IFS='|' read -r -a backends <<< "${backends_str}" + # parse each of them, set flags and set the real backend + for b in "${backends[@]}" ; do + case "${b}" in + urlencode) urlencode="${b}" ;; + git|svn|cvs|bzr|file|scp|hg) backend="${b}" ;; + # insert here supported second backends) backend2="${b}" ;; + esac + done uri=${uri#*+} - urlencode=${backend_urlencode#*|} - # urlencode must be "urlencode" - [ "${urlencode}" != "urlencode" ] && urlencode="" - # tmpd is a temporary directory in which backends may store # intermediate by-products of the download. # tmpf is the file in which the backends should put the downloaded @@ -138,6 +144,10 @@ main() { # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}" + if [ -n "${backend2}" ] ; then + support/download/${backend2} "${tmpf}" "${tmpd}" "${old_dl_dir}" + fi + # Check if the downloaded file is sane, and matches the stored hashes # for that file if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then