From patchwork Tue Feb 19 10:38:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044604 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="qY58k9Lf"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cf744fNz9s3l for ; Tue, 19 Feb 2019 21:38:39 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 52236866B3; Tue, 19 Feb 2019 10:38:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 48CGNA0YxFYz; Tue, 19 Feb 2019 10:38:34 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 030C2865CD; Tue, 19 Feb 2019 10:38:34 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id ADE811BF283 for ; Tue, 19 Feb 2019 10:38:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A2DF522EE6 for ; Tue, 19 Feb 2019 10:38:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kdm2kqMRz9jh for ; Tue, 19 Feb 2019 10:38:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f45.google.com (mail-ot1-f45.google.com [209.85.210.45]) by silver.osuosl.org (Postfix) with ESMTPS id 545A722EC9 for ; Tue, 19 Feb 2019 10:38:31 +0000 (UTC) Received: by mail-ot1-f45.google.com with SMTP id b3so33349650otp.4 for ; Tue, 19 Feb 2019 02:38:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=IVyrFuPBe0O/ORKv+8Lo8f3grzFcswMo8i8PKmjsL2c=; b=qY58k9LfdU8oIMluHZRxC5v/mI9DnO0cpJPF3x5oN39nfamwASH4kUafega+UUAly7 OUSbOt1I/Q9wNZ3bsXwj6fHbIAAie6IGsrx06z1SP9pGlTiShhYn7IP+JlJzWERGwHmh VVEB7jj7aJ2JHrSboWOnzge8pPoPS25RtzFGOo1vMj5tkT7IxRZlbZt6ml53vw2o3I9V DcH2RTRXRKWu1kyKmuuRI/tR7BfaGAaj+dmWEy7hIiKHSgI0KL5dcM7kHw+rH0wP5o8Q K8IyZ8YOgY8ihaHZT/YNdNzyHLyrKO4M1TbcsuOmFH5od6BW2Ah0PN5gYo8SbUDTir/b grjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=IVyrFuPBe0O/ORKv+8Lo8f3grzFcswMo8i8PKmjsL2c=; b=lVYnAIg9YbKYPLTWXORb0guM42KlSYWULyz/BfsuiC8iRroeU/3oVTi7hvpTigF3KL 2qCsbJBiJmt+LvHC39NvW/y+f1h8Sc+WrHmUiYMRRCcAc1jc7m6KBJ4bJUfFe6+6ZL7o g6u/xqquiLkT0vs0cbwSaCBskGjbvHoSAsNFdnS9sa3B6pCsCtcFg96UPdc9jwnkdB8k evl9FatkPGo+ziYO0QhgZT38Q+ZaqHzRRKZoILupFxS4Kv/SVPZhHEkccO37Hr25ZEmj LuX4WeNs9q8c+cyNPA6qZ6sDWL6WIlXqBfWsdNrVFcPaTLEE2oNKI1CeAXUrh9MH1PzF o6oQ== X-Gm-Message-State: AHQUAua63EpxtrQucclsTPoPj1pSr1EM5sNdm55GcQNmWHtuHRkz6fji CiB2vveUgtXkCTmfmpCGE4tv8iK+d7c= X-Google-Smtp-Source: AHgI3IZgOY68A4ajuSlL9j+3Us4BrtHj61qgn3VCnyKGMVQB5FBujO1hyawa2G8GsVao7i79pc91xg== X-Received: by 2002:aca:c307:: with SMTP id t7mr2169725oif.46.1550572709996; Tue, 19 Feb 2019 02:38:29 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id x127sm4745836oix.25.2019.02.19.02.38.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:29 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:31 +0100 Message-Id: <20190219103839.25409-1-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 1/9] support/download: reintroduce 'source-check' target X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Commit bf28a165d992681a85b43a886005e669b3cb579b removed the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target. However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal BR2_PRIMARY_SITE or (in case of a repository) pushing the new changesets to the repository. If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth. With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on). Subsequent commits will actually implement -C in the backends. Signed-off-by: Thomas De Schampheleire --- Makefile | 7 +++ package/pkg-download.mk | 6 +++ package/pkg-generic.mk | 14 ++++- support/download/dl-wrapper | 100 ++++++++++++++++++++---------------- 4 files changed, 82 insertions(+), 45 deletions(-) diff --git a/Makefile b/Makefile index f736ecfb3e..de1ffa9d0e 100644 --- a/Makefile +++ b/Makefile @@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf # (default target is to build), or when MAKECMDGOALS contains # something else than one of the nobuild_targets. nobuild_targets := source %-source \ + source-check %-source-check %-all-source-check \ legal-info %-legal-info external-deps _external-deps \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \ @@ -818,6 +819,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize .PHONY: source source: $(foreach p,$(PACKAGES),$(p)-all-source) +.PHONY: source-check +source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check) + .PHONY: _external-deps external-deps _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps) external-deps: @@ -1103,6 +1107,8 @@ help: @echo ' -dirclean - Remove build directory' @echo ' -reconfigure - Restart the build from the configure step' @echo ' -rebuild - Restart the build from the build step' + @echo ' -source-check - Check package for valid download URLs' + @echo ' -all-source-check - Check package and its dependencies for valid download URLs' $(foreach p,$(HELP_PACKAGES), \ @echo $(sep) \ @echo '$($(p)_NAME):' $(sep) \ @@ -1122,6 +1128,7 @@ help: @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build' + @echo ' source-check - check selected packages for valid download URLs' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' @echo ' printvars - dump all the internal variables' diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 7cd87c38ff..5384fc2af4 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -70,6 +70,7 @@ export BR_NO_CHECK_HASH_FOR = # 3) BR2_BACKUP_SITE if enabled, unless BR2_PRIMARY_SITE_ONLY is set # # Argument 1 is the source location +# Argument 2 (optionally) provides extra arguments to pass to DL_WRAPPER # ################################################################################ @@ -92,6 +93,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ + $(2) \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \ @@ -106,3 +108,7 @@ define DOWNLOAD -- \ $($(PKG)_DL_OPTS) endef + +define SOURCE_CHECK + $(call DOWNLOAD,$(1),-C) +endef diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 6168b40e89..bc57956867 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -788,6 +788,10 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE) endif # actual sources != sources endif # actual sources != "" +$(1)-source-check: PKG=$(2) +$(1)-source-check: + $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep)) + $(1)-external-deps: @for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \ echo `basename $$$$p` ; \ @@ -812,6 +816,9 @@ $(1)-rsync: $$($(2)_TARGET_RSYNC) $(1)-source: $(1)-legal-source: +$(1)-source-check: + test -d $$($(2)_OVERRIDE_SRCDIR) + $(1)-external-deps: @echo "file://$$($(2)_OVERRIDE_SRCDIR)" endif @@ -846,6 +853,9 @@ $(1)-graph-rdepends: graph-depends-requirements $(1)-all-source: $(1)-source $(1)-all-source: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source) +$(1)-all-source-check: $(1)-source-check +$(1)-all-source-check: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check) + $(1)-all-external-deps: $(1)-external-deps $(1)-all-external-deps: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps) @@ -1044,6 +1054,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-all-external-deps \ $(1)-all-legal-info \ $(1)-all-source \ + $(1)-all-source-check \ $(1)-build \ $(1)-clean-for-rebuild \ $(1)-clean-for-reconfigure \ @@ -1068,7 +1079,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-rsync \ $(1)-show-depends \ $(1)-show-version \ - $(1)-source + $(1)-source \ + $(1)-source-check ifneq ($$($(2)_SOURCE),) ifeq ($$($(2)_SITE),) diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 3315bd410e..920699e51d 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -17,7 +17,7 @@ # We want to catch any unexpected failure, and exit immediately. set -e -export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" +export BR_BACKEND_DL_GETOPTS=":hc:Cd:o:n:N:H:ru:qf:e" main() { local OPT OPTARG @@ -25,9 +25,10 @@ main() { local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":c:Cd:D:o:n:N:H:rf:u:q" OPT; do case "${OPT}" in c) cset="${OPTARG}";; + C) checkonly=-C;; d) dl_dir="${OPTARG}";; D) old_dl_dir="${OPTARG}";; o) output="${OPTARG}";; @@ -46,38 +47,40 @@ main() { # Forget our options, and keep only those for the backend shift $((OPTIND-1)) - if [ -z "${output}" ]; then - error "no output specified, use -o\n" - fi + if [ -z "${checkonly}" ]; then + if [ -z "${output}" ]; then + error "no output specified, use -o\n" + fi - # Legacy handling: check if the file already exists in the global - # download directory. If it does, hard-link it. If it turns out it - # was an incorrect download, we'd still check it below anyway. - # If we can neither link nor copy, fallback to doing a download. - # NOTE! This is not atomic, is subject to TOCTTOU, but the whole - # dl-wrapper runs under an flock, so we're safe. - if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then - ln "${old_dl_dir}/${filename}" "${output}" || \ - cp "${old_dl_dir}/${filename}" "${output}" || \ - true - fi + # Legacy handling: check if the file already exists in the global + # download directory. If it does, hard-link it. If it turns out it + # was an incorrect download, we'd still check it below anyway. + # If we can neither link nor copy, fallback to doing a download. + # NOTE! This is not atomic, is subject to TOCTTOU, but the whole + # dl-wrapper runs under an flock, so we're safe. + if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then + ln "${old_dl_dir}/${filename}" "${output}" || \ + cp "${old_dl_dir}/${filename}" "${output}" || \ + true + fi - # If the output file already exists and: - # - there's no .hash file: do not download it again and exit promptly - # - matches all its hashes: do not download it again and exit promptly - # - fails at least one of its hashes: force a re-download - # - there's no hash (but a .hash file): consider it a hard error - if [ -e "${output}" ]; then - if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then - exit 0 - elif [ ${?} -ne 2 ]; then - # Do not remove the file, otherwise it might get re-downloaded - # from a later location (i.e. primary -> upstream -> mirror). - # Do not print a message, check-hash already did. - exit 1 + # If the output file already exists and: + # - there's no .hash file: do not download it again and exit promptly + # - matches all its hashes: do not download it again and exit promptly + # - fails at least one of its hashes: force a re-download + # - there's no hash (but a .hash file): consider it a hard error + if [ -e "${output}" ]; then + if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then + exit 0 + elif [ ${?} -ne 2 ]; then + # Do not remove the file, otherwise it might get re-downloaded + # from a later location (i.e. primary -> upstream -> mirror). + # Do not print a message, check-hash already did. + exit 1 + fi + rm -f "${output}" + warn "Re-downloading '%s'...\n" "${output##*/}" fi - rm -f "${output}" - warn "Re-downloading '%s'...\n" "${output##*/}" fi # Look through all the uris that we were given to download the package @@ -127,7 +130,7 @@ main() { -f "${filename}" \ -u "${uri}" \ -o "${tmpf}" \ - ${quiet} ${recurse} -- "${@}" + ${quiet} ${recurse} ${checkonly} -- "${@}" then # cd back to keep path coherence cd "${OLDPWD}" @@ -138,19 +141,21 @@ main() { # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}" - # Check if the downloaded file is sane, and matches the stored hashes - # for that file - if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then - rc=0 - else - if [ ${?} -ne 3 ]; then - rm -rf "${tmpd}" - continue + if [ -z "${checkonly}" ]; then + # Check if the downloaded file is sane, and matches the stored hashes + # for that file + if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then + rc=0 + else + if [ ${?} -ne 3 ]; then + rm -rf "${tmpd}" + continue + fi + + # the hash file exists and there was no hash to check the file + # against + rc=1 fi - - # the hash file exists and there was no hash to check the file - # against - rc=1 fi download_and_check=1 break @@ -163,6 +168,13 @@ main() { exit 1 fi + # If we only need to check the presence of sources, stop here. + # No need to handle output files. + if [ -n "${checkonly}" ]; then + rm -rf "${tmpd}" + exit 0 + fi + # tmp_output is in the same directory as the final output, so we can # later move it atomically. tmp_output="$(mktemp "${output}.XXXXXX")" From patchwork Tue Feb 19 10:38:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044605 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="p6BFdyeG"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfG3ZJGz9s21 for ; Tue, 19 Feb 2019 21:38:46 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id C4EC08667B; Tue, 19 Feb 2019 10:38:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VZkcpJzwJfQG; Tue, 19 Feb 2019 10:38:37 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id CD97286B6F; Tue, 19 Feb 2019 10:38:37 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 5EB4E1BF283 for ; Tue, 19 Feb 2019 10:38:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 5B98D85C45 for ; Tue, 19 Feb 2019 10:38:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vRn2DjPEe+h8 for ; Tue, 19 Feb 2019 10:38:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) by whitealder.osuosl.org (Postfix) with ESMTPS id B8F4085B7C for ; Tue, 19 Feb 2019 10:38:35 +0000 (UTC) Received: by mail-ot1-f51.google.com with SMTP id n8so33338087otl.6 for ; Tue, 19 Feb 2019 02:38:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1ofoJu1JF6sCvFJXqkWSkl1Gqz8LnZeux/z1pcQS1iM=; b=p6BFdyeGHi3cHPRjOUMOMB+M1pRJ7KyBOEJTwpw0zLFVt2DIX+kFhdMAmF9ADAQtGV NAbdQGi/VlufYDTxD++2AFWBCqbSyqGFk4MVvgfDB/Naqcm0aE6L7o5GwTw/d1vYZi+L Tp0FlOdTGGiVcoRX2m7LZZ1HejdPh7isWAzLXFStDN774BWVHYU08muNRjSDntvjmuOe 7owtI/2Lmxg+/xgAgWMZbyWn75z7ImetAtzyWKpJaPAZKbcZ67gqDwP76W62Nutt0MWG Q9juoxyd5j+P06uuwy8wy5CYctlGDBNT5OWLBVbhBnN+Wp0ZICi6SDWrB6zfyvTxjqep GxlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1ofoJu1JF6sCvFJXqkWSkl1Gqz8LnZeux/z1pcQS1iM=; b=dBb/M38YtLNOZnpM4r3IuGGHXyCrxMcYyfdZ2BHlhNQeOd/2esnp1M2R57IQaHg5Ri gPJfa3hHHShoee+JK2LKNb5qJj32Tv4yMpMX+jZmF/iyP2uWKoNPfP9HSXTBeD5VmtEO VBriS3b/fuo46pyEtO9kHcv/9JysG18Hua/ovwQteSdt4QZJRUIpKHrqzXoPd2tq4mmb 7q6ZUO87yB35bPIADorVQ79kF14Ai9EJSqTgSnjwAqClizAOE8jzt0ZLw1HdFlxwDCvi +EOTnvT9bUZfbcMZEQjC6/2pKHwX7FW9Vh53hhMwuJstvhc1aFXxzZJbPIxvx6lOW0tb VL0A== X-Gm-Message-State: AHQUAuYF4/DhrT9NyIGmjTxa2gfUY5L78jjURAjhnpZB4OBptT8jn5Ff +QLGZu71EXpIJqvOpN2HRzq/IWOGmAQ= X-Google-Smtp-Source: AHgI3IZcO7yYWxUsrqouBAqUGjOgdTKcWIPdG0kOYRXlqUjg29pWtWIebPp3x+sn/2nhj+G2W+LSCw== X-Received: by 2002:aca:1e13:: with SMTP id m19mr2109795oic.42.1550572714669; Tue, 19 Feb 2019 02:38:34 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id f25sm6592035otl.75.2019.02.19.02.38.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:33 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:32 +0100 Message-Id: <20190219103839.25409-2-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 2/9] support/download/hg: implement source-check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed. The original implementation: hg incoming --force -l1 would only verify that the repository exists, not that the requested revision is present. An already better implementation is: hg incoming --force -l1 -r but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded. The implemented solution is: hg identify -r which operates directly on the remote repository. Signed-off-by: Thomas De Schampheleire --- support/download/hg | 12 ++++++++++++ 1 file changed, 12 insertions(+) v5: no changes v4: (feedback Yann E. Morin) - use true/false as values to 'checkonly' - always check revision, even if no source-check - replace incorrect 'exit $?' by explicit 'exit 0' v3: no changes diff --git a/support/download/hg b/support/download/hg index efb515fca5..8f7d1a5a90 100755 --- a/support/download/hg +++ b/support/download/hg @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the changeset exists in the remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET. @@ -16,9 +17,11 @@ set -e # HG : the hg command to call verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";; @@ -36,6 +39,15 @@ _hg() { eval ${HG} "${@}" } +if ! _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null; then + printf "Commit '%s' does not exist in this repository\n." "${cset}" + exit 1 +fi + +if ${checkonly}; then + exit 0 +fi + _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'" _hg archive ${verbose} --repository "'${basename}'" --type tgz \ From patchwork Tue Feb 19 10:38:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044606 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="r5Mxl0p2"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfH6TLCz9s21 for ; Tue, 19 Feb 2019 21:38:47 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 196F785580; Tue, 19 Feb 2019 10:38:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AIV_vCSOQjTQ; Tue, 19 Feb 2019 10:38:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9BACE84B91; Tue, 19 Feb 2019 10:38:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id BA1401BF283 for ; Tue, 19 Feb 2019 10:38:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id B5A0022EE7 for ; Tue, 19 Feb 2019 10:38:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R5etREr74icq for ; Tue, 19 Feb 2019 10:38:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) by silver.osuosl.org (Postfix) with ESMTPS id 0D22022EC9 for ; Tue, 19 Feb 2019 10:38:39 +0000 (UTC) Received: by mail-ot1-f41.google.com with SMTP id i20so33436442otl.0 for ; Tue, 19 Feb 2019 02:38:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FrnQmTFdJt/lH9RoazHUZLYmLyyzP0aRrBKfdFjPKag=; b=r5Mxl0p2BhFs1/IYFX6bT4a7UeDtoNsFjpF/DzL+TxlJLK5qj5YW92NY+UWRfsa7r/ 6SExvuY4WDpVeAM6xyZDwNXrFr5jT/gD9pRLt+oW544S5PhfTHUGGaHcH9QxyORKz2vG afvM44J3kYOLH3V97je16HRkN8UsHAJNgn+EdAQjxYMskZYXgdqN3FiqC0yFSjKHcYyJ JbN2Lx83jc8xgJ+ltze+xPxRvtuhRFwk8Vh58qTtiFBI9oF3XY+8Jg/VH3xcghAag377 F5Xpvnweo4uLm7y1515udopRnJ73/NZ3G0MIDnGHdljuJ1PaqFG+csvxSnlOeimDWjRr FmOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FrnQmTFdJt/lH9RoazHUZLYmLyyzP0aRrBKfdFjPKag=; b=YXlKzz13lABKb4DTSbty8mp/J2HF/0vedsiyL2SB/Z48gXTTVZV0gGURbP3IiH/74x p7TEyvNid1jRgrl1jTphcdcZIBe7SJOHqI4Alm4z0T1HnFAbFwcplH4/9awOQqefF7Je 88OZXj2ZRyk+GLG9Ukfte0uEgZMekZYTrPcRlSJDsJTfMxdUTGcastQtt4psTlPq8PIJ ZXxPMLGlhbkFSeO0KicvJ8atEFAvQoL1oGOUyrkrW/ngJjgQdvI8FmLcZO8Nh56K+Kod gh9LS6HwxWvtLauLdHSNuQieHp//ax6Ttm9iD5UrKheTMrhkcGoCAkPTyf86R9RH5DQM LmKQ== X-Gm-Message-State: AHQUAubCt/nt8N4A1NR5ZQkXSt0bUUSzdSnfIEGbsWSZe18w2+3Qskfr zSRSZv9QXDQuwR9n5kcOfFRHWoPGmVo= X-Google-Smtp-Source: AHgI3IaI/Eb1pbC9cuZTYCuvhrsPmEOPl2h7kVKyXrXXvqUGlq1D2Hui8pSWSzyhXQhCArvfoo5asA== X-Received: by 2002:aca:5807:: with SMTP id m7mr2061798oib.71.1550572718013; Tue, 19 Feb 2019 02:38:38 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id n8sm6714987otl.40.2019.02.19.02.38.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:37 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:33 +0100 Message-Id: <20190219103839.25409-3-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 3/9] support/download/wget: implement source-check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Signed-off-by: Thomas De Schampheleire --- support/download/wget | 8 ++++++++ 1 file changed, 8 insertions(+) v5: no changes v4: (feedback Yann E. Morin) - use true/false as values to 'checkonly' - replace incorrect 'exit $?' by explicit 'exit 0' v3: no changes diff --git a/support/download/wget b/support/download/wget index c69e6071aa..277eab6660 100755 --- a/support/download/wget +++ b/support/download/wget @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Save into file FILE. # -f FILENAME The filename of the tarball to get at URL # -u URL Download file at URL. @@ -16,9 +17,11 @@ set -e # WGET : the wget command to call verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) url="${OPTARG}";; @@ -40,4 +43,9 @@ _wget() { # mirror [ -n "${encode}" ] && filename=${filename//\?/%3F} +if ${checkonly}; then + _wget --spider ${verbose} "${@}" "'${url}/${filename}'" + exit 0 +fi + _wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'" From patchwork Tue Feb 19 10:38:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044607 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="u4YUjpgo"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfM06h4z9s3l for ; Tue, 19 Feb 2019 21:38:50 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 64E7822EC9; Tue, 19 Feb 2019 10:38:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id APYAxJQM+WIo; Tue, 19 Feb 2019 10:38:45 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id BD92A22EE6; Tue, 19 Feb 2019 10:38:45 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id EDF681BF283 for ; Tue, 19 Feb 2019 10:38:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E573D86BFB for ; Tue, 19 Feb 2019 10:38:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4w9kfckFse0h for ; Tue, 19 Feb 2019 10:38:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) by hemlock.osuosl.org (Postfix) with ESMTPS id 406A586B6F for ; Tue, 19 Feb 2019 10:38:42 +0000 (UTC) Received: by mail-ot1-f65.google.com with SMTP id c18so11758995otl.13 for ; Tue, 19 Feb 2019 02:38:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XwF31fwxKSz5WNtnqYw6PL24yLtUHYgOjBXrI3H+d9Y=; b=u4YUjpgoiGb0OCE7mgTRUvg4r4MJWUsQL3v3t6lcL5c/V26qEc9gnJbjdrPHfPWl5Z oZ+Zs6zLU1EOsdN6DT+9C8u1/2OWupnYv6bnU+7OV5JUmP1ePvnNkI9nmpDba17b0QBJ T/qogUC8OS0m+NZ80EgcG+0QCAR/dOdNbUs/RBl7nfjMu8JwCdln4Ma0XTNrg2vFdwOT YnuB7z1Y+loW0Wj36aG8iCOE6kceT7jBXBtIQxME2qvUX2eQ9uzUPXhD7je71y5Jj/W3 NU8pjeBsArEY7j89qpVBrqUFKcl7EPG2vHeXlEkQSa7JjDHsSmdjg02/AGfoCjYMjzgh LCow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XwF31fwxKSz5WNtnqYw6PL24yLtUHYgOjBXrI3H+d9Y=; b=CG4O6epSjTWS/21mEkhWTMVplMVaI4f3EZm5fMCGSstT2CfNsNkdFZl+1cn3MUCAJR 6hFUDlxeWDSfLfAgpLc55ngQoWunXRWoNL/eQSRJ9pParC2HP/QJce4hC9tyWuz8f1EH r8TwLypDzckI1TLb7JPFX9xeTh1XGqSsTXGXprWXmtIMoGipFZZ9TyzWx0Ns+3R0uBcD bXotIEEFC6KSjc7etTKgI8Jr3wF4SfN8aHGEyz5ndOdqEhlY65djjdcvwmU9MXI71rn6 a6wIePRjwCIKb/a7SNLbMKR4MSoZ6vXqls/HpkOCAc/aNHd75sTAIdc2sAR9NQkJti2G 2Kuw== X-Gm-Message-State: AHQUAuYH4Ts4AltK3/pwrXqdvUEYi7pAcScoMhhf8X7GpKOgsa/Ba9VC VIBE4paHY2OMyrHr3xXACjPo+0qGNaU= X-Google-Smtp-Source: AHgI3IaWeuPPQpacQbvpyYh1TQuzgY8xB9vRrv4l1J7G+pYgR50+aMMX0Niv6Nig+OF+rVzPsg0Tww== X-Received: by 2002:aca:c40c:: with SMTP id u12mr2131324oif.138.1550572721248; Tue, 19 Feb 2019 02:38:41 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id b20sm1313063otl.55.2019.02.19.02.38.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:40 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:34 +0100 Message-Id: <20190219103839.25409-4-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 4/9] support/download/file: implement source-check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Signed-off-by: Thomas De Schampheleire --- support/download/file | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) v5: no changes v4: - use true/false as values to 'checkonly' (Yann E. Morin) - replace incorrect 'exit $?' by explicit 'exit 0' (Yann E. Morin) - replace 'test -e' by 'test -f' which is more specific v3: no changes diff --git a/support/download/file b/support/download/file index e52fcf2c8c..059a8dde4c 100755 --- a/support/download/file +++ b/support/download/file @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the source file exists. # -o FILE Copy to file FILE. # -f FILE Copy from basename file FILE. # -u DIR Copy from FILE in DIR. @@ -20,9 +21,11 @@ set -e # Make 'cp' verbose by default, so it behaves a bit like the others. verbose=-v +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=;; + C) checkonly=true;; o) output="${OPTARG}";; f) file="${OPTARG}";; u) dir="${OPTARG}";; @@ -39,4 +42,12 @@ _localfiles() { eval ${LOCALFILES} "${@}" } -_localfiles ${verbose} "'${dir##file://}/${file}'" "'${output}'" +# Remove any scheme prefix +dir="${dir##file://}" + +if ${checkonly}; then + test -f "'${dir}/${file}'" + exit 0 +fi + +_localfiles ${verbose} "'${dir}/${file}'" "'${output}'" From patchwork Tue Feb 19 10:38:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044608 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="lNYReeZC"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfS5F6Hz9s21 for ; Tue, 19 Feb 2019 21:38:56 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 127E785C7D; Tue, 19 Feb 2019 10:38:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gharkzuGEitl; Tue, 19 Feb 2019 10:38:48 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 948BE85C56; Tue, 19 Feb 2019 10:38:48 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id E87611BF283 for ; Tue, 19 Feb 2019 10:38:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id E548685C45 for ; Tue, 19 Feb 2019 10:38:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7kIxKhgdHddY for ; Tue, 19 Feb 2019 10:38:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f68.google.com (mail-ot1-f68.google.com [209.85.210.68]) by whitealder.osuosl.org (Postfix) with ESMTPS id 67F6985B7C for ; Tue, 19 Feb 2019 10:38:45 +0000 (UTC) Received: by mail-ot1-f68.google.com with SMTP id 32so33263585ota.12 for ; Tue, 19 Feb 2019 02:38:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vB/7a923+I5LR3gmB1iLcB5PoSTF59j4+ErbwDtOeHk=; b=lNYReeZCBsgVuZX6FBJ1qtBNbgcsiVxJ5WNI9VTHRKZsBibMOjUkZXPhbRrLAYHknU N8pWoobNujeRgtLz3DikTOHbk3G7Ps7F1D2Y1ygnBLmrBK1FdodvzfOKneJD6YQ5MD7S QIyjiVlHL/BmjXgBHZubfhqPiScQ/5n8UeAhdAyNDQJQrcAQQpmVkNkM4DrbujyhTrWa dGZP6+ZGTUuTBza405xOUCxM46JG/YLSEPytjcxvbX8sJC43WVFK3qZhhaUTxWJHr96s NaQ9Xc5Jt4RC1toml9njAhKnAFmEgBUyJl/MRlaJfySkarVY4ab2QoxhJ03aSgEU4T1S dmAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vB/7a923+I5LR3gmB1iLcB5PoSTF59j4+ErbwDtOeHk=; b=l5C4HZdXVwNP+SfrGU+cl/tSuO0FIS1KALgyLdph9LMOqky4b2+HfuNhQwJbCALABO ikrk/QMvwoJQSjE2Qo2MQDWqV6XVjnaqua5hNgYmlMw2LBQhtF5IU7ft+iowdhIO9htj we3aDcsVErOQXH7gvO/ABqv/oTPiT1aVTz6RHXV4df5YfZsHS8uFvIYIIkH2YROoSLb7 uOQzgSCnZ+fhH0SH3n/J3bqvvk9WwvBGzTj7GWtQYqvdvnk2c5sGdcaPNBXyVOYl5ANL 6USQZHXxeJFGda/QanD4f5rmDVG0JaLekAYRRznN2Wy4P4rjjh57UxHFW87eUhbHzMDw TsFA== X-Gm-Message-State: AHQUAubIC99Qfx7M1RYKV4/sPjw/dZE0OrVWuOCagEWFf85oxM7Qw+tp ZCc1H2BnHR3glTMzyB33hRJxnnR3XUk= X-Google-Smtp-Source: AHgI3IZq7xzt8Qyd/gzZNLE9OlhWeKYkaHifOaO+1n85EFl0uj98/yOgv+m8VQ+FKaME6R1fC4tXEg== X-Received: by 2002:a9d:7986:: with SMTP id h6mr4590011otm.88.1550572724479; Tue, 19 Feb 2019 02:38:44 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id v62sm4146638oie.21.2019.02.19.02.38.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:43 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:35 +0100 Message-Id: <20190219103839.25409-5-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 5/9] Revert "core/download: drop the SSH command" X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire This reverts commit db9473bf6cd7bd12aa1f9faad0a917c973c33827. The SSH command will again be needed to support source-check for the 'scp' download backend. Signed-off-by: Thomas De Schampheleire --- Config.in | 4 ++++ package/pkg-download.mk | 1 + 2 files changed, 5 insertions(+) v5: use 'git revert' instead v4: no changes v3: no changes diff --git a/Config.in b/Config.in index d58d8dc04a..91d56907fe 100644 --- a/Config.in +++ b/Config.in @@ -136,6 +136,10 @@ config BR2_SCP string "Secure copy (scp) command" default "scp" +config BR2_SSH + string "Secure shell (ssh) command" + default "ssh" + config BR2_HG string "Mercurial (hg) command" default "hg" diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 5384fc2af4..dfc4460c6a 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -15,6 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) export HG := $(call qstrip,$(BR2_HG)) export SCP := $(call qstrip,$(BR2_SCP)) +SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper From patchwork Tue Feb 19 10:38:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044609 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="CXYOIH0k"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfT0yJLz9s3l for ; Tue, 19 Feb 2019 21:38:57 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 3CD9085C56; Tue, 19 Feb 2019 10:38:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9RPN-snF65ps; Tue, 19 Feb 2019 10:38:50 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 59ADB85CDC; Tue, 19 Feb 2019 10:38:50 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id F36031BF283 for ; Tue, 19 Feb 2019 10:38:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id F0B4686BFB for ; Tue, 19 Feb 2019 10:38:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id U7fVekdLyKUV for ; Tue, 19 Feb 2019 10:38:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f42.google.com (mail-ot1-f42.google.com [209.85.210.42]) by hemlock.osuosl.org (Postfix) with ESMTPS id 77CC486727 for ; Tue, 19 Feb 2019 10:38:48 +0000 (UTC) Received: by mail-ot1-f42.google.com with SMTP id i5so33304785oto.9 for ; Tue, 19 Feb 2019 02:38:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=p+BLK0HDy3UBcPZLB0uQFStbNPFrPV2SF/FavTLbTos=; b=CXYOIH0kwELkWiW574CLt5bvg9HK6HL2FczYMT5zsZ9Gw/o6cbr0vqS38mSiL8AWD0 EBtcttkK7SU2eF7enTdFjCJwLt+0NyqP4f/rsbdoVgqtSRV94sUmwLm4r5klPmNUCn98 2J7Y0wRIqIRrOzrl/sbTgMgBfkA8Pmj7MBT6ja23ywcRS9LdShWsffmRESrNt+VKoRdB aS0rdHoXr3yRu6zFiOwrZC6uZRKeqZMz17qkA0YbzXcTR+SNG4mA2aIonwW5sIOi5Z5x IJermNhuRxiWCI3hswUMR8s+iBHwPoafAyV75T3yA1kpTDPLaMr9JfakNLL2IqvjBk1h DODg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=p+BLK0HDy3UBcPZLB0uQFStbNPFrPV2SF/FavTLbTos=; b=C6SrdlO9iui6GU2+f2O5v0K3F/0zrZDLPz9HvPy9w3EszfIJVjV/s+R3iAy8vwCLu9 4EVCz8y3bt+W5RcFQLM5/7vSH1Ic2sZ3F1bUldXuqIRvFavfBawj90ltYzKEPuro8CTL TZ3u+Hy9jDFTrMwpV+Lmz2VtPjCWGq1l9HZQaAPH3D3OcB5ky6jQFkKo3JdxxEFgoEQN 7Moa6hGNmcw5xmj3Gs2L/N+6RoG2dBNmgzDKgc9/YkT8KHRdV/5zTI5vGKuhsXdBA3Zt 1IHr14KGxW8NUvFNmpcDJoOzDGk7ZGo6JKWV9Ab4mcjCp6Be4ANpVWMgJWivoMYiHkqq HGvg== X-Gm-Message-State: AHQUAubGT5Av9bibuZ3Jr3V1F5e7+gt6nlRF81ERvM/90+ci3QYG2j0l P1c9Ash+6XytMijDtBjPGVoYx8R+O1c= X-Google-Smtp-Source: AHgI3Ib+gf1uF5cbTs/c8D5BfcsnBBY9nsQPsbPAHfBpL5xMn7nFcRGDz/08DVdffdRkeAeAykkIzw== X-Received: by 2002:a9d:4685:: with SMTP id z5mr17124382ote.251.1550572727519; Tue, 19 Feb 2019 02:38:47 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id n65sm7189526oib.12.2019.02.19.02.38.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:47 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:36 +0100 Message-Id: <20190219103839.25409-6-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 6/9] package/pkg-download: export 'SSH' for use in the download backends X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Originally, the tools defined in pkg-download.mk were used from Makefiles and were directly available. Due to the introduction of the download infrastructure in shell scripts, these variables now need to be exported in order for the scripts to see them. Signed-off-by: Thomas De Schampheleire --- package/pkg-download.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) v5: new patch diff --git a/package/pkg-download.mk b/package/pkg-download.mk index dfc4460c6a..17d1ca99ee 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -15,7 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) export HG := $(call qstrip,$(BR2_HG)) export SCP := $(call qstrip,$(BR2_SCP)) -SSH := $(call qstrip,$(BR2_SSH)) +export SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper From patchwork Tue Feb 19 10:38:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044611 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="fsmv0r6Y"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfY3J8Fz9s21 for ; Tue, 19 Feb 2019 21:39:01 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 2A46F85CE1; Tue, 19 Feb 2019 10:38:56 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zik0kPYH4gws; Tue, 19 Feb 2019 10:38:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id C362C85D57; Tue, 19 Feb 2019 10:38:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 144C61BF283 for ; Tue, 19 Feb 2019 10:38:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 0F18F85CDC for ; Tue, 19 Feb 2019 10:38:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id isQosvR2wQY2 for ; Tue, 19 Feb 2019 10:38:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f46.google.com (mail-ot1-f46.google.com [209.85.210.46]) by whitealder.osuosl.org (Postfix) with ESMTPS id BED5085D21 for ; Tue, 19 Feb 2019 10:38:51 +0000 (UTC) Received: by mail-ot1-f46.google.com with SMTP id b3so33350889otp.4 for ; Tue, 19 Feb 2019 02:38:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wf1p9MUFfv7Gaz495RXsKwV19Au9dnSKFRHcBr8KC88=; b=fsmv0r6Ykz25tCj6KjgBbahZQRBtEJM1Az0o7CLjN+DI/a0FRZa/19V7ANSkQMU2Db EbH/Yo/EHN6BK7O67mRRTiscMfxaH97uT72GHv4ddF/QB6c9zJDcefSLHd7Kokp7NKw2 uOOUCQ501weLrFIUoJxu1KE/73T9JDlRn7hUO1ef+aRn3il2cbt/6voF5U87eWCPYJaL n90QDcyBosdipfFRk7InXlJzO66XMBgCTFDcvM0VfBa+fRabJZwH7qI3ReZeNoQ6bb2w 1RlwbSOpRiILVoBZX45xukIy/yi53rjQpVA9KKHrQB8P2TMSVJmiiZXDa1l/asZ2VO5O 11Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wf1p9MUFfv7Gaz495RXsKwV19Au9dnSKFRHcBr8KC88=; b=MdKVdtUU9c5j189j2KtiY+hmuq9Dlt2M03wgTv8ZM8BvJEYzRSHN1XdsEgPLX+ynzJ MIGCmqomCAxzLJSNbWC8cPqaSubxKsI9TnEa6lDgXzk2OyohitIjFaThTt3GRqQXF2CP wAlA3VOPGmiPHIH6T9Ywvapg4S1PcWXnpIwaeE0+FgydxJw3zEdSXCYNH2r2VbaJhHvI cg51bQiIGiaZPXnF5BIoB83bZ6DwLjxNLHEFw+zWKb151eXr4VrrI+fU8OcII9SUT1sr IN8PiBQ0O3PE92lZqSxBPkSAuP4kMKiO8TWxIfInh1VaKYStW43bULaRRrpXSWx0SqNI IEuA== X-Gm-Message-State: AHQUAuaUOlQqDn7FAGZHYZ18DtvkB2qcUvHG3JwdMiLj+inibou7mwUY 3QGmuUzmKofH0ECFKy75IDQg36yNRoI= X-Google-Smtp-Source: AHgI3IZQxwImpGVrjLZ2nxN9i3piS5yiD4xfdQFgcTkM4/LvdTg0nm4icEBobQHM0HAHtkJXPdMF9Q== X-Received: by 2002:aca:af83:: with SMTP id y125mr2105163oie.174.1550572730647; Tue, 19 Feb 2019 02:38:50 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id s186sm7054875oif.0.2019.02.19.02.38.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:50 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:37 +0100 Message-Id: <20190219103839.25409-7-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 7/9] support/download/scp: implement source-check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire We use 'test -f' rather than 'ls' because 'test' is a mandatory POSIX utility while 'ls' is not. Signed-off-by: Thomas De Schampheleire --- support/download/scp | 15 +++++++++++++++ 1 file changed, 15 insertions(+) v5: no changes v4: (feedback Yann E. Morin) - use true/false as values to 'checkonly' - replace incorrect 'exit $?' by explicit 'exit 0' - use 'test -f' instead of 'ls' which is more certain to be present (POSIX mandatory utility) v3: no changes diff --git a/support/download/scp b/support/download/scp index 80cf495c4e..52ccf06d6e 100755 --- a/support/download/scp +++ b/support/download/scp @@ -7,17 +7,21 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely. # -o FILE Copy to local file FILE. # -f FILE Copy from remote file FILE. # -u URI Download file at URI. # # Environment: # SCP : the scp command to call +# SSH : the ssh command to use for checkonly verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) uri="${OPTARG}";; @@ -33,8 +37,19 @@ shift $((OPTIND-1)) # Get rid of our options _scp() { eval ${SCP} "${@}" } +_ssh() { + eval ${SSH} "${@}" +} # Remove any scheme prefix uri="${uri##scp://}" +if ${checkonly}; then + # uri now looks like: foo.example.org:some/directory + domain="${uri%%:*}" + path="${uri#*:}/${filename}" + _ssh ${verbose} "${@}" "'${domain}'" test -f "'${path}'" > /dev/null + exit 0 +fi + _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'" From patchwork Tue Feb 19 10:38:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044610 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="E163doVY"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfW4WPHz9s21 for ; Tue, 19 Feb 2019 21:38:59 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2DFAB86CAC; Tue, 19 Feb 2019 10:38:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jBMrwZjS7LT5; Tue, 19 Feb 2019 10:38:57 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 20C2586727; Tue, 19 Feb 2019 10:38:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 747F61BF283 for ; Tue, 19 Feb 2019 10:38:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 716E022EE7 for ; Tue, 19 Feb 2019 10:38:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oOhMOYbIy-BT for ; Tue, 19 Feb 2019 10:38:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f68.google.com (mail-ot1-f68.google.com [209.85.210.68]) by silver.osuosl.org (Postfix) with ESMTPS id C79F322EE6 for ; Tue, 19 Feb 2019 10:38:54 +0000 (UTC) Received: by mail-ot1-f68.google.com with SMTP id v20so1391012otk.7 for ; Tue, 19 Feb 2019 02:38:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=df7oR3pBC9+bPr09l2M6npO7EITvSo8ao0YpyxodnCU=; b=E163doVYKI4Gh7hg+bywBkdSbIEMmFV2QmqPq35PG6mIUYt4MXKJvX3F9B5mz7zWB4 T8a5l3ra2qObO367KwPuJe+w7beq079+6Rn6/+X3WVLW2zsbauuzIvDI1mXkxF4LO0T4 ZCSjyFIalgPAmrMPxhSPqg2MKqWiaMh+ZpX+lbPTEV5/tEpQ36014jJezLBhsWxWmhLl RgHuJe8ArwF+n8P7PjKMgb5T84Est6UFdHMGbl4lGwSnmOYvcWwY0jsma61+9muiMVrB sI9co5YjE0JYqJMbYPUIzAmkXIEWryGAnxi1wt4pmkk0Z/sNhMZzeKf5II0eChlBFkQg IhHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=df7oR3pBC9+bPr09l2M6npO7EITvSo8ao0YpyxodnCU=; b=qq3BCoqU5Z25PSAr7fsN3076gPaGbia+fxHKCBYeMWzmgbNVca80jVjdHNKXdZrZcS 4SQ0b2PgihIsEc+YGYGv4Tc1lkktQy/G1Xnj+tWDpwGcEg5JRG7Z5yzBKZB40x1pJ57G AHQbBXA2nsNbyyJNGRK/m8nBFT2lBqe/6c1Hefhay7LVjOxn+bNHE4uc3gKtmXZub6hl nK+i0Ff70QHm9gTkJ4EJZMZj+T9Wn5QtuX5scQetKMgxJJ2GJyRCqevjTMDXRSouSEPs QyN/WIveeIZU9/KuLsLXCk83SPuFaucccy/VRpmWGZQ8YAM3l/lqYzhvpFG6UdWgqun6 pkpg== X-Gm-Message-State: AHQUAuambwnge6x8zuzN0VhVIhsfFaT83FY4hGCKdREbGOta6YEu+3OO TloQwcdCtVBYMdfb+mRLgDLPcNtNiEU= X-Google-Smtp-Source: AHgI3IZc/kSHSR2m7i5lhVerOgEvcQYTGzWl3t8UOBcqqdlcqlBPiuBMK9fvxQyVPxWHAQwTWRv8lw== X-Received: by 2002:a9d:2248:: with SMTP id o66mr18196748ota.61.1550572733831; Tue, 19 Feb 2019 02:38:53 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id n185sm7222211oih.18.2019.02.19.02.38.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:53 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:38 +0100 Message-Id: <20190219103839.25409-8-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 8/9] support/download/svn: implement source-check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Suggested-by: "Yann E. Morin" Signed-off-by: Thomas De Schampheleire --- support/download/svn | 12 ++++++++++++ 1 file changed, 12 insertions(+) v5: no changes v4: new patch diff --git a/support/download/svn b/support/download/svn index 542b25c0a2..93984b6c45 100755 --- a/support/download/svn +++ b/support/download/svn @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the revision exists in the remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV. @@ -16,9 +17,11 @@ set -e # SVN : the svn command to call verbose= +checkonly=false while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=true;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) rev="${OPTARG}";; @@ -36,6 +39,15 @@ _svn() { eval ${SVN} "${@}" } +if ! _svn ls ${verbose} -r "'${rev}'" "'${uri}'" > /dev/null; then + printf "Revision '%s' does not exist in this repository\n." "${rev}" + exit 1 +fi + +if ${checkonly}; then + exit 0 +fi + _svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'" tar czf "${output}" "${basename}" From patchwork Tue Feb 19 10:38:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1044612 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VOaewvDA"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 443cfZ1D9Dz9s3l for ; Tue, 19 Feb 2019 21:39:02 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 42B0A86BFB; Tue, 19 Feb 2019 10:39:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ESARPjI1jRzm; Tue, 19 Feb 2019 10:38:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id BAC6D86D41; Tue, 19 Feb 2019 10:38:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id AE08E1BF283 for ; Tue, 19 Feb 2019 10:38:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id AAE4122EE7 for ; Tue, 19 Feb 2019 10:38:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7wnvu6EWMyB4 for ; Tue, 19 Feb 2019 10:38:58 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ot1-f65.google.com (mail-ot1-f65.google.com [209.85.210.65]) by silver.osuosl.org (Postfix) with ESMTPS id E3C4D22EE6 for ; Tue, 19 Feb 2019 10:38:57 +0000 (UTC) Received: by mail-ot1-f65.google.com with SMTP id b3so33351358otp.4 for ; Tue, 19 Feb 2019 02:38:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gr1pEYmqxkvkUqdZuNvV19Nnda6k99B84dUk5OMz1uw=; b=VOaewvDAKkhRd5LT+xfBET0xB9oPp3TS0HXVsii/iyl/9+0IRPXtEuJGPVbkHMmimm MZEHmBCuxlJbYeUjV9YYcdVtTMzKMYOocRtDrlUAnIOSybCnr6bWWwS5jwlf5A0Dxpby mDnJZsSdpot9iIlwJBOlPMjlINXJ/uSd2RjyEuDQn7UhkQJj5WEtHDsLWilaBDYPO4Cl /5C7aWpeGVcI01/CvTZfe7aJmMI+cxzoWaVxSDEgdP4jLp0xBDqkqBs8PsJzkEBOGqo8 xrmjVBOBXRj1fIOi6tthBqyI4PnkqMzdTIKT4SeAyDXANLbD/3k1+si/R394vWOYQkbS 2X/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gr1pEYmqxkvkUqdZuNvV19Nnda6k99B84dUk5OMz1uw=; b=qxoIPG4EBuXz1nG84Ar4AS4EsQmR/fqPMmQkmKpvxCBwSl5266MnIqjIFTg0GUKZaE DvZ66Ngk7a3iO+Fqpd7XlYlHcIBIgCoOR1XkzTzE4jJmQEmqvw3MvLgXpzLOCpsIdl6D /kVhewrtJKUa0hIUzAsOvvm9WjeuoOKpY7tLPHVAGjHMuQjR/EQx0HBG1h3dfN/FNIfT Ulk1N2MM+BAJHkaTngJUKu4G2OLBM+EOsiq2gJhYdJv+Rv05j1PLGZSYegJ/RsBWhovT 0b89dCIloIE2Qfkli53WKmKeDf3YLKdxprjmzvqU31VWDkVAyJMjYYLCTm53E9F2C02t ebpg== X-Gm-Message-State: AHQUAuYShjqaRhuwIR8s+Ks4crwce6xkg1pi51msN1pqIpsv+Z+7ATx2 ccAWASJNznkGGnkzXQ+AI4brBA6+qnY= X-Google-Smtp-Source: AHgI3IYSr19zWNHMX4/MQwRZrbNULuWiDWecibIxZWaEuAh1vrhnS/HMmP9mmcdMNjT7JFUUoBeqOA== X-Received: by 2002:aca:b882:: with SMTP id i124mr2063657oif.127.1550572736978; Tue, 19 Feb 2019 02:38:56 -0800 (PST) Received: from localhost ([131.228.32.166]) by smtp.gmail.com with ESMTPSA id e12sm1907720otr.1.2019.02.19.02.38.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 19 Feb 2019 02:38:56 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 19 Feb 2019 11:38:39 +0100 Message-Id: <20190219103839.25409-9-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190219103839.25409-1-patrickdepinguin@gmail.com> References: <20190219103839.25409-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv5 9/9] support/download/{bzr, cvs, git}: highlight unimplemented source-check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire For bzr, cvs and git, there is no method of checking whether the revision exists remotely, without actually cloning the repository. Therefore, there is no actual implementation for '-C' (checkonly) for these download helpers. The script will fall back to the normal download logic, which would effectively fail if the revision does not exist. For completeness, mention this unimplemented feature in the usage text. Signed-off-by: Thomas De Schampheleire --- support/download/bzr | 2 ++ support/download/cvs | 2 ++ support/download/git | 2 ++ 3 files changed, 6 insertions(+) v5: no changes v4: remove 'svn' which now has an actual implementation v3: no changes diff --git a/support/download/bzr b/support/download/bzr index 5289a421cd..278d8778af 100755 --- a/support/download/bzr +++ b/support/download/bzr @@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET. diff --git a/support/download/cvs b/support/download/cvs index 9d0dc3cb3a..d9a586293a 100755 --- a/support/download/cvs +++ b/support/download/cvs @@ -7,6 +7,8 @@ set -e # # Options: # -q Be quiet +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV. diff --git a/support/download/git b/support/download/git index 17ca04eb98..e9c02d6712 100755 --- a/support/download/git +++ b/support/download/git @@ -7,6 +7,8 @@ set -E # # Options: # -q Be quiet. +# -C (unimplemented) Only check that the revision exists in the +# remote repository. # -r Clone and archive sub-modules. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI.