From patchwork Thu Jan 3 20:40:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020488 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ZUA2kQ0h"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W0905xySz9rxp for ; Fri, 4 Jan 2019 07:37:40 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6B6A5862E1; Thu, 3 Jan 2019 20:37:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LCDxLEIqpXWu; Thu, 3 Jan 2019 20:37:35 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 017C986256; Thu, 3 Jan 2019 20:37:35 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 3ECDD1C2A15 for ; Thu, 3 Jan 2019 20:37:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 37A45232D7 for ; Thu, 3 Jan 2019 20:37:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rlhwPzT2PGRP for ; Thu, 3 Jan 2019 20:37:33 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by silver.osuosl.org (Postfix) with ESMTPS id 68707232D2 for ; Thu, 3 Jan 2019 20:37:33 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id b14so30071157edt.6 for ; Thu, 03 Jan 2019 12:37:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=P3DbyZtXZl1MxpazIg0YYN7eoantPFREYT7V+u/vV64=; b=ZUA2kQ0hE57u8F+GRBB/EfFoZNw9UyYsDisUfBAZZzsmL3BAKL4kMjc6Ylv40GAzyQ ZiM/186jS3ky5D2FIuW6DEDPC8KwxH1v8KyHp24NXQzX7Xyz/zXc8eWNI8cxjk6pIZL1 aOR9rOtKyA+Gbx3BdKTBwVVvWPGYfQQlDi7u/ml9gubuwSogIXSpi58Uy+9IsJ7dkGaL gxAGPFXgauUy7rzY55LOQ3sT2qlL9Qz71ROpCPon3Mc5TjWH3uOL8VXEdn3SQu+1pAk5 XDQgiY26+mGMXyXmxo8M/erYYidKzV+vIjc2fUhgREWhQZ6hZVNYKGhHeYCZ5S7YAZVM HWkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=P3DbyZtXZl1MxpazIg0YYN7eoantPFREYT7V+u/vV64=; b=m4Il/nxqqjJHM9gpScwz+Xa8cfhbAEG7Fa8H0I/HEH1xUHdoEAXEwoAZf0qdp+panu dXg3xE5E8piwdViQpMjnbi6K2bGu2pGhg3c52+0L2KNyOMKzw3tdMF0Mls6xl+P/4MQW FBI1aCf5kgjCdVsHY4AXD5ZuKdF3uc0B0SnnIW0XcSYYf5TFIu9toiuHu71OhhWbRqZ2 +8S0Wh0V0ybXP8SwIpZOUYWR2SDdyjBGMh66yNuoy1X8c0XhMF2U5lMnqBSYVK6Gkgu4 MOLWqN/DqWHF1N4Q7RzHBmegmbqllBASqh9bAbFyVtYAZlSj1zR0q/bZQasrOBJ2+qUk I5cg== X-Gm-Message-State: AA+aEWZOVXqgBJLO7VeSS77M/wCxLT4dj4uDDPecziQ94d0m2bdKbv2s CeZWzwqvN/Su+1BLg5NDwhk+SicVoHU= X-Google-Smtp-Source: AFSGD/WpzbRpA6BtwAYnOl2zrQpP5FWeX+Q8Mg2HAqh1j0FNdCjZrglgB1YiznS41zA4uAAOmidTyg== X-Received: by 2002:a17:906:b303:: with SMTP id n3-v6mr36965186ejz.34.1546547851614; Thu, 03 Jan 2019 12:37:31 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:31 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:16 +0100 Message-Id: <20190103204026.23512-2-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 01/11] support/download: fix scp downloads X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire scp download is broken, because scp is called without filename argument and only the server is specified. The call is: scp but should be: scp / Instead of assuming '-u' lists a full URL including filename (which it is not), align with the wget helper where -u is the server URL and -f gives the filename. With this commit, an scp download can work if FOO_SITE_METHOD is explicitly set to 'scp' and the server does not have a scheme prefix 'scp://'. The next commit will handle the case where a scheme prefix is present. Signed-off-by: Thomas De Schampheleire Reviewed-by: "Yann E. MORIN" Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
--- support/download/scp | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/support/download/scp b/support/download/scp index 8ecf2f4b22..746c3c6ba0 100755 --- a/support/download/scp +++ b/support/download/scp @@ -8,7 +8,8 @@ set -e # Options: # -q Be quiet. # -o FILE Copy to local file FILE. -# -u FILE Copy from remote file FILE. +# -f FILE Copy from remote file FILE. +# -u URL Download file at URL. # # Environment: # SCP : the scp command to call @@ -18,6 +19,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; o) output="${OPTARG}";; + f) filename="${OPTARG}";; u) uri="${OPTARG}";; :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;; \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; @@ -32,4 +34,4 @@ _scp() { eval ${SCP} "${@}" } -_scp ${verbose} "${@}" "'${uri}'" "'${output}'" +_scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'" From patchwork Thu Jan 3 20:40:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020489 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="MJ82ZiMI"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09364KYz9s7h for ; Fri, 4 Jan 2019 07:37:43 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5F20D86F2E; Thu, 3 Jan 2019 20:37:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o7UFuiQB4ySD; Thu, 3 Jan 2019 20:37:38 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id BA835870A1; Thu, 3 Jan 2019 20:37:38 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 773FA1C2A15 for ; Thu, 3 Jan 2019 20:37:35 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 73C47232D7 for ; Thu, 3 Jan 2019 20:37:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WZXn1E17M-Ec for ; Thu, 3 Jan 2019 20:37:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by silver.osuosl.org (Postfix) with ESMTPS id BA87F232D2 for ; Thu, 3 Jan 2019 20:37:34 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id o10so30042338edt.13 for ; Thu, 03 Jan 2019 12:37:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=LR27dLbfebCLzAztG5xQ0zrJcDq3t6KVbK/ZgDyhR+c=; b=MJ82ZiMIFx7CUC79mgaL8C0zmzbgKy/XhZiRGP4Bo+dGchuPRi6pzJaX00kNlWrHjY g9TzH5Pmo6NafxVLXMWcbtxEkR+Epym7puuYlhFrnbqtl0FqOGz4ewRT6U1Nizbb7P7u bVSsDlSja3QQzhd8iNju3qXisvpG6B+xxycXP6Co45XDx16iOnOy3AnYliEJZtfTuDMQ 3v1DL3CDqX9/RWbGu+c86EOtYMTv3Sah7HOGSk7rxAlAvstU6hSGVMPsGJH6kOpF6hPd Vvqn1h6EdBlxWA9Ps8AO3d1CFGOjb9HfXbQl7DmZd8DGOMvwOfulHCyOaf6PdaUiOPxF R2Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=LR27dLbfebCLzAztG5xQ0zrJcDq3t6KVbK/ZgDyhR+c=; b=hytZHyB3Nd6yUgQIQAmVuJJjIrmd8KJcIvskIhrgpCHUwuzkHlJEOibNIWeawhwCyS NyGtkRkpmljg+wKWEMYUyZAN2OB6HfhREWzKhQC5dmqJkQUzxe4OTFfW5+n/hnL+iH1g OjW81SlCeo2zSFkNwFGS5TQVUMV/gu8CHTIf8dNT6aOyz3rhIlYy2xvXZefMzRdbZ6rQ FX8h5WIa57yqlXpJqCtuFOz3T3d3uS11QS2hMAtQ8eQoLJDqxMZ40Q2nnkoLNjV7gsD1 xQeH6XL4zv+ZVR+KmoJFhABeXSPbnXZJ+gkvg451LSa2r1q0zMhjpmTWcYWv47kVUXXf oe9A== X-Gm-Message-State: AA+aEWZqpzDz8GGKrNiOtIa6Kl4RbfUbDdCV2oU18f+cGU6DqSYnPg6R ttRoy3Te4DUJ8noFQgTofUIeZSIvlKk= X-Google-Smtp-Source: AFSGD/UFqGeRCB12BqHfkYujTzEPcQlZun7JQRcDX3i4aKTgKGrTRintHdZK0IkYKzmp5d/xXD4qIQ== X-Received: by 2002:a50:f5af:: with SMTP id u44mr45390775edm.172.1546547852900; Thu, 03 Jan 2019 12:37:32 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:32 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:17 +0100 Message-Id: <20190103204026.23512-3-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 02/11] support/download: fix scp download with scheme prefix 'scp://' X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire The scp download helper is broken when the server URL starts with 'scp://'. Such prefix is used in two situations: 1. to let FOO_SITE point to an scp location without explicitly having to set 'FOO_SITE_METHOD = scp' 2. when BR2_PRIMARY_SITE or BR2_BACKUP_SITE points to an scp location. In this case, there is no equivalent of 'SITE_METHOD'. Strip out the scheme prefix, similarly to how the 'file' download helper does it. That helper has the same cases as above. Signed-off-by: Thomas De Schampheleire Reviewed-by: "Yann E. MORIN" --- support/download/scp | 3 +++ 1 file changed, 3 insertions(+) diff --git a/support/download/scp b/support/download/scp index 746c3c6ba0..55f588e157 100755 --- a/support/download/scp +++ b/support/download/scp @@ -34,4 +34,7 @@ _scp() { eval ${SCP} "${@}" } +# Remove any scheme prefix +uri="${uri##scp://}" + _scp ${verbose} "${@}" "'${uri}/${filename}'" "'${output}'" From patchwork Thu Jan 3 20:40:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020491 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="WrfSmNpA"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09551czz9rxp for ; Fri, 4 Jan 2019 07:37:45 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0D21E23376; Thu, 3 Jan 2019 20:37:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9+MNC+NRqa7f; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id EF019232D2; Thu, 3 Jan 2019 20:37:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id EB6E71C2A15 for ; Thu, 3 Jan 2019 20:37:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id E535E8628E for ; Thu, 3 Jan 2019 20:37:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ifzsIFBcldvQ for ; Thu, 3 Jan 2019 20:37:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by whitealder.osuosl.org (Postfix) with ESMTPS id 1E6D985582 for ; Thu, 3 Jan 2019 20:37:36 +0000 (UTC) Received: by mail-ed1-f52.google.com with SMTP id o10so30042385edt.13 for ; Thu, 03 Jan 2019 12:37:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=uNouf16yY6CoWh2OUGPgUk5ONgYN5Igss9AT+4a+Qqk=; b=WrfSmNpAP8QhbGHY6e+NqqruQXbs2+ohLfbza9qvU89R7FBzw2Zvx7tqeQZDURHnxQ WwESMMorJAB0eyvjtjsf1zlFywLvfVcqmeZHN+fMYev7koGEjHQgEj89K6V8XhZ5gCfU fisviPk6l6XKpuOUQpr/706U5wf5FXyIlwpq0u+QbYSvr6azN9/njNcctV31PCOtRxR9 1WGGrYBsOW7k+gcpm3uRultaHssUWfrbLVlU/zO1slkuoQFyJ1C4CUUxVw9sU8MLqHK3 SU35tnNniVfVg0k+PgWQ5p3oILTk4iZQ9i3Ion141IzuP6eaPB5G+tAAzL2a0wOXLd2a B7+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=uNouf16yY6CoWh2OUGPgUk5ONgYN5Igss9AT+4a+Qqk=; b=lSvMpw9wPD8eUprwS7QsJr2xi/FWVMtHo91n02ZthaQv5tGts9pT2LegNZL6o0HTLY JcFyXNurdRje3gWRiY2r9FG9IOKET2HpDDT2wMd4rQFVJaJDCc8BLeT1ql69pyMpKwNp ZegWMRAQUNlk7uWcK/uR8AWEKu0TF6vdAt5pFnpFoF6y1K0Sqd15DSuELNXW9BTf4fgQ UoAaS1EI5daaZkAPJWugMWfReA+8drYgQcMFrq6dFNEMtIw/4iK4KnvA2p6I/ZIgERNl IWXVERJq6WCQXbQUhI/AOYQMIofiNqjc6da7SnbBCKU++6d8+WltyvkFjpS+s0FXbWal ssUw== X-Gm-Message-State: AA+aEWbfNYWyMwscYNlWBwOoqUb6Lj4KmILquyGtK9DlQTiqDkxBHptq U4FG5RRB7esYGbd8wsATzBzwfa/0oos= X-Google-Smtp-Source: AFSGD/X9vLM78ZZeGAwEAxY+6kOPr9i5jBxMxRy3CkG+B5I+uPEjHkU9nPnrmy+8ky8WDFGXyOoJKQ== X-Received: by 2002:a05:6402:3d1:: with SMTP id t17mr45003538edw.21.1546547854165; Thu, 03 Jan 2019 12:37:34 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:33 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:18 +0100 Message-Id: <20190103204026.23512-4-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 03/11] support/download: reintroduce 'source-check' target X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Commit bf28a165d992681a85b43a886005e669b3cb579b removed the source-check target to allow easier refactoring of the download infrastructure. It was partly based on the fact that no-one really used this target. However, it turns out that there _are_ good uses for it. In a work environment where many people are submitting changes to a Buildroot repository, one cannot always blindly trust every change. A typical case of human error is when bumping the version of a package but forgetting to upload the tarball to the internal BR2_PRIMARY_SITE or (in case of a repository) pushing the new changesets to the repository. If a user cannot directly push to the Buildroot repository but needs to queue their change via an automatic validation system, that validation system can use 'make source-check' on the relevant defconfigs to protect against such errors. A full download would also be possible but takes longer and unnecessarily uses internal network bandwidth. With that use case in mind, this commit reintroduces the 'source-check' target, but embedded in the current situation with a dl-wrapper. The changes to the wrapper are minimal (not considering additional indentation). A new option '-C' means 'check only' and will be passed to the download backends. If the backend supports the option, no download will happen. If it does not, then the backend will actually perform a download as a means of checking that the source exists (a form of graceful degradation). In neither case, though, hash checking is performed (as the standard case is without download thus without file to calculate hashes on). Subsequent commits will actually implement -C in the backends. Signed-off-by: Thomas De Schampheleire --- Makefile | 7 +++ package/pkg-download.mk | 19 +++++++ package/pkg-generic.mk | 14 +++++- support/download/dl-wrapper | 99 ++++++++++++++++++++----------------- 4 files changed, 94 insertions(+), 45 deletions(-) diff --git a/Makefile b/Makefile index c5b78b3274..f3e22d213f 100644 --- a/Makefile +++ b/Makefile @@ -131,6 +131,7 @@ noconfig_targets := menuconfig nconfig gconfig xconfig config oldconfig randconf # (default target is to build), or when MAKECMDGOALS contains # something else than one of the nobuild_targets. nobuild_targets := source %-source \ + source-check %-source-check %-all-source-check \ legal-info %-legal-info external-deps _external-deps \ clean distclean help show-targets graph-depends \ %-graph-depends %-show-depends %-show-version \ @@ -788,6 +789,9 @@ target-post-image: $(TARGETS_ROOTFS) target-finalize staging-finalize .PHONY: source source: $(foreach p,$(PACKAGES),$(p)-all-source) +.PHONY: source-check +source-check: $(foreach p,$(PACKAGES),$(p)-all-source-check) + .PHONY: _external-deps external-deps _external-deps: $(foreach p,$(PACKAGES),$(p)-all-external-deps) external-deps: @@ -1070,6 +1074,8 @@ help: @echo ' -dirclean - Remove build directory' @echo ' -reconfigure - Restart the build from the configure step' @echo ' -rebuild - Restart the build from the build step' + @echo ' -source-check - Check package for valid download URLs' + @echo ' -all-source-check - Check package and its dependencies for valid download URLs' $(foreach p,$(HELP_PACKAGES), \ @echo $(sep) \ @echo '$($(p)_NAME):' $(sep) \ @@ -1089,6 +1095,7 @@ help: @echo @echo 'Miscellaneous:' @echo ' source - download all sources needed for offline-build' + @echo ' source-check - check selected packages for valid download URLs' @echo ' external-deps - list external packages used' @echo ' legal-info - generate info about license compliance' @echo ' printvars - dump all the internal variables' diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 7cd87c38ff..cc04e316e2 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -106,3 +106,22 @@ define DOWNLOAD -- \ $($(PKG)_DL_OPTS) endef + +define SOURCE_CHECK + $(Q)mkdir -p $($(PKG)_DL_DIR) + $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ + -C \ + -c '$($(PKG)_DL_VERSION)' \ + -d '$($(PKG)_DL_DIR)' \ + -D '$(DL_DIR)' \ + -f '$(notdir $(1))' \ + -H '$($(PKG)_HASH_FILE)' \ + -n '$($(PKG)_BASENAME_RAW)' \ + -N '$($(PKG)_RAWNAME)' \ + -o '$($(PKG)_DL_DIR)/$(notdir $(1))' \ + $(if $($(PKG)_GIT_SUBMODULES),-r) \ + $(DOWNLOAD_URIS) \ + $(QUIET) \ + -- \ + $($(PKG)_DL_OPTS) +endef diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index f5cab2b9c2..707996e384 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -780,6 +780,10 @@ $(1)-legal-source: $$($(2)_TARGET_ACTUAL_SOURCE) endif # actual sources != sources endif # actual sources != "" +$(1)-source-check: PKG=$(2) +$(1)-source-check: + $$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep)) + $(1)-external-deps: @for p in $$($(2)_SOURCE) $$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS) ; do \ echo `basename $$$$p` ; \ @@ -804,6 +808,9 @@ $(1)-rsync: $$($(2)_TARGET_RSYNC) $(1)-source: $(1)-legal-source: +$(1)-source-check: + test -d $$($(2)_OVERRIDE_SRCDIR) + $(1)-external-deps: @echo "file://$$($(2)_OVERRIDE_SRCDIR)" endif @@ -838,6 +845,9 @@ $(1)-graph-rdepends: graph-depends-requirements $(1)-all-source: $(1)-source $(1)-all-source: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source) +$(1)-all-source-check: $(1)-source-check +$(1)-all-source-check: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-source-check) + $(1)-all-external-deps: $(1)-external-deps $(1)-all-external-deps: $$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-external-deps) @@ -1036,6 +1046,7 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-all-external-deps \ $(1)-all-legal-info \ $(1)-all-source \ + $(1)-all-source-check \ $(1)-build \ $(1)-clean-for-rebuild \ $(1)-clean-for-reconfigure \ @@ -1060,7 +1071,8 @@ DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) $(1)-rsync \ $(1)-show-depends \ $(1)-show-version \ - $(1)-source + $(1)-source \ + $(1)-source-check ifneq ($$($(2)_SOURCE),) ifeq ($$($(2)_SITE),) diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 3315bd410e..1ed2e654de 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -17,7 +17,7 @@ # We want to catch any unexpected failure, and exit immediately. set -e -export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" +export BR_BACKEND_DL_GETOPTS=":hc:Cd:o:n:N:H:ru:qf:e" main() { local OPT OPTARG @@ -25,9 +25,10 @@ main() { local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":c:Cd:D:o:n:N:H:rf:u:q" OPT; do case "${OPT}" in c) cset="${OPTARG}";; + C) checkonly=-C;; d) dl_dir="${OPTARG}";; D) old_dl_dir="${OPTARG}";; o) output="${OPTARG}";; @@ -46,38 +47,40 @@ main() { # Forget our options, and keep only those for the backend shift $((OPTIND-1)) - if [ -z "${output}" ]; then - error "no output specified, use -o\n" - fi + if [ -z "${checkonly}" ]; then + if [ -z "${output}" ]; then + error "no output specified, use -o\n" + fi - # Legacy handling: check if the file already exists in the global - # download directory. If it does, hard-link it. If it turns out it - # was an incorrect download, we'd still check it below anyway. - # If we can neither link nor copy, fallback to doing a download. - # NOTE! This is not atomic, is subject to TOCTTOU, but the whole - # dl-wrapper runs under an flock, so we're safe. - if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then - ln "${old_dl_dir}/${filename}" "${output}" || \ - cp "${old_dl_dir}/${filename}" "${output}" || \ - true - fi + # Legacy handling: check if the file already exists in the global + # download directory. If it does, hard-link it. If it turns out it + # was an incorrect download, we'd still check it below anyway. + # If we can neither link nor copy, fallback to doing a download. + # NOTE! This is not atomic, is subject to TOCTTOU, but the whole + # dl-wrapper runs under an flock, so we're safe. + if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then + ln "${old_dl_dir}/${filename}" "${output}" || \ + cp "${old_dl_dir}/${filename}" "${output}" || \ + true + fi - # If the output file already exists and: - # - there's no .hash file: do not download it again and exit promptly - # - matches all its hashes: do not download it again and exit promptly - # - fails at least one of its hashes: force a re-download - # - there's no hash (but a .hash file): consider it a hard error - if [ -e "${output}" ]; then - if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then - exit 0 - elif [ ${?} -ne 2 ]; then - # Do not remove the file, otherwise it might get re-downloaded - # from a later location (i.e. primary -> upstream -> mirror). - # Do not print a message, check-hash already did. - exit 1 + # If the output file already exists and: + # - there's no .hash file: do not download it again and exit promptly + # - matches all its hashes: do not download it again and exit promptly + # - fails at least one of its hashes: force a re-download + # - there's no hash (but a .hash file): consider it a hard error + if [ -e "${output}" ]; then + if support/download/check-hash ${quiet} "${hfile}" "${output}" "${output##*/}"; then + exit 0 + elif [ ${?} -ne 2 ]; then + # Do not remove the file, otherwise it might get re-downloaded + # from a later location (i.e. primary -> upstream -> mirror). + # Do not print a message, check-hash already did. + exit 1 + fi + rm -f "${output}" + warn "Re-downloading '%s'...\n" "${output##*/}" fi - rm -f "${output}" - warn "Re-downloading '%s'...\n" "${output##*/}" fi # Look through all the uris that we were given to download the package @@ -127,7 +130,7 @@ main() { -f "${filename}" \ -u "${uri}" \ -o "${tmpf}" \ - ${quiet} ${recurse} -- "${@}" + ${quiet} ${recurse} ${checkonly} -- "${@}" then # cd back to keep path coherence cd "${OLDPWD}" @@ -138,19 +141,21 @@ main() { # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}" - # Check if the downloaded file is sane, and matches the stored hashes - # for that file - if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then - rc=0 - else - if [ ${?} -ne 3 ]; then - rm -rf "${tmpd}" - continue + if [ -z "${checkonly}" ]; then + # Check if the downloaded file is sane, and matches the stored hashes + # for that file + if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then + rc=0 + else + if [ ${?} -ne 3 ]; then + rm -rf "${tmpd}" + continue + fi + + # the hash file exists and there was no hash to check the file + # against + rc=1 fi - - # the hash file exists and there was no hash to check the file - # against - rc=1 fi download_and_check=1 break @@ -163,6 +168,12 @@ main() { exit 1 fi + # If we only need to check the presence of sources, stop here. + # No need to handle output files. + if [ -n "${checkonly}" ]; then + exit 0 + fi + # tmp_output is in the same directory as the final output, so we can # later move it atomically. tmp_output="$(mktemp "${output}.XXXXXX")" From patchwork Thu Jan 3 20:40:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020490 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="o74Ya+tP"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09425qhz9s9G for ; Fri, 4 Jan 2019 07:37:44 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D26FA85582; Thu, 3 Jan 2019 20:37:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id i3Fe59TIMYNs; Thu, 3 Jan 2019 20:37:39 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 7C5AB8628E; Thu, 3 Jan 2019 20:37:39 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id C89E91C2A15 for ; Thu, 3 Jan 2019 20:37:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C616B232D7 for ; Thu, 3 Jan 2019 20:37:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8jk3HvXWtUil for ; Thu, 3 Jan 2019 20:37:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by silver.osuosl.org (Postfix) with ESMTPS id 29F8B232D2 for ; Thu, 3 Jan 2019 20:37:37 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id f9so30062051eds.10 for ; Thu, 03 Jan 2019 12:37:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=PfFF4FqMT7iRPFZbnCs/n035fbd6sij1nQwSd/LT0+M=; b=o74Ya+tPJ33b7Ziau2CpjxOApr6WR4VU5AZNBBbVT/gHjesiXvDf/+nuF+94IGBChS nkRUop9CjCAv5dMWsRXuUWy+Qa2Edj0XSdr4Z+/3YWLE6qZ6WxDEccYr2cxkdi60uZoZ gePMyt4wM7zyf+Ub4AHpAIItQm+Qqkf7cp+ZPPhIyDSfU8CKiJqbmlnKC7MThuSXgvw5 WJO2FbirXHTmAz6amMNQ77CEHsmmPsvYR7jhJQniMkSTxo1Jh5FfaeeKBADOLcHGf9VS DN/sfHV4hjVt57w97K6A/4MdoZwaS0Vu9asqoNTePeE0DSVHQ6cYfW2IIx9lcvZj0Ywc Y0CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=PfFF4FqMT7iRPFZbnCs/n035fbd6sij1nQwSd/LT0+M=; b=QTHS4anXRADqewAKsshvQkDD+DGdR33ODSkIveURjLheBmd4DopVi5IECGHALm5MtL e5GIkgBlh1jSxv4wW1lRmwvxkG5ZFAyEp4TQBf2FZKgo57DWaG7aL5kcHRkAy5r7PAu/ uqGSwdGHaz8IsUIXbJQvaSipwp39ncpio2yCDA7LepKn7cXROVoH2hKrnABrv1y5ZMxr NPhZZU3/RuB2iBr6NRqjc8E5+BZjlPzU9CKR/aa4QRoH40GiMrixWdTdHeWceF2dZfGa 777o+tni/j/cxuVYg0Z54+7RjR9NywW+Td2SaDkTujHd7fdVaNrofxBzkb5iJJHkoN6n Cy6g== X-Gm-Message-State: AA+aEWaWJpVsUK1z0uuTSg4DcB/zQJi+EQLoj2JGBEpDwPJj7ca1yDPS 8zVgJTcZa0hEGzvmIqQg8qv8ukMuPHw= X-Google-Smtp-Source: AFSGD/VqGq8w+0ZXx0K25qfrXFbg3qOjA8ZHA7ArwOX9A+N5/ILQ2D84oJI7WyxLpQPpa115puSjKA== X-Received: by 2002:a17:906:6c14:: with SMTP id j20-v6mr36875826ejr.41.1546547855279; Thu, 03 Jan 2019 12:37:35 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:34 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:19 +0100 Message-Id: <20190103204026.23512-5-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 04/11] support/download: implement source-check in hg backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Note: the implementation is different (better) than what used to be in Buildroot before source-check was removed. The original implementation: hg incoming --force -l1 would only verify that the repository exists, not that the requested revision is present. An already better implementation is: hg incoming --force -l1 -r but compared to the next solution it has a large resource consumption on the local machine. In the background, the full repository is first downloaded. The implemented solution is: hg identify -r which operates directly on the remote repository. Signed-off-by: Thomas De Schampheleire --- support/download/hg | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/support/download/hg b/support/download/hg index efb515fca5..7c698ee7e3 100755 --- a/support/download/hg +++ b/support/download/hg @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the changeset exists in the remote repository # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET. @@ -19,6 +20,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";; @@ -36,6 +38,11 @@ _hg() { eval ${HG} "${@}" } +if [ -n "${checkonly}" ]; then + _hg identify ${verbose} "${@}" --rev "'${cset}'" "'${uri}'" > /dev/null + exit ${?} +fi + _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'" _hg archive ${verbose} --repository "'${basename}'" --type tgz \ From patchwork Thu Jan 3 20:40:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020493 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Wi+GakoH"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W0983Lyqz9s7h for ; Fri, 4 Jan 2019 07:37:48 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 26264862ED; Thu, 3 Jan 2019 20:37:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id h2M1V1Ss+6w3; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id A8CDC8633E; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 2D8B31C2A15 for ; Thu, 3 Jan 2019 20:37:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2B57985C1D for ; Thu, 3 Jan 2019 20:37:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FzBD1LUMKcQp for ; Thu, 3 Jan 2019 20:37:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 7622585C09 for ; Thu, 3 Jan 2019 20:37:38 +0000 (UTC) Received: by mail-ed1-f68.google.com with SMTP id b3so30179975ede.1 for ; Thu, 03 Jan 2019 12:37:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=eCeFnpyV7n+n+NNo2JoNNgV3zzzWHA1UlZuUoSC1K38=; b=Wi+GakoH8EHHAuuISrdYo3EQzNZUoZ+4LmL+HqmxmVpqt6rw+eR5PRNAkPnTDAjeVR mHI96iHgeQ4//zf75mqRXkCJ7M4+Kk1J8Zg/POLZ5g0tghwnW2+zuVuGNPuityRrTqw2 6U5OW4/dE9OKP/bMmNmJGpBS4LgRv028AfD6i2bHWhdMFFmbXsQ6M+8oxb6g72SeB2RR 633iR+tTVYqlbVmdiAgtGa9xZd1/HerR4JUb4aQxWf5QHAfnRKUisC8YEHpRHAwGFLSP xEvfIrcZWWNiFLECklpW2LuPawdGXhBaXjnbwDKBO65XvBVKkzEecIFtyGg5K4e2duDj ZG0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=eCeFnpyV7n+n+NNo2JoNNgV3zzzWHA1UlZuUoSC1K38=; b=d9q0cr0wSzbVtYySLodns1Yx+a4z7K2t3J5ReAQjd9obvl9zC5dgVNP1xYzgu0Fm8l fcZxtAysgleWGltpHDEFD395cxwCaT8nOsnlg5I2JXDtYC9iNKEqktVq5ywxgLH1AqXz atbPZ736MxvM/jn1M3rJIAg/gVsAx4O4iuRP2CEDWEays1BN+9yoZMA4gTsXfgKbw1Rg vcDZRQYXXMK0Vmp6pXfzpNfig4gZTLJNz9aHawGfRvZzoCQb2vVsZmUoO5mcWolb6i58 m4+ncpSTt4FkT1Et4agT88Xku5uCCFjseuIc/mM2iQVCh0Hbz2LlmpAGl0WrwxwT9aoe Qeqw== X-Gm-Message-State: AA+aEWbvCERQu6PGoYLukxWgAVkAU+mVV7dLCCs9RlGLexWD30Ze3HQI sySB/GuOv2aIwq+LdM0mvJWxuOPdIGc= X-Google-Smtp-Source: AFSGD/XrOqPb7c0iDRJY7rIiue1Qt9pLm356Et5SsK1pwTeWbO+Q6UOl1mcY3mqu88+0Q75R85Mh0Q== X-Received: by 2002:a50:9724:: with SMTP id c33mr44090230edb.288.1546547856620; Thu, 03 Jan 2019 12:37:36 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:36 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:20 +0100 Message-Id: <20190103204026.23512-6-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 05/11] support/download: implement source-check in wget backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Signed-off-by: Thomas De Schampheleire --- support/download/wget | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/support/download/wget b/support/download/wget index c69e6071aa..7f626e0071 100755 --- a/support/download/wget +++ b/support/download/wget @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely # -o FILE Save into file FILE. # -f FILENAME The filename of the tarball to get at URL # -u URL Download file at URL. @@ -19,6 +20,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) url="${OPTARG}";; @@ -40,4 +42,9 @@ _wget() { # mirror [ -n "${encode}" ] && filename=${filename//\?/%3F} +if [ -n "${checkonly}" ]; then + _wget --spider ${verbose} "${@}" "'${url}/${filename}'" + exit ${?} +fi + _wget ${verbose} "${@}" -O "'${output}'" "'${url}/${filename}'" From patchwork Thu Jan 3 20:40:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020495 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="d56pyFjF"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09C5R1hz9rxp for ; Fri, 4 Jan 2019 07:37:51 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 04CDB85C1D; Thu, 3 Jan 2019 20:37:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id thEdIXTuf-_Z; Thu, 3 Jan 2019 20:37:46 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5DA9085C4B; Thu, 3 Jan 2019 20:37:46 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id DA7181C2A15 for ; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B737C86356 for ; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NUxQNFIrB49k for ; Thu, 3 Jan 2019 20:37:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by whitealder.osuosl.org (Postfix) with ESMTPS id B4C38862ED for ; Thu, 3 Jan 2019 20:37:39 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id g22so30056134edr.7 for ; Thu, 03 Jan 2019 12:37:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iizFFcSmDp3jsxzMIfAIbAXdCp8pa/xYopnakhJEhUQ=; b=d56pyFjFwF26zFf8HJE/H8o4bQ9EL4eNp2O5piLumd0nFrvGDlE37SZBckTQSd1hAW o+pfNuvElu52k6wad6e0Hkn6NaMjwWIlW5mGnflLmTD1k1NtG+ev3JizivAQ4sFjMc/m gCunJeEPlazcb26rWdVosZ7epvy2SF6UeyuOmuIkYjs8f1zv53rcWk3CT+/bW3jPyy0Q KrVgdAyt86eZHwkdQtVP2oEea/uArap/XXr/H3Gzj+aCiDgrexVJhlIt9CW9RGqDsWsS 0YinasFfLWH5CSVrxssOKZ0a+C5jsoJ+rBhrG2FOrgSKyUibu8nezlnzNyAbmA6YOtAv ALvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iizFFcSmDp3jsxzMIfAIbAXdCp8pa/xYopnakhJEhUQ=; b=P8G/D/TEnCEbBJGXTWRtdcwOnfyRUAiJtj4latvTz0FufeTQ63KbRmWOrfOWZEtKPn oWM3245DNcBmZ7oi9rRQyaYI3AhZlrENC8Qxc/RwXc+ULHUKvoRqBtj36n2nO1M2LjD4 fD3LbNIevr0GOI4UPGFJ6xiY5jzdfp0kEOKlT5yMY+CS0Y+JcyigEVz9XD4DXuFZsqrs etGcw75Jx1jTERGlmAocuzgHhP+MJtu1b5XarnX9DltYR+QWD8LBDBBuhBCP2LXitDiq oV0Ffo3xhO/J4tWpYMYhoyKVBX4hto7/pd0U2zf92d6qR+8TMTf8jYooa2T8+fYP9pRa WtFw== X-Gm-Message-State: AA+aEWZWK7MBBMNR7Kjj5q3rRslPNJ95jm8jNfdBoa76LCvQRWqCup5I hTvjqQFbK7hBsxgZ28PwFjm2Az3pg88= X-Google-Smtp-Source: AFSGD/UW3fNtxAncLouEGXICF9raDMrSTRWg+isxdAqYwwrxubNsrtvV8t0bB3dJ7ncDVtIUl9nHtQ== X-Received: by 2002:a50:a55c:: with SMTP id z28mr43801412edb.124.1546547857954; Thu, 03 Jan 2019 12:37:37 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:37 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:21 +0100 Message-Id: <20190103204026.23512-7-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 06/11] support/download: implement source-check in file backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Signed-off-by: Thomas De Schampheleire --- support/download/file | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/support/download/file b/support/download/file index e52fcf2c8c..3a92fa9f67 100755 --- a/support/download/file +++ b/support/download/file @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the source file exists # -o FILE Copy to file FILE. # -f FILE Copy from basename file FILE. # -u DIR Copy from FILE in DIR. @@ -23,6 +24,7 @@ verbose=-v while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=;; + C) checkonly=1;; o) output="${OPTARG}";; f) file="${OPTARG}";; u) dir="${OPTARG}";; @@ -39,4 +41,12 @@ _localfiles() { eval ${LOCALFILES} "${@}" } -_localfiles ${verbose} "'${dir##file://}/${file}'" "'${output}'" +# Remove any scheme prefix +dir="${dir##file://}" + +if [ -n "${checkonly}" ]; then + test -e "'${dir}/${file}'" + exit ${?} +fi + +_localfiles ${verbose} "'${dir}/${file}'" "'${output}'" From patchwork Thu Jan 3 20:40:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020492 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AU1wUZb5"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09754J1z9rxp for ; Fri, 4 Jan 2019 07:37:47 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6180185C2E; Thu, 3 Jan 2019 20:37:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gk87MXdIvYmL; Thu, 3 Jan 2019 20:37:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id B0E9F85C14; Thu, 3 Jan 2019 20:37:44 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id A56B41C2A15 for ; Thu, 3 Jan 2019 20:37:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A2EBB85C1D for ; Thu, 3 Jan 2019 20:37:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6b_tWpiswP19 for ; Thu, 3 Jan 2019 20:37:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f67.google.com (mail-ed1-f67.google.com [209.85.208.67]) by fraxinus.osuosl.org (Postfix) with ESMTPS id F038C85C09 for ; Thu, 3 Jan 2019 20:37:40 +0000 (UTC) Received: by mail-ed1-f67.google.com with SMTP id f9so30062168eds.10 for ; Thu, 03 Jan 2019 12:37:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wNJzaVT74E1+t1bYWlgHHRTngbsfbw3AaS3otyKhNHQ=; b=AU1wUZb5GxawZYCK5Bh6gT37Yx0ZkwpvkJzwRsmznwO4rAgXF6DJtvnAaVc/dX7c2a GOCvRK3CKU0KXT+Cbjf2192sQcd694DuGTMprrGf3naLIXXO6QceW2rFkyYeXril+782 mm/1g8tUH3So/u9TVlbRUKMXnL1u7hEQLztYrzez++elg9PpwvUZC38I6KumMToETirb 68UyJfIaYbCCoDStFSYJ0Nucuw3R6GFYxYuUlk6oRmoU+Oqknv4cG3Azwk5SMysb2N28 0Rpjc0jMl6rhkVq5XizADQgEYd5N4YGozjt4yCE0ZlKK1nRwE84nKXjDLLEhr945TfOU htFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wNJzaVT74E1+t1bYWlgHHRTngbsfbw3AaS3otyKhNHQ=; b=cU0qqiZQjR0X3/xYn4Axjc+NgAn2yDIVEKn4pWReg4JEEuTgxmtLivpZjeJhjr/QqK NdahhKUBg8pj7yr27L4vN6YIcJKhwnEmouXN/hxo5KaMHmhhPwsvXbynGSnRtmlMr41C KmaS5vQD/7k3TMvR/+NUbQPc6i6w/LTrUOL07p9bzmJG0qnrk6UJRgFacF0ElMPYVNU6 nYlz6w7LoXNiASwaJ5ZjpXt94wcc3MLlgso02Ysrc0LfhPSv9VTjzozjQSmyZV+0Ij41 rviiCJkFzh6T9ChBYd5/GXt5G41uIRchqa9NUw5L+ZWuDIaHOl7iGqIP/viuJpF6Rdkm dtvw== X-Gm-Message-State: AA+aEWZGNyorWkFrVGYcsl6tvq38YuLyXbVkO/JmTkY7l4L8HctOy07O LqqIjTYG9Omc49eASpUSowTQ9I49N/o= X-Google-Smtp-Source: AFSGD/U9A+VBRlTSRYnclyBdAC6TtSewpDNbKf/de04q/12u9sPhUTJbK6kY1oquTPPYlNrsIRXnwg== X-Received: by 2002:a05:6402:144c:: with SMTP id d12mr45433511edx.83.1546547859260; Thu, 03 Jan 2019 12:37:39 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:38 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:22 +0100 Message-Id: <20190103204026.23512-8-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 07/11] support/download: implement source-check in scp backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Note that this reintroduces BR2_SSH removed with db9473bf6cd7bd12aa1f9faad0a917c973c33827 ("core/download: drop the SSH command"). Signed-off-by: Thomas De Schampheleire --- Config.in | 4 ++++ package/pkg-download.mk | 1 + support/download/scp | 13 +++++++++++++ 3 files changed, 18 insertions(+) diff --git a/Config.in b/Config.in index f965e9d6d8..03e4eb3928 100644 --- a/Config.in +++ b/Config.in @@ -136,6 +136,10 @@ config BR2_SCP string "Secure copy (scp) command" default "scp" +config BR2_SSH + string "Secure shell (ssh) command" + default "ssh" + config BR2_HG string "Mercurial (hg) command" default "hg" diff --git a/package/pkg-download.mk b/package/pkg-download.mk index cc04e316e2..e9506ae2a5 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -15,6 +15,7 @@ export BZR := $(call qstrip,$(BR2_BZR)) export GIT := $(call qstrip,$(BR2_GIT)) export HG := $(call qstrip,$(BR2_HG)) export SCP := $(call qstrip,$(BR2_SCP)) +export SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper diff --git a/support/download/scp b/support/download/scp index 55f588e157..c8fa58f952 100755 --- a/support/download/scp +++ b/support/download/scp @@ -7,17 +7,20 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the file exists remotely # -o FILE Copy to local file FILE. # -f FILE Copy from remote file FILE. # -u URL Download file at URL. # # Environment: # SCP : the scp command to call +# SSH : the ssh command to use for checkonly verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; f) filename="${OPTARG}";; u) uri="${OPTARG}";; @@ -33,6 +36,16 @@ shift $((OPTIND-1)) # Get rid of our options _scp() { eval ${SCP} "${@}" } +_ssh() { + eval ${SSH} "${@}" +} + +if [ -n "${checkonly}" ]; then + domain="${uri%%:*}" + path="${uri#*:}" + _ssh ${verbose} "${@}" "'${domain}'" ls "'${path}'" > /dev/null + exit ${?} +fi # Remove any scheme prefix uri="${uri##scp://}" From patchwork Thu Jan 3 20:40:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020494 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="RFJAVosd"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09B1FSvz9rxp for ; Fri, 4 Jan 2019 07:37:50 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A1C9023420; Thu, 3 Jan 2019 20:37:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9neDYFCzr+Rl; Thu, 3 Jan 2019 20:37:47 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 484DA232F2; Thu, 3 Jan 2019 20:37:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 015001C327D for ; Thu, 3 Jan 2019 20:37:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id F30D885C1D for ; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id EDzNttr7W-2g for ; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 3EE3085C09 for ; Thu, 3 Jan 2019 20:37:42 +0000 (UTC) Received: by mail-ed1-f68.google.com with SMTP id b14so30071442edt.6 for ; Thu, 03 Jan 2019 12:37:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=pU3/3hss6s9/J6ZpSMUB5caTWwFULvVTEJ0Wy1XBnhQ=; b=RFJAVosdZ+pMDxZSk0hFKpLI+fx+FDdKv+30puYFgStzzoGQX+IbdM06PhLYE3tVdO q3xVpKOV6OmKCalKB3as1tjWejRaF39fZ9mAxPFv0TMInJ126Et6HKkhYJNkiWhKe6Jz GabjLUYCJ37oMgHLEahhQh5aGoxhrMkEst9sK7oi/BkmPRbo7WhmhJXymcPGF51YHapI pxg39+jjz57G9yHJg7JSAYugi3yB6SoN9kMjsXHQeuURDzsH5KKzJ26j3x+sYdjXIBU4 80Y4wxJRlUp2Em7C/qhTWH9j82dHtBcT4/CT901ijHECYTLbj/ZuOk91TYCNNnsMfnLF 0ZnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=pU3/3hss6s9/J6ZpSMUB5caTWwFULvVTEJ0Wy1XBnhQ=; b=rT7cZPUsZs9uzfNSD4jXu9C0V9XQ8EvuDIBJ8RvTlt+2QZ00WQk/B2kYLFAOd/FTPH A4RoR6umxaC0dtdRrLKwdfH+PG/ZoyKicTVO06z7/elW+kutxbvzskfctHXZKDCzUtF4 A//S8E3YZEpxfNvyqX4xxLtQgTPaYsHiKNH/kgGIN2yDs3f8n9aKC3yqPYS0fwv9IEea 38HG8EVgLZ3O7k4z5CAEx5HIn73ZB+PWmhOXD5yCYNRrThEw0X+AQjZW5Yw3+dZRSyVH k78LXKHCz6aY695bevajj68broTfowMTfxRSkESlz7DrgjU8E8xLDBiL4MUGQq3e5Erj f19w== X-Gm-Message-State: AA+aEWYxv5lS1ajmqwBDhZqOEIj5haIMpNumvjiyB/fn0YI4MU/847MO Yp/uoC0g21fRN8AnVpRWUd4kYIbGAsE= X-Google-Smtp-Source: AFSGD/XrHPnztF3uGZJ4VenXj5qX8bgujE8YrWtXS6dm5V544VuUdH8YZePE5ZRjivgxwGPrSVsn/w== X-Received: by 2002:a50:fe15:: with SMTP id f21mr42099347edt.116.1546547860308; Thu, 03 Jan 2019 12:37:40 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:39 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:23 +0100 Message-Id: <20190103204026.23512-9-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 08/11] support/download: implement source-check in git backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire The implementation is the same as originally was present. It suffers from the disadvantage that an invalid revision on a valid URL will not be detected. However, git does not seem to allow a good way to remotely check the validity of a revision, without cloning the repository. For source-check, we don't want to do such download which can be large. Signed-off-by: Thomas De Schampheleire --- support/download/git | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/support/download/git b/support/download/git index 17ca04eb98..54f3494c61 100755 --- a/support/download/git +++ b/support/download/git @@ -7,6 +7,7 @@ set -E # # Options: # -q Be quiet. +# -C Only check that the changeset exists in the remote repository # -r Clone and archive sub-modules. # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. @@ -48,6 +49,7 @@ recurse=0 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q; exec >/dev/null;; + C) checkonly=1;; r) recurse=1;; o) output="${OPTARG}";; u) uri="${OPTARG}";; @@ -61,6 +63,20 @@ done shift $((OPTIND-1)) # Get rid of our options +# Caller needs to single-quote its arguments to prevent them from +# being expanded a second time (in case there are spaces in them) +_git() { + eval GIT_DIR="${git_cache}/.git" ${GIT} "${@}" +} + +if [ -n "${checkonly}" ]; then + # TODO this check only checks that the remote repository exists, not that + # it actually contains the requested revision. And for checkonly, we want + # to avoid creating a clone first. + _git ls-remote --heads "${@}" "'${uri}'" > /dev/null + exit ${?} +fi + # Create and cd into the directory that will contain the local git cache git_cache="${dl_dir}/git" mkdir -p "${git_cache}" @@ -69,12 +85,6 @@ pushd "${git_cache}" >/dev/null # Any error now should try to recover trap _on_error ERR -# Caller needs to single-quote its arguments to prevent them from -# being expanded a second time (in case there are spaces in them) -_git() { - eval GIT_DIR="${git_cache}/.git" ${GIT} "${@}" -} - # Create a warning file, that the user should not use the git cache. # It's ours. Our precious. cat <<-_EOF_ >"${dl_dir}/git.readme" From patchwork Thu Jan 3 20:40:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020496 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="D0CENrwe"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09D5WMDz9s7h for ; Fri, 4 Jan 2019 07:37:52 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id A7A368637A; Thu, 3 Jan 2019 20:37:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id meZKTutIHgEj; Thu, 3 Jan 2019 20:37:48 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id DFEBC86365; Thu, 3 Jan 2019 20:37:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 20DCE1C2A15 for ; Thu, 3 Jan 2019 20:37:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1C041870C7 for ; Thu, 3 Jan 2019 20:37:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IMVGMAI99arT for ; Thu, 3 Jan 2019 20:37:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f65.google.com (mail-ed1-f65.google.com [209.85.208.65]) by hemlock.osuosl.org (Postfix) with ESMTPS id 40A50870A1 for ; Thu, 3 Jan 2019 20:37:43 +0000 (UTC) Received: by mail-ed1-f65.google.com with SMTP id f9so30062259eds.10 for ; Thu, 03 Jan 2019 12:37:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=loHmI8p/auT2QOPf8L09hln9fh0echZTDCj8gw5imyI=; b=D0CENrwel9/lXzgcmuYAvSb84/+MhAA9k1x0y4HZAwqfIHrMMnrv6+Bld3b2QhKfQ7 Ww+HRECkHyPgtrwMVAABSI8+30jsiVNTxqUaQWZiFpzw+TkB3fCSANUKjh5OC7PjXadH 0Ab4/TvyFr2t0CFLS3QjKN9gc9Vti7zPo2B52UEqIxYYFCiSYkxs2HeRbIWsz9Nso4yd 072rJWp82FH5mcqfW8pJOYn7du+8Pkz2mRJjgXB07P7RgQnym5bhRy2kamU/6CHCdqBN 2J7FO/8Mb1GzWp21Ps5g71yWzC97K6RQwfPeyhAhlPydesPHeY/u3qNwn/SNSLvdQzUl NmhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=loHmI8p/auT2QOPf8L09hln9fh0echZTDCj8gw5imyI=; b=D2kNDwJgQZEC7KDpYLcjQ8+DsTl6iHMVtqJdgyuy7g0Ogdo5Nv40xn9tFOHxvHdNX4 LR6dslJq5LYJqsN3a9SWpnQQnWA3LlbTdWyF9qlpIHrDOLBiU2dh7uXlgBNf2Phn9CTv 3RtO/npLFg1a9/HkdSjAf8qUxIcR9GzxQes2slQwhi5OuBMOR561La4/GOvsmU1LUV9R paC2wFXKoheWdqHRZVz7CPVSIY3Rptn/hJ+7Q3Urr5YXvjRser61SA+rxMejnrnF8M7e 1tC1uFI3z0xYkWC55NJtBfBueUMylczuaPOpcVTixSndEMrKFgKFvhOCL/SqZi/MxJMt FQMw== X-Gm-Message-State: AA+aEWbM+QJq1DSusKIAujhelXMnGrW+1SGzwqWIMR0PBchOLEci3dNx zCBdTp+QvgXfjhpuq62xdjFC9L8DnEA= X-Google-Smtp-Source: AFSGD/WTcyQldj8EpaIH/QKiTTOWsL5Pkqbgc3HNVXi+1K21CWdQVAD0ZogQqrOnx3GZHhD01zNLsg== X-Received: by 2002:a17:906:90d9:: with SMTP id v25-v6mr37149477ejw.214.1546547861392; Thu, 03 Jan 2019 12:37:41 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:40 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:24 +0100 Message-Id: <20190103204026.23512-10-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 09/11] support/download: implement source-check in bzr backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire The implementation is the same as originally was present. It suffers from the disadvantage that an invalid revision on a valid URL will not be detected. If someone is proficient in bzr-speak, please improve. Signed-off-by: Thomas De Schampheleire --- support/download/bzr | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/support/download/bzr b/support/download/bzr index 5289a421cd..2f13a607c9 100755 --- a/support/download/bzr +++ b/support/download/bzr @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet +# -C Only check that the changeset exists in the remote repository # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET. @@ -20,6 +21,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";; @@ -37,6 +39,13 @@ _bzr() { eval ${BZR} "${@}" } +if [ -n "${checkonly}" ]; then + # TODO this check only checks that the remote repository exists, not that + # it actually contains the requested revision. + _bzr ls --quiet "${@}" "'${uri}'" > /dev/null + exit ${?} +fi + # --per-file-timestamps comes with bzr-2.2 (released August 2010), # so only pass it if bzr is recent enough. We compute versions as: # major*1000 + minor From patchwork Thu Jan 3 20:40:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020497 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="YA4GpBsT"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09K32Z8z9s7h for ; Fri, 4 Jan 2019 07:37:57 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id F08F185C62; Thu, 3 Jan 2019 20:37:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id B0LhEWpTiKwu; Thu, 3 Jan 2019 20:37:49 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2351585C67; Thu, 3 Jan 2019 20:37:49 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 7D13B1C2A15 for ; Thu, 3 Jan 2019 20:37:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 75C55232D7 for ; Thu, 3 Jan 2019 20:37:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VXRZU1g0OS8L for ; Thu, 3 Jan 2019 20:37:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) by silver.osuosl.org (Postfix) with ESMTPS id 706242338F for ; Thu, 3 Jan 2019 20:37:44 +0000 (UTC) Received: by mail-ed1-f68.google.com with SMTP id h15so30088488edb.4 for ; Thu, 03 Jan 2019 12:37:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=g4BN/ENPs4z2IHUVYA4Vz7LG1x6NqDdHL5ZmWileXms=; b=YA4GpBsTI8DI9AGFK3isAb3oMauKqx6DeC0GyFoX7uvI58P1NnrwF8wgeMpdKdrC2i Y5yFfi5RjPBdjHT9JSRKUYM2/1urSo0tv9e8PaVGZFGoTEQyAcTXO8sD305xK/RdJcU7 1d8googkZJdSrBHY9NIbzKy6FhPnOsNneK1rqHBpsbaaCrC4PD595D0iXY7LVy/YEIsH DTjgmQipGulpxCpbEuVZKVrLlSh4RbH+hWfPaGqtic2z46F4XY1D7NUlXp+cTUismZx+ tMc9qjqmwhPI8ozA+Hv3+VcIUj0FcXuRIh5b5Y/sOiiSVjLl9RChnD4Op7GjzkcQtpKE Iquw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=g4BN/ENPs4z2IHUVYA4Vz7LG1x6NqDdHL5ZmWileXms=; b=n0zQfqpMbhbRj58/5eTUbIx9orRfKp0bwU/OSf3D2f+xn5PIi5er8nfXY3t0KT9/P2 iyJYzgAWWbK8Zy4Ud8WFeUvZpC+4Gseg5PYg5GwBng58RWz/7/Iuf0xg1A8GOpt26bSg Z7Lsw7+vG7ZM1HA5Pn6ed3v9PLXlmLT0mIuwwM/qYtixNZxvewnnmHHJzdPu86N5U7th ZDhBv7qVNZDaax65XIdWcSEvIMPrjhObttyl3oY/EaP+HDjhh4T+AXg4BpA0crnQhW0q ULSE4MrVFxK8h2YqLKrbYpTlKQGT2v/kEbjCPjwP/rrnN6Q+qAcAlhvptox6omyL1PXU 7nsQ== X-Gm-Message-State: AA+aEWbA+lO7vP8+DqJqhT7oPZhKZrQLoCTkoMDAu2+juzUxWr6IMW8F V59Hj/ym07PBT0fJasfr7WZeRP3pK04= X-Google-Smtp-Source: AFSGD/U42JdpV5lCDJ5pVNCRFGBjqe++N7QFBMuwf96uFlpiRoFy8FPVRsTUK62I4GWl09wW4x+oOQ== X-Received: by 2002:a50:de88:: with SMTP id c8mr43490790edl.129.1546547862674; Thu, 03 Jan 2019 12:37:42 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:42 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:25 +0100 Message-Id: <20190103204026.23512-11-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 10/11] support/download: implement source-check in cvs backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire The implementation is the same as originally was present. It suffers from the disadvantage that an invalid revision on a valid URL will not be detected. Signed-off-by: Thomas De Schampheleire --- support/download/cvs | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/support/download/cvs b/support/download/cvs index 9d0dc3cb3a..0ae7aa30d2 100755 --- a/support/download/cvs +++ b/support/download/cvs @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet +# -C Only check that the revision exists in the remote repository # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV. @@ -20,6 +21,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-Q;; + C) checkonly=1;; o) output="${OPTARG}";; u) uri="${OPTARG#*://}";; c) rev="${OPTARG}";; @@ -57,6 +59,15 @@ if [[ ! "${uri}" =~ ^: ]]; then fi export TZ=UTC + +if [ -n "${checkonly}" ]; then + # Not all CVS servers support ls/rls, use login to see if we can connect. + # TODO this check only checks that the remote repository exists, not that + # it actually contains the requested revision. + _cvs ${verbose} -d"'${uri}'" login + exit ${?} +fi + _cvs ${verbose} -z3 -d"'${uri}'" \ co "${@}" -d "'${basename}'" ${select} "'${rev}'" -P "'${rawname}'" From patchwork Thu Jan 3 20:40:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1020498 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="UDVjVUme"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43W09L6fcGz9s7h for ; Fri, 4 Jan 2019 07:37:58 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5628285C67; Thu, 3 Jan 2019 20:37:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4USA7BCbyvWK; Thu, 3 Jan 2019 20:37:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id CDA0285D08; Thu, 3 Jan 2019 20:37:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 2E9FA1C2A15 for ; Thu, 3 Jan 2019 20:37:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 24E388630A for ; Thu, 3 Jan 2019 20:37:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IPna20wmu0P4 for ; Thu, 3 Jan 2019 20:37:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f68.google.com (mail-ed1-f68.google.com [209.85.208.68]) by whitealder.osuosl.org (Postfix) with ESMTPS id B341C86372 for ; Thu, 3 Jan 2019 20:37:45 +0000 (UTC) Received: by mail-ed1-f68.google.com with SMTP id x30so30131715edx.2 for ; Thu, 03 Jan 2019 12:37:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mcQObzknjLBMFyRL/FhY66X2aMTdvG4QDDfxBIQyDH0=; b=UDVjVUmevwDO9snVlbEOioRb23kDUdFeTIH0541Ni57plZwVRyC7GpwDu0JN8VH7CL tm4fVpHdSKFdR1ij2/J1ZDTzGQV4DFtnTEX3KputoWtAMnlGpFYmy6WeubRZCdtl7W/V tb1/8JsbPfjUCH22jV5Pa1Ts6hlJQwfGJ9JABLgBQ9qI9+v7U2SwCmpYVJLN7tNlHOXW 8vhFJWf1XgY+BccgYXJeY+0+2mqyI6gQM0y8PJEjlVg3dMNEIAkiEPMIefNjKr1taVVV 5ANUaqJ2WHB7Zwn7MVSmtG4JleMfVeKpTAiJTpAtiyJFfdZy4yAnnAMEB4YWEjOC4vgs AcOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mcQObzknjLBMFyRL/FhY66X2aMTdvG4QDDfxBIQyDH0=; b=DH50HvGLLaJY5nQe59FCweVr4hbu70N+TsFwS0Oa3fdDwGyIfRngFkc4XkDLQhbEMJ XUG4+RNe3Zts1irLK9I2SI+MOSYqNOtFmN2TZog6x7BHYRrH24e4AEkGJXJN9k7pZQP+ y95E7M2zWQT37t4kfpmdEDoZRJOuTHZbwkhxI1kUHtPRC+qsVd72qnSQv+xhh/tWpnxC YMso+u9Mp9D6TPbVAsJP9OJjeRFfWu2ca9IzhvqZBCwbJKSD7KW0TIY/1LOkbtfdLRth 8jndRIIzdTW4fzTURr8XLxedWM6xE+p/BIaVoZe7ujIWEfDwi81vEMYuv9uj1hIN5PxI r2+A== X-Gm-Message-State: AA+aEWbnNrTYfvked+FT3t7HRuqMzIxAMDex3T4hIwLWgoMeakJ44qvd FRoakd74CUIMVOhKcG6C1zIQKdLMUTE= X-Google-Smtp-Source: AFSGD/WZTAmC8ZKhAak7i4DjCkg37XJdDfHYSu0IGe8uT9K9/cvUfAJgTAZTPPF5EWDA/XHiKR4Lug== X-Received: by 2002:a50:8863:: with SMTP id c32mr42299785edc.156.1546547863924; Thu, 03 Jan 2019 12:37:43 -0800 (PST) Received: from localhost.localdomain (d54C493B0.access.telenet.be. [84.196.147.176]) by smtp.gmail.com with ESMTPSA id p1-v6sm13452926ejo.49.2019.01.03.12.37.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 03 Jan 2019 12:37:43 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Thu, 3 Jan 2019 21:40:26 +0100 Message-Id: <20190103204026.23512-12-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: <20190103204026.23512-1-patrickdepinguin@gmail.com> References: <20190103204026.23512-1-patrickdepinguin@gmail.com> Subject: [Buildroot] [PATCH 11/11] support/download: implement source-check in svn backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Signed-off-by: Thomas De Schampheleire --- support/download/svn | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/support/download/svn b/support/download/svn index 542b25c0a2..e6f41f24fc 100755 --- a/support/download/svn +++ b/support/download/svn @@ -7,6 +7,7 @@ set -e # # Options: # -q Be quiet. +# -C Only check that the revision exists in the remote repository # -o FILE Generate archive in FILE. # -u URI Checkout from repository at URI. # -c REV Use revision REV. @@ -19,6 +20,7 @@ verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in q) verbose=-q;; + C) checkonly=1;; o) output="${OPTARG}";; u) uri="${OPTARG}";; c) rev="${OPTARG}";; @@ -36,6 +38,11 @@ _svn() { eval ${SVN} "${@}" } +if [ -n "${checkonly}" ]; then + _svn ls ${verbose} "${@}" "'${uri}@${rev}'" > /dev/null + exit ${?} +fi + _svn export ${verbose} "${@}" "'${uri}@${rev}'" "'${basename}'" tar czf "${output}" "${basename}"