From patchwork Tue May 1 08:44:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 906977 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="JYgbAR2L"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Zw220b56z9rvt for ; Tue, 1 May 2018 18:44:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 5BAF6880CB; Tue, 1 May 2018 08:44:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OG+EqfV7UasY; Tue, 1 May 2018 08:44:20 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 77E3F880DE; Tue, 1 May 2018 08:44:20 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 30BB41C0DEF for ; Tue, 1 May 2018 08:44:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2C1F9864EE for ; Tue, 1 May 2018 08:44:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z1AqIdud-SaK for ; Tue, 1 May 2018 08:44:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 2161486475 for ; Tue, 1 May 2018 08:44:18 +0000 (UTC) Received: by mail-wm0-f67.google.com with SMTP id a67so9263924wmf.3 for ; Tue, 01 May 2018 01:44:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=6OTk2QampXpUOLc0DS9t8pVGCCv3S7NYIduCqaBZCbo=; b=JYgbAR2Lkg1UYhA0//tKhvgXGEi5VBs64PX00bG94HL8+VXZ2ntqce9943HszJQBIn b8xWCsGvYyKQjkoMFvzqW8QUDgTnVyFUNdWMAllhF8xbW5epxW007JRLBv0H3Vhlafhq l5HQIINEf9zU8+0hePhXMhUUB7em9YEkfZFy7GSnj8/2p5A/zRy4lE783HyuT2glYC+F rumaxNnalhtye7l2Mo4zi1hLSJmDw7ULtw79gPslZG4IwOZVYlk8SxcBXBQvasK6bkxY f3FTV2lEW0BMaNWdp2lb5hNB/F1Apflu8E2Hqzvh5ZHQfXfchOnAGE51AwyhMkADE6Ss Rfxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=6OTk2QampXpUOLc0DS9t8pVGCCv3S7NYIduCqaBZCbo=; b=gb93/WpHEOc9cRpfGQxa/EDIZ2bfQqMiKJ6W5+QytfqB+xx1ZDYhMIKFPTsrVPgz4J RpbWFUDCv+eLtCtHrW/A3CieG+qlwwYnY7OHIXPuvAzbpXwpm6AhVUBTy4E2zXB4ZroX 4hmExR5oD2tmNdQdOBkhLijWdtzUUaSBIpQtG4Bq6T8uiYkyxvSNlSCKSchW7erA4Ysg Wpt5+XSTQw2chr5tRhaYRv3WGMxFZYKJ6XdYbUlLbjnUzIHme477iAmY2F2BVbR4GvdA NxuZEQXIViAV+UpyT8i+BO4HjFRYNccO8Vxn+EzMq4Wcbu8FaK1yjyS5MvjKg7394EEt 1sdQ== X-Gm-Message-State: ALQs6tCGJiEiyt3JYfF7wTIQJ1KSp/LfiasXbr5hS0sehgARP7Nom1FT BkBprfag69uHR/BBkBwRNxpfQg== X-Google-Smtp-Source: AB8JxZrK74znBVjkTnbfG1pfeD+KjGaeor+7hdtIr1C3uLcsegUyWJF0kttwxGcI8Fv1PwT4jq3+zQ== X-Received: by 10.28.24.11 with SMTP id 11mr9955219wmy.21.1525164256493; Tue, 01 May 2018 01:44:16 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb088610730058f24ca3e844065b.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:58f2:4ca3:e844:65b]) by smtp.gmail.com with ESMTPSA id t189sm11019720wmf.22.2018.05.01.01.44.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 01:44:15 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 1 May 2018 10:44:09 +0200 Message-Id: <5e2081998de90532c315df1aaeb064e5324bee4d.1525164241.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 1/7 v5] download/git: add warning not to use our git cache X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski , "Yann E. MORIN" , Thomas Petazzoni , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" We really want the user not to use our git cache manually, or their changes (committed or not) may eventually get lost. So, add a warning file, not unlike the one we put in the target/ directory, to warn the user not to use the git tree. Ideally, we would have carried this file in support/misc/, but the git backend does not have access to it: the working directory is somewhere unknown, and TOPDIR is not exported in the environment. So, we have to carry it in-line in the backend instead. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Cc: Thomas Petazzoni Cc: Ricardo Martincoski Cc: Arnout Vandecappelle Reviewed-by: Ricardo Martincoski Tested-by: Ricardo Martincoski --- support/download/git | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/support/download/git b/support/download/git index bf05c595a5..3b5c8a6cfe 100755 --- a/support/download/git +++ b/support/download/git @@ -43,6 +43,24 @@ _git() { eval GIT_DIR="${git_cache}/.git" ${GIT} "${@}" } +# Create a warning file, that the user should not use the git cache. +# It's ours. Our precious. +cat <<-_EOF_ >"${dl_dir}/git.readme" + IMPORTANT NOTE! + + The git tree located in this directory is for the exclusive use + by Buildroot, which uses it as a local cache to reduce bandwidth + usage. + + Buildroot *will* trash any changes in that tree whenever it needs + to use it. Buildroot may even remove it in case it detects the + repository may have been damaged or corrupted. + + Do *not* work in that directory; your changes will eventually get + lost. Do *not* even use it as a remote, or as the source for new + worktrees; your commits will eventually get lost. +_EOF_ + # Initialise a repository in the git cache. If the repository already # existed, this is a noop, unless the repository was broken, in which # case this magically restores it to working conditions. In the latter From patchwork Tue May 1 08:44:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 906976 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="GZv/bcwl"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Zw203WXSz9rvt for ; Tue, 1 May 2018 18:44:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C3E188521E; Tue, 1 May 2018 08:44:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id V674XjDX-uxo; Tue, 1 May 2018 08:44:21 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1E4518086A; Tue, 1 May 2018 08:44:21 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 07CC71C0DEF for ; Tue, 1 May 2018 08:44:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 051C822158 for ; Tue, 1 May 2018 08:44:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id betU9IGNTp07 for ; Tue, 1 May 2018 08:44:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by silver.osuosl.org (Postfix) with ESMTPS id 17C2222052 for ; Tue, 1 May 2018 08:44:19 +0000 (UTC) Received: by mail-wm0-f67.google.com with SMTP id t11so16896947wmt.0 for ; Tue, 01 May 2018 01:44:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=BslPkCsQZIwzcZGPVRM75cIfF6phTvnnGDyON502F+4=; b=GZv/bcwl+9YyKQTjOUm/S3HojjNZBfA4TVikhYpkrBfMRwXXmgMCK8e1dcM3cmhBsy zdpmLnI4mjk1OiVbgPR6eprLaS9VFvz3nZM+6o3AxQX89otWPvFr7j9d4HSMOf2LQ/fa 0Vg/YIhXy2NRevaAlHkQncGDEuaEeBip9kmcqFipzUwGbqrwON3STNlL0e7rvRNVsRNi uVoY9U68yQiRs675OFchETY4eM3LrTDtE/vL3kFhmGfg7oaJrgacGe/7f9NLx5KIddMf KST4i+39HNhbyyBgNvu7kDByv39PJsp/0gtOWaSmFNoj3Nllm+/1L5a7JDc6l7cQNtvc SqRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=BslPkCsQZIwzcZGPVRM75cIfF6phTvnnGDyON502F+4=; b=e5LaQGdCjLUG45+i4TQPZmIAOAdqgpTbdZie/2y3lkISFzhr2l/LDLn/4IDNXl1GBm CYPsPBVN0rFhYQ1WMtIRZPrR9useVloocLjGnqFVbqPA06LvesTtlCD7Ir7mWRFSKaBR 5nwTSM+ZXh1sAd/A6BHuODjzuhaKIS4T8oJGfElSHZqSykkrEJShE9qMTa8nbgP9rPlp gZlDD109VndvDNm7BL7QSWJ4l1oh+5JPbe/rJbn28WyMLECB8Keedm+j6i46gzs6ZtoS 0iuLK8/mDaCJ9x+tBNvFa7z2NlU5evzhBD2g59XIwXrnjKm+d2Wt+0cSgFJpBvtC1ZEK rl4g== X-Gm-Message-State: ALQs6tAnL4uIVeZB6lo83SZN1C5H/dyYtM5iecWXDw2/3CmljJfCLLOm 87bVhk2bNv5cnjJ6FpSO0V33xA== X-Google-Smtp-Source: AB8JxZqd3I/kI/pNR3P8cK1gd1JVSkXcdoJY1IGlYrlH2GM6+azwy/UHDVYg5tHjP2EBuLvi3zGTGQ== X-Received: by 10.28.69.132 with SMTP id l4mr9129706wmi.142.1525164257428; Tue, 01 May 2018 01:44:17 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb088610730058f24ca3e844065b.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:58f2:4ca3:e844:65b]) by smtp.gmail.com with ESMTPSA id t189sm11019720wmf.22.2018.05.01.01.44.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 01:44:16 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 1 May 2018 10:44:10 +0200 Message-Id: <4391a8c286ace9244f750cca693bd265095cbb9e.1525164241.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 2/7 v5] download/git: run all git commands in the current directory X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski , "Yann E. MORIN" , Thomas Petazzoni , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" That way, we can pushd earlier, which will help with last-ditch recovery in a followup commit. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Cc: Thomas Petazzoni Cc: Ricardo Martincoski Cc: Arnout Vandecappelle Reviewed-by: Ricardo Martincoski Tested-by: Ricardo Martincoski --- support/download/git | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/support/download/git b/support/download/git index 3b5c8a6cfe..60d6c24f1e 100755 --- a/support/download/git +++ b/support/download/git @@ -34,8 +34,10 @@ done shift $((OPTIND-1)) # Get rid of our options -# We want to check if a cache of the git clone of this repo already exists. +# Create and cd into the directory that will contain the local git cache git_cache="${dl_dir}/git" +mkdir -p "${git_cache}" +pushd "${git_cache}" >/dev/null # Caller needs to single-quote its arguments to prevent them from # being expanded a second time (in case there are spaces in them) @@ -70,9 +72,7 @@ _EOF_ # We can still go through the wrapper, because 'init' does not use the # path pointed to by GIT_DIR, but really uses the directory passed as # argument. -_git init "'${git_cache}'" - -pushd "${git_cache}" >/dev/null +_git init . # Ensure the repo has an origin (in case a previous run was killed). if ! _git remote |grep -q -E '^origin$'; then From patchwork Tue May 1 08:44:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 906978 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Guvzndcv"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Zw236TBwz9rvt for ; Tue, 1 May 2018 18:44:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9174F854C4; Tue, 1 May 2018 08:44:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XDyexAQQ_F0c; Tue, 1 May 2018 08:44:24 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id F18B585D61; Tue, 1 May 2018 08:44:23 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id C80381C0DEF for ; Tue, 1 May 2018 08:44:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C5631864EE for ; Tue, 1 May 2018 08:44:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vB2BEHDNfyag for ; Tue, 1 May 2018 08:44:20 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 289E686475 for ; Tue, 1 May 2018 08:44:20 +0000 (UTC) Received: by mail-wm0-f65.google.com with SMTP id a8so16862483wmg.5 for ; Tue, 01 May 2018 01:44:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=e5VCcMyRXtwIu6lM62fo8uPsYW1rvNOIKWVEdAdaE+4=; b=GuvzndcvJJ+agURAbR+SAOzbeZcVchBjYOp+LwPs1ITxqP0i5eTKwXhcFz4d3ZcIHE RfCikUV0NpmIWRFQTDHh97yrAAMQjhjGIvw1Uda/9M9FVmc2I0vuP8tBwHtS4RPjZVwX RGED8K+ntY2xeRuYVXsXL8Vve9GILVznCVd4N3JINrtyFVqk6pPEEl5mE1YM8uWgb/Ee OtnD9k6H9uukan6jqQPiQrROYZrNe10Gad0ju/L2OAVpwQLPqMf/HEudQgZdpVqbegqw Dl19v8sZH/6w9AaA/4Dq4MvYnt7/mPokbsWDFn1AFkuO3AHyfGKeBZGZ5XluGEfZAZA6 ve3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=e5VCcMyRXtwIu6lM62fo8uPsYW1rvNOIKWVEdAdaE+4=; b=QnxGNcshLk2TwzKQ0oZxgaZOIh5l7+Ut1l+tyURFz1dqBBbz9/cq9+nq3MUebI+IMZ qjbIoCM1vbcJi6M8kMtbiAeTy3MtE7G2CJzeXNWg7PSbzFWhCPdcF9bkLMI93Wplj64G vY6FNrZs30ZjD14VE0i4be9NtIWNojkbGcpsxFQ6mG/wnpxaiTwzaEiLaZTb6x9MqbrJ tnV1T1n9KEoB6bZ5BhngV383iOoy26M604oE/3ZqORy4PKrNGoD0IC+Zqlyj0RqKkhI/ qJHVdrxhXeGFnjQyPOa/+WrcopGJ7lHTondBZhxU4+/cxoXLaE75qKkOcwB3oxLGCaTT EsdQ== X-Gm-Message-State: ALQs6tDkbuq3r8tGWLUKQGYApudGq/3jUdOgjWaonvgLXkcLFUr5uXTo FZrC4vLRmW7WGnwDxOI5Qy/61A== X-Google-Smtp-Source: AB8JxZrB0Y5UZrhdldyBsYPlBpiQu56MdCU4s3nxDn9w/XOv5+C9BZQHyzT4zhKG7jB6bMxHu4LVFg== X-Received: by 10.28.224.68 with SMTP id x65mr8939931wmg.83.1525164258518; Tue, 01 May 2018 01:44:18 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb088610730058f24ca3e844065b.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:58f2:4ca3:e844:65b]) by smtp.gmail.com with ESMTPSA id t189sm11019720wmf.22.2018.05.01.01.44.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 01:44:18 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 1 May 2018 10:44:11 +0200 Message-Id: <70922ed1e84a5c9f87f7414e788299b080e85388.1525164241.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 3/7 v5] download/git: quickly exit when the cset does not exist X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski , "Yann E. MORIN" , Thomas Petazzoni , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Check that the given cset is indeed something we can checkout. If not, then exit early. This will be useful when a later commit will trap any failing git command to try to recover the repository by doing a clone from scratch: when the cset is not a commit, it does not mean the repository is broken or what, and re-cloning from scratch would not help, so no need to trash a good cache. Reported-by: Thomas Petazzoni Signed-off-by: "Yann E. MORIN" Cc: Thomas Petazzoni Cc: Maxime Hadjinlian Cc: Ricardo Martincoski Cc: Arnout Vandecappelle Reviewed-by: Ricardo Martincoski Tested-by: Ricardo Martincoski --- support/download/git | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/support/download/git b/support/download/git index 60d6c24f1e..bd37a0a8d9 100755 --- a/support/download/git +++ b/support/download/git @@ -114,6 +114,13 @@ if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi +# Check that the changeset does exist. If it does not, no reason to go +# on, we can fast-track to the exit path. +if ! _git rev-parse --quiet --verify "'${cset}^{commit}'" >/dev/null 2>&1; then + printf "Commit '%s' does not exist in this repository\n." "${cset}" + exit 1 +fi + # Checkout the required changeset, so that we can update the required # submodules. _git checkout -q "'${cset}'" From patchwork Tue May 1 08:44:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 906979 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="kOx9JE7m"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Zw2407Vqz9s0W for ; Tue, 1 May 2018 18:44:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 4FBAB8758B; Tue, 1 May 2018 08:44:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f2NDhmTikTMf; Tue, 1 May 2018 08:44:25 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 414BC8689B; Tue, 1 May 2018 08:44:25 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id CE8081C0DEF for ; Tue, 1 May 2018 08:44:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id CC2AA22158 for ; Tue, 1 May 2018 08:44:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o+d85sXlcHxO for ; Tue, 1 May 2018 08:44:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by silver.osuosl.org (Postfix) with ESMTPS id 2571522052 for ; Tue, 1 May 2018 08:44:21 +0000 (UTC) Received: by mail-wm0-f65.google.com with SMTP id m70so16477347wma.2 for ; Tue, 01 May 2018 01:44:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=aqz9PS+qv5gTfwoxPE6wbJodbXQn+MG47H29k6zWVkI=; b=kOx9JE7mEnI5QrcX56DtA8yGkV76oA8HvgSH2kl0Jn+BQimuPkkVfITw0sAlCgVa91 xAUHvDQmBYumtIm1PTzoCYHkeqT5qvMclejAI/eT/qaq/SemOZxXrifXFrgqeEX3qctO gvSQEL+BypS9l6XVyeza1vbNW0C7Xz4Cbj18v3EusEkC6kVFyXRSACpgTzeTXY3JXctS bwd8er9HQhcs/GqN+qU6yiMppgHOhVHc4rZVFVl/H/D+IHVadTWQVXCmEso2VIOighxm izJ+GgzVfXblL0ubMkps/fzfwxzSWhoI3Rh6Ycx6BhAcP9mKlemYQJluQGnedBvH9k0C D4MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=aqz9PS+qv5gTfwoxPE6wbJodbXQn+MG47H29k6zWVkI=; b=Z0Lb+BXMW/gMUlyIZGjoIJtg1Lqu8ezQOVRwWh2EC1RJNk/iKmzbbqIIqLwyvkGONk LDsKMcimtrELn2tZwsN4/S44uVN8/KZDTtHneWMTAx9IURXmYPsBPzgHEMuYRtpJGSWJ 8bXtSqiZ5BuLlF7DD9nOKpe/IY7Lk10C4KeC1TxuDDAeCUHYP+ut7K61Sh73UXdwYf8S RP3KOw+aIlY01ymqHIwhFuuY1zwhvbZSW3XO/umy5dOt9aLDgWRzb28wWQ5ldaRgofM5 zCe8SVHe4jOHqAqaAcMFcLPosXtVs69aPDwV+d5p0pEW/I9LRIj3BuwUFGM+CcnSObbh IBFw== X-Gm-Message-State: ALQs6tCqMKcgFu/q0id1rtJ2mWdHLFBzBVKEEIpLG/Z6XQT7pB6KvNNb NqKaoOPQD0eH3kRS/vznysqQYg== X-Google-Smtp-Source: AB8JxZqpZPNt4JnvnyXfNSoEcHWA3VAyMgWgp0yZ+pQ5eQ/gP74aWFYtd3GsFysA1aC95QHFpYrr4Q== X-Received: by 10.28.126.141 with SMTP id z135mr8952646wmc.6.1525164259472; Tue, 01 May 2018 01:44:19 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb088610730058f24ca3e844065b.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:58f2:4ca3:e844:65b]) by smtp.gmail.com with ESMTPSA id t189sm11019720wmf.22.2018.05.01.01.44.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 01:44:18 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 1 May 2018 10:44:12 +0200 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 4/7 v5] download/git: try to recover from utterly-broken repositories X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski , "Yann E. MORIN" , Thomas Petazzoni , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In some cases, the repository may be in a state we can't automatically recover from, especially since we must still support oldish git versions that do not provide the necessary commands or options thereof. As a last-ditch recovery, delete the repository and recreate the cache from scratch. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Cc: Thomas Petazzoni Cc: Ricardo Martincoski Cc: Arnout Vandecappelle --- support/download/git | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-) diff --git a/support/download/git b/support/download/git index bd37a0a8d9..39e57aac34 100755 --- a/support/download/git +++ b/support/download/git @@ -1,7 +1,7 @@ #!/usr/bin/env bash # We want to catch any unexpected failure, and exit immediately -set -e +set -E # Download helper for git, to be called from the download wrapper script # @@ -16,6 +16,33 @@ set -e # Environment: # GIT : the git command to call +# Save our path and options in case we need to call ourselves again +myname="${0}" +declare -a OPTS=("${@}") + +# This function is called when an error occurs. Its job is to attempt a +# clone from scratch (only once!) in case the git tree is borked, or in +# case an unexpected and unsupported situation arises with submodules +# or uncommitted stuff (e.g. if the user manually mucked around in the +# git cache). +_on_error() { + local ret=${?} + + printf "Detected a corrupted git cache.\n" >&2 + if ${BR_GIT_BACKEND_FIRST_FAULT:-false}; then + printf "This is the second time in a row; bailing out\n" >&2 + exit ${ret} + fi + export BR_GIT_BACKEND_FIRST_FAULT=true + + printf "Removing it and starting afresh.\n" >&2 + + popd >/dev/null + rm -rf "${git_cache}" + + exec "${myname}" "${OPTS[@]}" || exit ${ret} +} + verbose= recurse=0 while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do @@ -39,6 +66,9 @@ git_cache="${dl_dir}/git" mkdir -p "${git_cache}" pushd "${git_cache}" >/dev/null +# Any error now should try to recover +trap _on_error ERR + # Caller needs to single-quote its arguments to prevent them from # being expanded a second time (in case there are spaces in them) _git() { @@ -114,8 +144,9 @@ if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi -# Check that the changeset does exist. If it does not, no reason to go -# on, we can fast-track to the exit path. +# Check that the changeset does exist. If it does not, re-cloning from +# scratch won't help, so we don't want to trash the repository for a +# missing commit. We just exit without going through the ERR trap. if ! _git rev-parse --quiet --verify "'${cset}^{commit}'" >/dev/null 2>&1; then printf "Commit '%s' does not exist in this repository\n." "${cset}" exit 1 From patchwork Tue May 1 08:44:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 906980 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DbWjsqG5"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Zw263x9qz9rvt for ; Tue, 1 May 2018 18:44:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 2BD7F231AC; Tue, 1 May 2018 08:44:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rQ5tZ8SWe2NZ; Tue, 1 May 2018 08:44:26 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 0690E22158; Tue, 1 May 2018 08:44:26 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id C16BE1C0DEF for ; Tue, 1 May 2018 08:44:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id BD75D880DE for ; Tue, 1 May 2018 08:44:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F4HEnWNzkBJI for ; Tue, 1 May 2018 08:44:22 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by hemlock.osuosl.org (Postfix) with ESMTPS id 11D07881C7 for ; Tue, 1 May 2018 08:44:22 +0000 (UTC) Received: by mail-wm0-f65.google.com with SMTP id a67so9264101wmf.3 for ; Tue, 01 May 2018 01:44:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=mHtbYUCYC2aQTlOXxDuanIggpfWpeZBJ43hDWuTlRVM=; b=DbWjsqG5rzPKK/RqC9BulXdVxvYihiGAJmPIQgz8597OTeEVrze7g6ZadfsKmkiH73 /N4XvuQuPNfhvlzvmeRmOR4afbqnFMmx/z08JXtdgjhZYnBs5DOOlDu9Cufi+vjXYBX7 Y6aWkT0yEC7mprOfAkeZNHmUUf3mRv0kaAYGIDKImROuVCLzugSnmBGxfNsSdDIpsjfY a9EdfouvEViXkHX/joc5J9Bq/jaUeTseeXaadPGFUcxP277GTNpoHi5ZU2Y5B0g/AZPI Y+C6Dre8dGYZcfw8efBZXbc81UK1SFy2KOaqqRdBeqfGhi4L5dggUxoQpfxlQyoyI+oI XarA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=mHtbYUCYC2aQTlOXxDuanIggpfWpeZBJ43hDWuTlRVM=; b=YVDXsG3I/oNKLH5Hw6nlzcqDywBczdoI8wSBgh7gC/WkF3seljC4HLXxUVP4eerwB6 Y+QS+I9PGaOPdrFbmvwPKQUEx1WmrZ8ApCxs2c2AXjkxCOSMCrw2oQT4+XSFHjpntSwK bXpT8FRVnJekW8AMYAsTFUOAaYIBVmEim3ulH7bH8q+yS2sc6TWemJA4LeIRWgRGjXrL KIX0+48X8xoh+mpXsbPHgo+wbRDnZpVsN8VETWHbFzXP9PnEFOQM0iJYTHgD/BGkwKbd ZxaDtCAbu08XoUZ9gqBgeMD+iQnQRWw6svnqPeHoMfR9G9rUcPCm2Ti4+8YhdnpI90tl gdHQ== X-Gm-Message-State: ALQs6tCfHGIlr8dNwVmFn6tO9hK0YSNCFD94o+3Soc3fq6k3A8xEq/bc aDzf4rfr/18tEsfvLKw2CHT4Gw== X-Google-Smtp-Source: AB8JxZo9mi4IyoHP0+whuptWck0ih9heettOk5QwEenUlgC8uIRmziTGm/i+Rj9kCBbyWkBqPEQQIQ== X-Received: by 10.28.87.148 with SMTP id l142mr10043243wmb.123.1525164260452; Tue, 01 May 2018 01:44:20 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb088610730058f24ca3e844065b.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:58f2:4ca3:e844:65b]) by smtp.gmail.com with ESMTPSA id t189sm11019720wmf.22.2018.05.01.01.44.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 01:44:19 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 1 May 2018 10:44:13 +0200 Message-Id: <00cc8a121634c0ba040e4cbb6ab078c49f075263.1525164241.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 5/7 v5] download/git: ensure we checkout to a clean state X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , "Yann E. MORIN" , Ricardo Martincoski , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Force the checkout to ignore and throw away any local changes. This allows recovering from a previous partial checkout (e.g. killed by the user, or by a CI job...) git checkout -f has been supported since the inception of git, so we can use it without any second thought. Also do a forced-forced clean, to really get rid of all untracked stuff. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Cc: Ricardo Martincoski Cc: Thomas Petazzoni Cc: Arnout Vandecappelle Reviewed-by: Ricardo Martincoski Tested-by: Ricardo Martincoski --- support/download/git | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/support/download/git b/support/download/git index 39e57aac34..c88249ee63 100755 --- a/support/download/git +++ b/support/download/git @@ -154,7 +154,11 @@ fi # Checkout the required changeset, so that we can update the required # submodules. -_git checkout -q "'${cset}'" +_git checkout -f -q "'${cset}'" + +# Get rid of now-untracked directories (in case a git operation was +# interrupted in a previous run). +_git clean -ffdx # Get date of commit to generate a reproducible archive. # %cD is RFC2822, so it's fully qualified, with TZ and all. From patchwork Tue May 1 08:44:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 906981 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="D+0bATwD"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Zw273t99z9s0W for ; Tue, 1 May 2018 18:44:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 7C45E8869D; Tue, 1 May 2018 08:44:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QHrp+sQnDUeW; Tue, 1 May 2018 08:44:26 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id AAF3B880F2; Tue, 1 May 2018 08:44:26 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id DEC4A1C0DEF for ; Tue, 1 May 2018 08:44:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id DC68A22158 for ; Tue, 1 May 2018 08:44:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TIwyuF5sUAED for ; Tue, 1 May 2018 08:44:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f177.google.com (mail-wr0-f177.google.com [209.85.128.177]) by silver.osuosl.org (Postfix) with ESMTPS id 3097522052 for ; Tue, 1 May 2018 08:44:23 +0000 (UTC) Received: by mail-wr0-f177.google.com with SMTP id q3-v6so10235557wrj.6 for ; Tue, 01 May 2018 01:44:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=90ITgpEnkAWCeo7U//pWuJwdEU0AUpuDNq6hKSN06Lo=; b=D+0bATwDGXjB1y0SMhtzATFX/GOWw0tevvrUM/7uOmncBX+DQAQ15FtRvnAyC3HMco rvIku/wtup5hIchWNFDjpfBEsOMEiQhVgNQ5jx7MSxaT84+Zb835FgOCVnE2nys5SJ49 gWBUZqSg2Flkqu7QIKQZh4frzORXut0KFrBLJYrlj4DAmc4TqxR+sBd+O+fIuv4jtani iK0/jaVgbVBivDFe5DsbdbfSOgDfAOTTcVHW9M50KX7ov5oL4poQaPY/Esno4opZ+H/s fH3rhvk3NhZN4tBSsjoUCelt5ZvTZ3UcL99l0rAqbttMpP95K3vR8XH5yC2llkZ6xi+c m2fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=90ITgpEnkAWCeo7U//pWuJwdEU0AUpuDNq6hKSN06Lo=; b=r03JhNwTAxNmima36j5anG6oyutCn3nrCfjspnDmekcayXi9tzd+H4krqfGhIOJ/NB b8efBR+b1SN7Uawt6LdW7TtpQAauOVx4jW7aEyKmJlHZg5P9S+LmJwc5CSq02983d1JE 2jKHvSSUjVwkOvKZyfgJuzJ7aJ3Nzs4MjFn5thghp1D/hiRjTen8Fa1hvOUhijDC3+4d RXglYRxHpuNMtgnap3nZ06OJKNsZ8eqBmhTTe+x83p0sp+BIGISMY3CWxkkhfK1Ti3kj EmJVtZeIBwKb6Te9C1HioDYJYP3YW8r4bgN/DA8xOlVsErDftPOK4lOdER48G50+CIVl DcSA== X-Gm-Message-State: ALQs6tCXLoLAcQUesEXkWZd8+kS2WdlAkkrGvWN5CmWv8IYDciw3QBTm 59y47hCgX25PhRbYGkX7WwgyHA== X-Google-Smtp-Source: AB8JxZq9XCEYYley9KwHVkG59PJgmxWisgFw6PVQgrpWv1qxhR0b6AvXWVd2zggrr67kHEobzmKvdA== X-Received: by 2002:adf:b356:: with SMTP id k22-v6mr10711732wrd.207.1525164261508; Tue, 01 May 2018 01:44:21 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb088610730058f24ca3e844065b.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:58f2:4ca3:e844:65b]) by smtp.gmail.com with ESMTPSA id t189sm11019720wmf.22.2018.05.01.01.44.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 01:44:20 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 1 May 2018 10:44:14 +0200 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 6/7 v5] download/git: ensure we can checkout repos with submodule conversions X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski , "Yann E. MORIN" , Thomas Petazzoni , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" When a git tree has had sub-dir <-> sub-module conversions, or has had submodules added or removed over the course of time, checking out a changeset across those conversions/additions/removals may leave untracked files, or may fail because of a conflict of type. So, before we checkout the new changeset, we forcibly remove the submodules. The new set of submodules, if any, will be restored later. Ideally, we would use a native git command: git submodule deinit --all. However, that was only introduced in git 1.8.3 which, while not being recent by modern standards, is still too old for some enterprise-grade distributions (RHEL6 only has git-1.7.1). So, instead, we just use git submodule foreach, to rm -rf the submodules directory. Again, we would ideally use 'cd $toplevel && rm -rf $path', but $toplevel was only introduced in git 1.7.2. $path has always been there. So, instead, we just cd back one level, and remove the basename of the directory. Eventually, we need to get rid of now-empty and untracked directories, that were parents of a removed submodule. For example. ./foo/bar/ was a submodule, so ./foo/bar/ was removed, which left ./foo/ around. Yet again, recent-ish git versions would have removed it during the forced checkout, but old-ish versions (e.g. 1.7.1) do not remove it with the forced checkout. Instead we rely on the already used forced-forced clean of directories, untracked, and ignored content, to really get rid of extra stuff we are not interested in. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Cc: Thomas Petazzoni Cc: Ricardo Martincoski Cc: Arnout Vandecappelle Reviewed-by: Ricardo Martincoski Tested-by: Ricardo Martincoski --- support/download/git | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/support/download/git b/support/download/git index c88249ee63..931694f89c 100755 --- a/support/download/git +++ b/support/download/git @@ -152,12 +152,39 @@ if ! _git rev-parse --quiet --verify "'${cset}^{commit}'" >/dev/null 2>&1; then exit 1 fi +# The new cset we want to checkout might have different submodules, or +# have sub-dirs converted to/from a submodule. So we would need to +# deregister _current_ submodules before we checkout. +# +# Using "git submodule deinit --all" would remove all the files for +# all submodules, including the corresponding .git files or directories. +# However, it was only introduced with git-1.8.3, which is too recent +# for some enterprise-grade distros. +# +# So, we fall-back to just removing all submodules directories. We do +# not need to be recursive, as removing a submodule will de-facto remove +# its own submodules. +# +# For recent git versions, the repository for submodules is stored +# inside the repository of the super repository, so the following will +# only remove the working copies of submodules, effectively caching the +# submodules. +# +# For older versions however, the repository is stored in the .git/ of +# the submodule directory, so the following will effectively remove the +# the working copy as well as the repository, which means submodules +# will not be cached for older versions. +# +cmd='printf "Deregistering submodule \"%s\"\n" "${path}" && cd .. && rm -rf "${path##*/}"' +_git submodule --quiet foreach "'${cmd}'" + # Checkout the required changeset, so that we can update the required # submodules. _git checkout -f -q "'${cset}'" # Get rid of now-untracked directories (in case a git operation was -# interrupted in a previous run). +# interrupted in a previous run, or to get rid of empty directories +# that were parents of submodules removed above). _git clean -ffdx # Get date of commit to generate a reproducible archive. From patchwork Tue May 1 08:44:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 906982 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Jfh9ztDb"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40Zw2B4jctz9rvt for ; Tue, 1 May 2018 18:44:34 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 2E7C22637A; Tue, 1 May 2018 08:44:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vc-XT2edOO-q; Tue, 1 May 2018 08:44:27 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 528F422158; Tue, 1 May 2018 08:44:27 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id C103E1C0DEF for ; Tue, 1 May 2018 08:44:24 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id BEC35880F2 for ; Tue, 1 May 2018 08:44:24 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JHbW9fu6UCET for ; Tue, 1 May 2018 08:44:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f194.google.com (mail-wr0-f194.google.com [209.85.128.194]) by hemlock.osuosl.org (Postfix) with ESMTPS id 15EC6880DE for ; Tue, 1 May 2018 08:44:24 +0000 (UTC) Received: by mail-wr0-f194.google.com with SMTP id i14-v6so7070820wre.2 for ; Tue, 01 May 2018 01:44:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=jAIJoxJKG2wnRM5VHQ8Wimutth3qhPtBhosKvWVxeI4=; b=Jfh9ztDb+JvSdBZx51MFu1PmygChLpbegbaP1rJRcLkqy/GPb1BH+CrWjQqAeYI7Z0 rn62dP3Wh4fEEYqzSCzrj8+IACK944RgKerCvXKi/t5fM8rFAfLxyplTmjtTvbScyno4 b8Ep4CCOZ6UaGVQ0loHh2xVhXIxT/R7cMRRuXuwBi5LTJWxMeUjc5a8SrH8jXDpRr45f QMz/4SDHXWKg056cqjTgClKCQz922vFi6IvaQZQOrdroV50aqKcaoMidXZ8I6Zri5+5U pe+jd4YgwHsew8oDWh123f5RGGmLqc0J1QwcyD3EhCHz45Se61WDRUdiI3r7pLo4KQxv HA7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=jAIJoxJKG2wnRM5VHQ8Wimutth3qhPtBhosKvWVxeI4=; b=XtjsQLVo352G6kWBZhq1sLwKMepe7FKAr3yFnV+qm1xlS1eI8N0u9mlwDNQFBn0RIL d6QsWxqVKwzojxtizrh0iiYefBRS1/VCd79/1aDkOLAsp2f51/B7cnmOs49S5Sbxg+h6 LY8JduU+Bfw3Z6S6/4oqD9Itej1S6Q3lhIXYnKOfMx5KWJQFU7H+vSHkQ0lO5VcXC3+w E+uGHGWZ+9pU57nTVTuM+Y/FbcVH5tqwT73WSZ00TyLec8wafgmrHijtn5rYhS30j6nI iQyMVbLe3COCoKfnuR9r4gWi9Pnuy1TK907Kxx8at+HigliRNM8EY7NqdQ6jv0kGcBu6 6v2A== X-Gm-Message-State: ALQs6tAw6Rfwpowi4P0Ivui+HDvc+K+LBE0lo2tKQlQgtADf/Zf+kqpN qqO2Yckmu8MWB1/6a0pc0DnJMA== X-Google-Smtp-Source: AB8JxZq0CLXPaooIUvc8zwZVFKaHNax5X5hV98gkyn07hjPV63k7heXPTrRH5ZnGGRjoq4KgkTRdZA== X-Received: by 2002:adf:c4c4:: with SMTP id o4-v6mr11025403wrf.173.1525164262459; Tue, 01 May 2018 01:44:22 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb088610730058f24ca3e844065b.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:58f2:4ca3:e844:65b]) by smtp.gmail.com with ESMTPSA id t189sm11019720wmf.22.2018.05.01.01.44.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 01 May 2018 01:44:21 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 1 May 2018 10:44:15 +0200 Message-Id: <657e448ced76f18222d9987baf7bca48d550f13b.1525164241.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 7/7 v5] download/git: always do full-clone X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ricardo Martincoski , "Yann E. MORIN" , Thomas Petazzoni , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" We currently attempt a shallow clone, as tentative to save bandwidth and download time. However, now that we keep the git tree as a cache, it may happen that we need to checkout an earlier commit, and that would not be present with a shallow clone. Furthermore, the shallow fetch is already really broken, and just happens to work by chance. Consider the following actions, which are basically what happens today: mkdir git git init git cd git git remote add origin https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git git fetch origin --depth 1 v4.17-rc1 if ! git fetch origin v4.17-rc1:v4.17-rc1 ; then echo "warning" fi git checkout v4.17-rc1 The checkout succeeds just because of the git-fetch in the if-condition, which is initially there to fetch the special refs from github PRs, or gerrit reviews. That fails, but we just print a warning. If we were to ever remove support for special refs, then the checkout would fail. The whole purpose of the git cache is to actually save bandwidth and download time, but in the long run. For one-offs, people would preferably use a wget download (e.g. with the github macro) instead of a git clone. We switch to always doing a full clone. It is more correct, and pays off in the long run... Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian Cc: Thomas Petazzoni Cc: Ricardo Martincoski Cc: Arnout Vandecappelle --- support/download/git | 24 +++--------------------- 1 file changed, 3 insertions(+), 21 deletions(-) diff --git a/support/download/git b/support/download/git index 931694f89c..11bb52c1e1 100755 --- a/support/download/git +++ b/support/download/git @@ -111,27 +111,9 @@ fi _git remote set-url origin "'${uri}'" -# Try to fetch with limited depth, since it is faster than a full clone - but -# that only works if the version is a ref (tag or branch). Before trying to do -# a shallow clone we check if ${cset} is in the list provided by git ls-remote. -# If not we fallback to a full fetch. -# -# Messages for the type of clone used are provided to ease debugging in -# case of problems -git_done=0 -if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then - printf "Doing a shallow fetch\n" - if _git fetch "${@}" --depth 1 origin "'${cset}'"; then - git_done=1 - else - printf "Shallow fetch failed, falling back to fetching all refs\n" - fi -fi -if [ ${git_done} -eq 0 ]; then - printf "Fetching all references\n" - _git fetch origin - _git fetch origin -t -fi +printf "Fetching all references\n" +_git fetch origin +_git fetch origin -t # Try to get the special refs exposed by some forges (pull-requests for # github, changes for gerrit...). There is no easy way to know whether