From patchwork Tue Apr 17 07:12:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ricardo Martincoski X-Patchwork-Id: 899101 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="EXk61AoT"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40QGsp3qf2z9s02 for ; Tue, 17 Apr 2018 17:22:22 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id E2ACD26162; Tue, 17 Apr 2018 07:22:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZKMbqt9beva1; Tue, 17 Apr 2018 07:22:16 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 99FC02614E; Tue, 17 Apr 2018 07:22:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 605391C06E8 for ; Tue, 17 Apr 2018 07:22:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5C98386CDF for ; Tue, 17 Apr 2018 07:22:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fpDLM5vFy6yH for ; Tue, 17 Apr 2018 07:22:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-vk0-f51.google.com (mail-vk0-f51.google.com [209.85.213.51]) by fraxinus.osuosl.org (Postfix) with ESMTPS id B5B39869E4 for ; Tue, 17 Apr 2018 07:22:13 +0000 (UTC) Received: by mail-vk0-f51.google.com with SMTP id r184so11088902vke.11 for ; Tue, 17 Apr 2018 00:22:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9WbO2ROQ/i8/+KfHi3Y/dqiUAk8Me60VCaBUklkGvSY=; b=EXk61AoTL+tFrH/xD5nnrFxtV+PxftymG9M2/cNXds2mew9rqvNGAzYS8zS8yGXYoa PQuXr4lLOKPiIGaI5dMA4h3c0kbJj1AbEUmX14AeH5dVJvlowzaScvoTrhpmCipf6lTv tjofer8NChhXHdihXyJ/hbTqqRv/yo0p6OJ4wGdWoMc+rArvB8YRHMwUDNqd/vP5irGh u9OGVw52jD3Pq+EdN16rBQZ1mSa9TvoLdJkZbxiRloCyyeSwCPcLue//BCiXpmmgKtq3 2NNfE7JTyHlS50D4+wFfC0B09cIoxZRl1Hm9ZTx8DGr++mQ/gofkdbseATkXMzKk9Sjj 3JJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9WbO2ROQ/i8/+KfHi3Y/dqiUAk8Me60VCaBUklkGvSY=; b=bzLwid58j/7W5wKTDoPrPOU3Cu9F5WXmmXihaPdejK57jp23c+akVRmKW6EULZywS6 HCU1anLe6ukZlIh3XRiP42koSJ3B/c2//Mrfk5iqH4hk56Ezr77vd0QwDAVrv8pOk9fC G024kLkwLb/Qs/zw2VWnCJeZOSewGDwfwfOj0nS4JOTP2zXLvIIj7jynskONbeBWaZjW rYsfVn0U7b+fA7N8izBayvjPcrNMs69FtZPTi6tcPNzeSbJXMKC5nEePViBJ21CR8hZj B4QM7ERRRRm/Tn5n3v00nn21Uu01PbICm+pqoqq9SeqEu5EYLOwU0/F+asXnTA7+ayO0 9fXQ== X-Gm-Message-State: ALQs6tC48TXe2oQJikZMJt4Kdut4pJqSdKRIwEF7KsRI4H9fcOOeT3kH l8Lg3lZ4bUWXaePp8JKm1M36Cw== X-Google-Smtp-Source: AIpwx48L4VhdVCmcHKUTpLAJ+QyjiOf0N1bayae1+rHRbrKCs4EOD+WV3qmGxA8OW5rRNuWWGcyF+g== X-Received: by 10.31.152.214 with SMTP id a205mr627136vke.96.1523949732478; Tue, 17 Apr 2018 00:22:12 -0700 (PDT) Received: from ultri4.home ([177.18.73.157]) by smtp.gmail.com with ESMTPSA id x20sm4689756uab.1.2018.04.17.00.22.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Apr 2018 00:22:11 -0700 (PDT) From: Ricardo Martincoski To: buildroot@buildroot.org Date: Tue, 17 Apr 2018 04:12:46 -0300 Message-Id: <20180417071246.12046-1-ricardo.martincoski@gmail.com> X-Mailer: git-send-email 2.14.1 In-Reply-To: <20180413133431.19449-1-ricardo.martincoski@gmail.com> References: <20180413133431.19449-1-ricardo.martincoski@gmail.com> Subject: [Buildroot] [PATCH v2] autobuild-run: remove only tarballs from download dir X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , Ricardo Martincoski , "Yann E . MORIN" MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The current logic that removes 5 files at random to force the download to be tested eventually ends up removing files from the git repositories used as cache by the download infra. When a file that was previously checked out in a git cache is removed, the next 'git checkout' succeeds but does not checkout that file. The repo remains with 'Changes not staged for commit', the tarball is generated and the hash mismatches, causing a build failure. Ensure only the tarballs are removed by excluding any file inside a git repository from the list of potential files to remove. The download infra for git method uses 'git fetch' when regenerating the tarball, this command fails as expected if the upstream location changed. Reported-by: Thomas Petazzoni Reported-by: Yann E. MORIN Signed-off-by: Ricardo Martincoski Cc: Peter Korsgaard Cc: Thomas Petazzoni Cc: Yann E. MORIN --- Changes v1 -> v2: (after review by Thomas Petazzoni) - add comment to the code to clarify why we test for '.git' and not 'git'; - use the most correct syntax: del; I tested this by first populating the instance-0/dl/ with 22GB of tarballs + git cache that I have locally, then I added 'continue' in the main loop of the script just after prepare_build() returned. After the script removed all tarballs from instance-0/dl/, I manually checked the tarball for git package was removed, there are git repositories and they are sane (using git fsck) and have a clean worktree (using git status). Tested with Python 2.7.14 --- scripts/autobuild-run | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/scripts/autobuild-run b/scripts/autobuild-run index 33d0ae9..5b3d1c5 100755 --- a/scripts/autobuild-run +++ b/scripts/autobuild-run @@ -291,6 +291,13 @@ def prepare_build(**kwargs): # recursively find files under root def find_files(root): for r, d, f in os.walk(root): + # do not remove individual files from git caches. 'git' can + # be either dl//git or dl/git and we want to + # eventually remove tarballs for the git package, so check + # for '.git' instead to match only dl//git/.git . + if '.git' in d: + del d[:] + continue for i in f: yield os.path.join(r, i)