From patchwork Wed Aug 22 21:10:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 961059 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="gsmR1zlx"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41wgFT2BzFz9s3Z for ; Thu, 23 Aug 2018 07:11:09 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3A0A2867B5; Wed, 22 Aug 2018 21:11:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 807bsJu6RALq; Wed, 22 Aug 2018 21:11:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1A44986687; Wed, 22 Aug 2018 21:11:04 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 441D21C1217 for ; Wed, 22 Aug 2018 21:11:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 41EC687DAC for ; Wed, 22 Aug 2018 21:11:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Hk6c8EOct1cL for ; Wed, 22 Aug 2018 21:11:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by hemlock.osuosl.org (Postfix) with ESMTPS id 986F987DB2 for ; Wed, 22 Aug 2018 21:11:01 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id g33-v6so2756660wrd.1 for ; Wed, 22 Aug 2018 14:11:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=03DzX76P5zXQ8m7GbfxK6EGfRTRXFhwxOu/XYPpagok=; b=gsmR1zlxCYrkml2fd2sEt6tSDy/0/3PlF4AeaW2Ww1BoKCDxpGL95QcfjyD7Op2Ari VX3eFSLKU2s0DeGrdjAQL/mr3bMPMxyJAu76n9n9P4L/lds4vq8mLdarSApcM1tzwCop JSUtwEbEHposyLvctZvAcHKhAViYPmMhPzk/fo8wg5B9xf1ETj4DzhSqbS/S1LareYEP ilrXRpRMO0XQu4lgTwbQJHKIwE7LC7Sflf771FgFyIBpuYktd8Zi/utx5cvuVfqcyN/d zUr75pbUeTRXF9YHu/17KcBNfaoJYmQkNBw6YSjw4M+TgAz8luoQhvbnAYm2ffGFtyK1 8kqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=03DzX76P5zXQ8m7GbfxK6EGfRTRXFhwxOu/XYPpagok=; b=udcA5HDbUpIYr2u01tp2a5Z6mMnwtHaWUQw219lrCw5xEFNtv/hMSBbuU9kbAYtr7A 9MxKWRIWAQkxg1DjIXZOxxYYa+LCFVgGp1XKQ2SRvL7EQl7FXWdnqimMP0MU0pf13cOc OgXkKwSFYQL2mBtQS/5NafAqKBBXhvkxLnmN/YVqXOagMcDbGMVYu1nXPmZ6fyRsun2C U2+rod19Le4yvwaPhbXGUy81H5ABpmEwN1/vYzH1CeM+yiut5NSvTYjg+6VqoiBd/i8q i/ucHq+HbIHeoxWeWbLPcq81XgGqbflzlOwtr7SYkOYlc/0NlmwBlnW80u0U01G3OA8e KcNg== X-Gm-Message-State: APzg51CwXsNtrY9RYl214bs8/M3tbnxA4eG87Ii3ZKMNh07f0KokLAMn v2thR/DK6wlr/qKu0xA/RwW0UWQU X-Google-Smtp-Source: ANB0VdbN09bfwLE6+tn5ocIPYchmhHVAowiWE0kZ32YmI+TAg4WyqlQvlNqGhxt9agS02xkA9HYm9w== X-Received: by 2002:adf:9281:: with SMTP id 1-v6mr7877896wrn.69.1534972260004; Wed, 22 Aug 2018 14:11:00 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb08861073009c77937a9139bfde.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:9c77:937a:9139:bfde]) by smtp.gmail.com with ESMTPSA id z14-v6sm2691971wma.18.2018.08.22.14.10.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 14:10:59 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Wed, 22 Aug 2018 23:10:54 +0200 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 1/5] download/git: fix code-style X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This file uses leading spaces, not TABs. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian --- ;-) --- support/download/git | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/support/download/git b/support/download/git index 11bb52c1e1..17ca04eb98 100755 --- a/support/download/git +++ b/support/download/git @@ -185,7 +185,7 @@ fi # the state of the remote server. It also would generate large tarballs # (gigabytes for some linux trees) when a full clone took place. find . -not -type d \ - -and -not -path "./.git/*" >"${output}.list" + -and -not -path "./.git/*" >"${output}.list" LC_ALL=C sort <"${output}.list" >"${output}.list.sorted" # Create GNU-format tarballs, since that's the format of the tarballs on From patchwork Wed Aug 22 21:10:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 961060 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="qkw5v/X0"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41wgFV2qfXz9s0n for ; Thu, 23 Aug 2018 07:11:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id DFAE9876E4; Wed, 22 Aug 2018 21:11:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y9lv6risTJVV; Wed, 22 Aug 2018 21:11:06 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 3C4878769A; Wed, 22 Aug 2018 21:11:06 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 90BD51C1217 for ; Wed, 22 Aug 2018 21:11:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 8E50387646 for ; Wed, 22 Aug 2018 21:11:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qBMd0kAyDm78 for ; Wed, 22 Aug 2018 21:11:03 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f67.google.com (mail-wm0-f67.google.com [74.125.82.67]) by whitealder.osuosl.org (Postfix) with ESMTPS id A94E687691 for ; Wed, 22 Aug 2018 21:11:02 +0000 (UTC) Received: by mail-wm0-f67.google.com with SMTP id 207-v6so3553176wme.5 for ; Wed, 22 Aug 2018 14:11:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=hCQRdNOS1/j9+kZANCC1muQlBc6a1MrzJp6YcIYYdWo=; b=qkw5v/X0X8yBfefkY6QGJhfFddq5KJHklMtBeePYMyCohSlu7sQ7sktPMVDy8MHybG o5YYAmHsYNAdvOpXIclPexHyNddZ+KIwKbzX6JdT+BRWp6abTKNDpKDFAtR3FMH8Bjk7 BDBeo23q6eTM/CTdpNzUJApU/8lPKsHSWZrgayB8PWIold5vHoiH0L+uuDzV3WxIV85D fxvbycem/es3XJYIfvnP/Uw1/k7PTqw1ADXMTyz+82MvAgCSJs7M4RiW+I8yg1I8UPOo /KgviM1XplCT9hoaRODp1b/nIELVspRzMUgdpEw3nkKHFt7m1A8eMVuxpQ4YO5LFuQjt gKpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=hCQRdNOS1/j9+kZANCC1muQlBc6a1MrzJp6YcIYYdWo=; b=k8KE8OYAwKEwcU8IsprFsWPswCA0bFhtmPiAq54jcwLtSxZCpSgoOueeQIp0hZ3ryu XR5eRv9PjHJaUYu9KLfmDz/DPyuJmlj71fA3lYO53juyVNfU8Us9LNZmGb9g67FDdzsl rzTRHkUJpve4Urf4y0pIGQy0ay18ZGjuBqSpR/SM6QNQL6XPD7wLCMfOphlhwpBt4Sk/ 1LBO91Q+y64NujCxGK3zKmLq26vtN57Iam5lpRrH6sMt7VgSipF7T+Mi+58kVBQOkyH2 4aZQswKjyGCBcXxLZtmHuEYOk7zmkjTcxeolwBUWRfp/mx0Hbr2mnKXvIp5+te92ikA8 GW2A== X-Gm-Message-State: APzg51B51VQLfiU7ABkiS9YWA5MFbwhpJXeH4fGhBdEkxP+Oa1rqxOJu ZstOYVWqhtLr6T2U+M3QbV1JSa6e X-Google-Smtp-Source: ANB0Vdac24q3sFy94gydRLFa15jf6ZUAx1xKoeo9SeBETM2IptBL/CMoA/15sxPdUE0QQYIFPi96EA== X-Received: by 2002:a1c:1bc2:: with SMTP id b185-v6mr3672467wmb.160.1534972260955; Wed, 22 Aug 2018 14:11:00 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb08861073009c77937a9139bfde.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:9c77:937a:9139:bfde]) by smtp.gmail.com with ESMTPSA id z14-v6sm2691971wma.18.2018.08.22.14.11.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 14:11:00 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Wed, 22 Aug 2018 23:10:55 +0200 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 2/5] download/git: re-run the backend with a lock to the git tree X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The access-pattern to the git repository must be entirely atomic. This atomicity is currently guaranteed by the caller, from a higher level. But this is not optimum, and it is better that the backend does the locking it needs by itself, so that other backends from other builds can still download for the same package (e.g. another build uses a wget download for that package). One would think that, ideally, we'd have three atomic sections, and that we could release the lock in-between those sections: - setting the remote URL and fetching, - checking out, updating sub-modules, generating the archive, - compressing the archive. However, this is not true, because another backend may acquire the lock right after our first section, and decide to entirely remove the repository (in the _on_error() trap), and thus, in the second section, the commits we had fetched in the first section may now have disapeared. So, in fact, the first two sections are really only one. Now, once we have generated the tarball, we could indeed release the lock, as we no longer need access to the repository. But releasing a lock acquired by flock(1) is not trivial: we don't know the filedescriptor that was used to open the directory, so we can't close it to release the lock... So, we just lock the whole backend: once we reach compressing the archive, we're almost done anyway. We do implement this locking policy by locking the git cache directory right after it got created, and before we do any git command in there. We do so by re-running the git backend under an flock(1), using a trick explained in the manpage (except we don't use the script as lockfile, but the git cache directory). Even though using an exclusive lock is the default, we do request it, to be sure. When running under the lock, we then know we have exclusive access to the repository, and we can do whatever we need in it. When the backend ends, its filesdecriptors are closed (by the kernel), thus freeing the lock for another to acquire. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian --- support/download/git | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/support/download/git b/support/download/git index 17ca04eb98..1a3d562605 100755 --- a/support/download/git +++ b/support/download/git @@ -25,6 +25,10 @@ declare -a OPTS=("${@}") # case an unexpected and unsupported situation arises with submodules # or uncommitted stuff (e.g. if the user manually mucked around in the # git cache). +# +# To be noted: this function is only called while we have the lock on +# the git cache directory, so we can't remove it. +# _on_error() { local ret=${?} @@ -38,7 +42,10 @@ _on_error() { printf "Removing it and starting afresh.\n" >&2 popd >/dev/null - rm -rf "${git_cache}" + + # Remove everything in the git cache, but the directory itself, + # so as not to have to play tricks with the lock + find "${git_cache}" -mindepth 1 -maxdepth 1 -exec rm -rf {} \; exec "${myname}" "${OPTS[@]}" || exit ${ret} } @@ -64,6 +71,21 @@ shift $((OPTIND-1)) # Get rid of our options # Create and cd into the directory that will contain the local git cache git_cache="${dl_dir}/git" mkdir -p "${git_cache}" + +# From now-on, we need exclusive access to repository. But this git +# backend is not concurrent-safe; others instances may have to touch +# the same git repository. So, we re-run under flock if not already +# the case, but only after we've created the cache dir, so that we +# only lock the cache, not the full dl-dir. +# +# Note that if this is our second-chance (see _on_error(), above), then +# we already have the lock, and won't try to reacquire it (BR_GIT_FLOCK +# is already set to the correct value). +if [ "${BR_GIT_FLOCK}" != "${git_cache}" ]; then + export BR_GIT_FLOCK="${git_cache}" + exec flock --exclusive --no-fork "${git_cache}" "${0}" "${OPTS[@]}" +fi + pushd "${git_cache}" >/dev/null # Any error now should try to recover @@ -193,6 +215,12 @@ LC_ALL=C sort <"${output}.list" >"${output}.list.sorted" tar cf - --transform="s#^\./#${basename}/#" \ --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ -T "${output}.list.sorted" >"${output}.tar" + +# From now on, we could very well release the lock we have on the git +# cache directory, but there is no simple solution for that (we don't +# know its filedescriptor). So, just go on with our business, we're +# almost done anyway... + gzip -6 -n <"${output}.tar" >"${output}" rm -f "${output}.list" From patchwork Wed Aug 22 21:10:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 961062 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="URkWJKXL"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41wgFb3HWCz9s0n for ; Thu, 23 Aug 2018 07:11:15 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 9E13F876EF; Wed, 22 Aug 2018 21:11:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9EJpsC3YvZF2; Wed, 22 Aug 2018 21:11:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 989D9876EE; Wed, 22 Aug 2018 21:11:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 6BDCD1C1217 for ; Wed, 22 Aug 2018 21:11:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id EA7108679B for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bqRSbXNMsvhD for ; Wed, 22 Aug 2018 21:11:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by fraxinus.osuosl.org (Postfix) with ESMTPS id AF9D78677F for ; Wed, 22 Aug 2018 21:11:03 +0000 (UTC) Received: by mail-wm0-f41.google.com with SMTP id m199-v6so300292wma.1 for ; Wed, 22 Aug 2018 14:11:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=QoPLQt4w2DU27BzoxtIQvy6GjEz1b4fjM7qyrhqW5lA=; b=URkWJKXLH4HSTVX6AM8ycrUzSFj1TlAJ5r9K47W/ms7yDk105+epr1byJ5226CMKcz jJbahWJX+ccr8sljwhjtp7/5yxNIYA4mb2kYPrwFgQMpCnYsj0m3IsRnjwu0JlNDtjpb ks/AVTeXN9s65Xhhu4G9ynXyORpizTziCmFF4jSenklsIF7AEvee8TR4MoTq4i9UhMV2 Hxio+3jVpvgujVNQ+AknFTJfhPy3qV7/YzsCtGwWytK4P90KzSjFoiOnJQqmzSdLvYPp Xl9m5toAh3aaJsl333PfYG7lDyG4t/bV61+YB3vR95yVrEM8Do6yACgABe6Hqb4+NGpW 1YNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=QoPLQt4w2DU27BzoxtIQvy6GjEz1b4fjM7qyrhqW5lA=; b=cbu9/jmt/rAv2sWQN5wygtr3Bwf+K9oX1sPYFkUUE5buUPDk2HLFepJcWC4FC+dNmu IoSXip3rVTm4/bw4u28wA5bImvWeLDEijvFinrepZm8DG/uvcT1MNEERUqjS7gmNCm2F 6PvsDBpCo1CnMGt/4H4bXMh4/EFQOSsniLA9KTERPv1xQWKOU6USB/O8o8cEzxEk4rE2 W2C6bf7ahz7nEyd3MrNwx2snXOTMI89NPgVExh+8N/fJq9H0ktPZdm6iIoA3FYR2ns/s Dv6kBJjw9aDFyXavR76pRZQ9cISmPdd5QowlE51HKA91yW/SDyvaPDqPp2KvNJZXhJoi bN0g== X-Gm-Message-State: APzg51C/os+4MCB0KZtjzIAQdLpCnB9rvMh7i1WHyoxFcpVSVSzlA4H0 ctnB3UsjvmJ2rHCLeWtLsL9vBTJZ X-Google-Smtp-Source: ANB0Vdb1ZQjJiUELy7gCDGM0AzUwh7Wy2REMaPxrEi9qVUSbVrIEgE98TbmHDb9AKJAsrvdZmF9HDQ== X-Received: by 2002:a1c:30c3:: with SMTP id w186-v6mr3155154wmw.153.1534972261890; Wed, 22 Aug 2018 14:11:01 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb08861073009c77937a9139bfde.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:9c77:937a:9139:bfde]) by smtp.gmail.com with ESMTPSA id z14-v6sm2691971wma.18.2018.08.22.14.11.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 14:11:01 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Wed, 22 Aug 2018 23:10:56 +0200 Message-Id: <8c2f0ed9cee73879c6bd388ee8bf077c875b0ab8.1534972250.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 3/5] download: move locking into download wrapper X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The script as a whole does not need to be atomic. The only section that needs to be atomic is checking for, and grabbing a download from the legacy, global directory, and linking/copying it to the new location. The rest of the script goes at great length to ensure that every operations are concurrent-safe, by using temporary filenames and directories, with only the last operation being atomic to put the download in its final location. As for the backends, all but the git backend are concurrent-safe, as they work on aforementioned temporary files and directories. As for the git backend, it internally locks whatever needs to be atomic, so is concurrent-safe. Note: the trick to execute only a portion of the shell script is taken directly from the manpage for flock(1). Even though using an exclusive lock is the default, we do request it, to be extra sure. Signed-off-by: "Yann E. MORIN" Cc: Maxime Hadjinlian --- package/pkg-download.mk | 3 +-- support/download/dl-wrapper | 16 +++++++++------- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index bf93b9a08e..cd77ca5394 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -19,7 +19,6 @@ SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper -FLOCK = flock $($(PKG)_DL_DIR)/ # DL_DIR may have been set already from the environment ifeq ($(origin DL_DIR),undefined) @@ -92,7 +91,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) - $(Q)$(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ + $(Q)$(EXTRA_ENV) $(DL_WRAPPER) \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \ diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 490335c859..723c89b7d5 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -57,13 +57,15 @@ main() { # download directory. If it does, hard-link it. If it turns out it # was an incorrect download, we'd still check it below anyway. # If we can neither link nor copy, fallback to doing a download. - # NOTE! This is not atomic, is subject to TOCTTOU, but the whole - # dl-wrapper runs under an flock, so we're safe. - if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then - ln "${old_dl_dir}/${filename}" "${output}" || \ - cp "${old_dl_dir}/${filename}" "${output}" || \ - true - fi + ( + flock --exclusive 42 || exit 1 + if [ ! -e "${output}" -a -e "${old_dl_dir}/${filename}" ]; then + ln "${old_dl_dir}/${filename}" "${output}" || \ + cp "${old_dl_dir}/${filename}" "${output}" || \ + true + fi + # Yes, we can open a directory for reading! + ) 42<"${dl_dir}" || exit ${?} # If the output file already exists and: # - there's no .hash file: do not download it again and exit promptly From patchwork Wed Aug 22 21:10:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 961061 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="R6MI+sQw"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41wgFY2LQHz9s0n for ; Thu, 23 Aug 2018 07:11:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 26F9D86799; Wed, 22 Aug 2018 21:11:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nYS9-ZH_a0nl; Wed, 22 Aug 2018 21:11:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id E79CC8678B; Wed, 22 Aug 2018 21:11:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 902401C1217 for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8ED24226A2 for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JRWc56FOkTJl for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by silver.osuosl.org (Postfix) with ESMTPS id B395C24C39 for ; Wed, 22 Aug 2018 21:11:04 +0000 (UTC) Received: by mail-wm0-f41.google.com with SMTP id s9-v6so3207054wmh.3 for ; Wed, 22 Aug 2018 14:11:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=MfgbvoRmsKmN6Ivn16QEJRXj0OADY0w2uIAYYo0Nat0=; b=R6MI+sQwxW3J507/0wZ9mEJSEuy6Z5ptH31BJq1Te0x3DZ2x282Yu6xKd0oKmVHy0R 1KtUwdILURO+1BoCd2g1e8PW1WHP2Ag5g7OGfvzRUy7mnpXLkwUZwwOEqZYDf16wNiQf +pAxT9ry/XC22KrejGY8zOrNsPyiR6+sJNCRNuUNAVVPI/E1/dD8xQ6LUmx/O7tkHcQi m3j+mkn+8lNvCCNlryKcsQmvPn9oZNdgHBtLVkNo9WgUEYrq+TM1SZy1S0YRGZ+Jz+qa 0Y/r+8D1et1cVIN1ZxyT0ukISz1uNImHFvrMBeLw6eVOS0ywhCAvoxhurR4ZC70QO2OV mFTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=MfgbvoRmsKmN6Ivn16QEJRXj0OADY0w2uIAYYo0Nat0=; b=HPCgSfw66+pH56qzaCR+MFAwp4ofJPgGkY7Ixfdhuc9J972C3EHmFwLpmTtCSuFhNn cH93HOwvwJrpdbXHaruALy+Sp5WEICUPHS1hTHd1R6GaawU+njxGsoDRiG4srffWXj5o dUZD/qVn7Jx076zUgWYZ4n0u37OOcuJldwMzEcmoWYfhKO8nEGZ14kKXU3FcGwY38Kue HeYnm53mBJsDMFx/E45TMu2XetoDYJIHF3l3w4TaOJKIslTks4GAt1NhZwFQKW2j7CQf FvzKZf3AulyyBZZ5Bb5NfsHw2YPg5swDDqiRXFA+rHs2utsBDbc04B8UuXbHpfELVvp2 bPTw== X-Gm-Message-State: APzg51D1ubOTbHN/aJm/FhEGSWORcVJjLSBfh+zXCGgz1uqyvRyaTfJH zDz5zmw6kN1v5bn4rTXQaL/lNikL X-Google-Smtp-Source: ANB0Vda7p4qb00mPfUWjWXB9Nk/oM8k0YIWrO/5kJDiVpXfsrVUXLnKejj5DG9hr0FcYdMQzJdxNzw== X-Received: by 2002:a1c:8a04:: with SMTP id m4-v6mr3264208wmd.137.1534972262790; Wed, 22 Aug 2018 14:11:02 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb08861073009c77937a9139bfde.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:9c77:937a:9139:bfde]) by smtp.gmail.com with ESMTPSA id z14-v6sm2691971wma.18.2018.08.22.14.11.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 14:11:02 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Wed, 22 Aug 2018 23:10:57 +0200 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 4/5] core/download: add per-download timeout X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Hollis Blanchard , "Yann E. MORIN" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In case an remote sitre is slow, or hangs for whatever reasons, of the backend is somehow stuck (e.g. locally waiting on a lock that is never released), the whole build can be stuck forever. But in such circumstances, it may happen that another download location (e.g. a mirror on the LAN) may already have the required tarball, and downloading from there would be faster and would succeed. Add an optional, configurable, per-backend timeout. Note: all the FDs of a backend will be forcibly closed by the kernel when the backend 15-terminates or is 9-killed; any lock held on those FDs would be automatically released by the kernel, thus releasing any concurrent download. Signed-off-by: "Yann E. MORIN" Cc: Hollis Blanchard Cc: Maxime Hadjinlian --- Note: with the follow-up patch, this would probably fix build issues like the ones on Hollis' autobuilder: http://autobuild.buildroot.org/results/ddb/ddbc96b24017f2a2b06c6091dea3e19520bf2dd1/ --- Config.in | 14 ++++++++++++++ package/pkg-download.mk | 1 + support/download/dl-wrapper | 15 ++++++++++++--- 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/Config.in b/Config.in index 6b5b2b043c..654734855a 100644 --- a/Config.in +++ b/Config.in @@ -312,6 +312,20 @@ endif endmenu +config BR2_DL_TIMEOUT + string "Download timeout" + help + Timeout after which a non-completed download will be considered + failed. Suffix with 's' (or nothing), 'm', 'h', or 'd' for, + respectively, seconds, minutes, hours, or days. + + When a download times out, the next download location, if any, is + attempted, which means that, if the primary site (if any) is too + slow, then upstream will be used; if that is then still too slow, + then the backup mirror, if any, is used. + + Leave empty for no timeout (the default). + config BR2_JLEVEL int "Number of jobs to run simultaneously (0 for auto)" default "0" diff --git a/package/pkg-download.mk b/package/pkg-download.mk index cd77ca5394..832346d3d5 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -92,6 +92,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(EXTRA_ENV) $(DL_WRAPPER) \ + -t '$(call qstrip,$(BR2_DL_TIMEOUT))' \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \ diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 723c89b7d5..fff27db497 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -23,11 +23,11 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG - local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc timeout timeout_cmd local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":hc:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":hc:d:D:o:n:N:H:rf:u:t:q" OPT; do case "${OPT}" in h) help; exit 0;; c) cset="${OPTARG}";; @@ -40,6 +40,7 @@ main() { r) recurse="-r";; f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; + t) timeout="${OPTARG}";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; @@ -53,6 +54,14 @@ main() { error "no output specified, use -o\n" fi + if [ -n "${timeout}" ]; then + # Timeout after the specified delay; additionaly, leave + # 30 more seconds for the backend to properly terminate + # (e.g. to cleanup behind itself), after which forcibly + # kill the backend. + timeout_cmd="timeout --kill-after=30s ${timeout}" + fi + # Legacy handling: check if the file already exists in the global # download directory. If it does, hard-link it. If it turns out it # was an incorrect download, we'd still check it below anyway. @@ -123,7 +132,7 @@ main() { # directory to remove all the cruft it may have left behind, and try # the next URI until it succeeds. Once out of URI to try, we need to # cleanup and exit. - if ! "${OLDPWD}/support/download/${backend}" \ + if ! ${timeout_cmd} "${OLDPWD}/support/download/${backend}" \ $([ -n "${urlencode}" ] && printf %s '-e') \ -c "${cset}" \ -d "${dl_dir}" \ From patchwork Wed Aug 22 21:10:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 961063 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="TflaYPLr"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41wgFg4H1Lz9s0n for ; Thu, 23 Aug 2018 07:11:19 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6842087727; Wed, 22 Aug 2018 21:11:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id qQgG8KUIeIAI; Wed, 22 Aug 2018 21:11:13 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id A5606876F7; Wed, 22 Aug 2018 21:11:10 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id AD8921C1217 for ; Wed, 22 Aug 2018 21:11:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5336686781 for ; Wed, 22 Aug 2018 21:11:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gAqYEsjOJt5P for ; Wed, 22 Aug 2018 21:11:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f49.google.com (mail-wm0-f49.google.com [74.125.82.49]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 7F5B8867C1 for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) Received: by mail-wm0-f49.google.com with SMTP id s12-v6so3229537wmc.0 for ; Wed, 22 Aug 2018 14:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=pzL3pP9jiTw6sQB4ySqwsI1lIvQT30THkpjCD/BntFs=; b=TflaYPLrsQvdC7rCWXbuheZCaenFOdcxdjVg7epmyiyuSHuONsG7twwfDBfTelzqCN cNBzmcouDz9kMc6Mqg4Sm0+YDWkRQcI0ey9y2vBOupH0MMTo/SOuw63t5l8ZoVQMI2aq p6cp7niWhMkGokP7AraTqxToGPNSHspoGruh9jSjJTxuOa2tiVeDVVAHmgrHaOoZJVO0 AhY/MN0hjvdOev9hU/hvjUcopjlDKXT19u5XjL+EEnoC+xJiEws3BEGnF9PGAVNHPKNv kpKm84X/fNQ+KJb3oEDtvFrYJSwzJ++XTYvayP40IVAlEKDRU2kmzBjrqOKBSDe1fIOf RYPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=pzL3pP9jiTw6sQB4ySqwsI1lIvQT30THkpjCD/BntFs=; b=RVFB4EltRZc1D5u3K7jF/VupuIKtCjN/IZlZVASlv1OsJl0roQTuOfDIge3HUeoQaf ibJxDm+R1nOWElM8rjT8/u8RSz+tyGeWlKF9bSJJFLWfKhFs6HVCsw2ATJm15oPS9OZv cTEhYNQAftDPWEcKSCif3ZNma24KVqGWk9somyAwFr+T0J1lpAzQlujGx9r/1CgrhN+r P0PEPVpIODXhZ3ho/NUHT8C24A8Iymy27KsMrT88APT8auYkOMop6vhhW88uaMYKXlnp ZNq2tgYLWQZ5hB2rQojnQg5VLyPIkBSyCpHb7kiFse+X6ltk2OdQcbOFeTJUh+d4NasU KtcQ== X-Gm-Message-State: APzg51Dpixh97hPX4tVZUObwHLnIphohV/Gwvn+/sN1pA8PuW3oPOx8K b68/Z0CZeS6uX7elxZFuwxb/Y9aw X-Google-Smtp-Source: ANB0Vdbbwp1UuhI3B5Mss+Unz8GZumgoq3hE38CYbWoaKI5BMaLWlO+K3gfXOWv+xz/yEhMy3NKIKA== X-Received: by 2002:a1c:f611:: with SMTP id w17-v6mr3226752wmc.143.1534972263823; Wed, 22 Aug 2018 14:11:03 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb08861073009c77937a9139bfde.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:9c77:937a:9139:bfde]) by smtp.gmail.com with ESMTPSA id z14-v6sm2691971wma.18.2018.08.22.14.11.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 14:11:03 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Wed, 22 Aug 2018 23:10:58 +0200 Message-Id: <17a71c0c569e1b66a46ef40f14f07db8adf847b9.1534972250.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 5/5] utils/genrandconfig: add a 30-minute timeout X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Hollis Blanchard , "Yann E. MORIN" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Some autobuilders seem to have difficulties downloading big git trees. Sicne those are autobuilders, they only test known git sha1s or tags, so they eventually get cached in our primary mirror, and thus we can expect that slow autobuilders can eventually fetch them from there. Add a relatively high timeout, so that those autobuilders are not stuck forever in downloads. Should fix autobuild issues like: http://autobuild.buildroot.org/results/ddb/ddbc96b24017f2a2b06c6091dea3e19520bf2dd1/ Signed-off-by: "Yann E. MORIN" Cc: Hollis Blanchard Cc: Maxime Hadjinlian --- utils/genrandconfig | 3 +++ 1 file changed, 3 insertions(+) diff --git a/utils/genrandconfig b/utils/genrandconfig index 27f84ea555..2dbdc2ccf6 100755 --- a/utils/genrandconfig +++ b/utils/genrandconfig @@ -353,6 +353,9 @@ def gen_config(args): # Allow hosts with old certificates to download over https configlines.append("BR2_WGET=\"wget --passive-ftp -nd -t 3 --no-check-certificate\"") + # Don't be stuck forever downloading big git trees, timeout after 30 min + configlines.append("BR2_DL_TIMEOUT=\"30m\"") + # Amend the configuration with a few things. if randint(0, 20) == 0: configlines.append("BR2_ENABLE_DEBUG=y\n")