From patchwork Wed Aug 22 21:10:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 961061 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="R6MI+sQw"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41wgFY2LQHz9s0n for ; Thu, 23 Aug 2018 07:11:13 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 26F9D86799; Wed, 22 Aug 2018 21:11:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nYS9-ZH_a0nl; Wed, 22 Aug 2018 21:11:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id E79CC8678B; Wed, 22 Aug 2018 21:11:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 902401C1217 for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8ED24226A2 for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JRWc56FOkTJl for ; Wed, 22 Aug 2018 21:11:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f41.google.com (mail-wm0-f41.google.com [74.125.82.41]) by silver.osuosl.org (Postfix) with ESMTPS id B395C24C39 for ; Wed, 22 Aug 2018 21:11:04 +0000 (UTC) Received: by mail-wm0-f41.google.com with SMTP id s9-v6so3207054wmh.3 for ; Wed, 22 Aug 2018 14:11:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=MfgbvoRmsKmN6Ivn16QEJRXj0OADY0w2uIAYYo0Nat0=; b=R6MI+sQwxW3J507/0wZ9mEJSEuy6Z5ptH31BJq1Te0x3DZ2x282Yu6xKd0oKmVHy0R 1KtUwdILURO+1BoCd2g1e8PW1WHP2Ag5g7OGfvzRUy7mnpXLkwUZwwOEqZYDf16wNiQf +pAxT9ry/XC22KrejGY8zOrNsPyiR6+sJNCRNuUNAVVPI/E1/dD8xQ6LUmx/O7tkHcQi m3j+mkn+8lNvCCNlryKcsQmvPn9oZNdgHBtLVkNo9WgUEYrq+TM1SZy1S0YRGZ+Jz+qa 0Y/r+8D1et1cVIN1ZxyT0ukISz1uNImHFvrMBeLw6eVOS0ywhCAvoxhurR4ZC70QO2OV mFTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=MfgbvoRmsKmN6Ivn16QEJRXj0OADY0w2uIAYYo0Nat0=; b=HPCgSfw66+pH56qzaCR+MFAwp4ofJPgGkY7Ixfdhuc9J972C3EHmFwLpmTtCSuFhNn cH93HOwvwJrpdbXHaruALy+Sp5WEICUPHS1hTHd1R6GaawU+njxGsoDRiG4srffWXj5o dUZD/qVn7Jx076zUgWYZ4n0u37OOcuJldwMzEcmoWYfhKO8nEGZ14kKXU3FcGwY38Kue HeYnm53mBJsDMFx/E45TMu2XetoDYJIHF3l3w4TaOJKIslTks4GAt1NhZwFQKW2j7CQf FvzKZf3AulyyBZZ5Bb5NfsHw2YPg5swDDqiRXFA+rHs2utsBDbc04B8UuXbHpfELVvp2 bPTw== X-Gm-Message-State: APzg51D1ubOTbHN/aJm/FhEGSWORcVJjLSBfh+zXCGgz1uqyvRyaTfJH zDz5zmw6kN1v5bn4rTXQaL/lNikL X-Google-Smtp-Source: ANB0Vda7p4qb00mPfUWjWXB9Nk/oM8k0YIWrO/5kJDiVpXfsrVUXLnKejj5DG9hr0FcYdMQzJdxNzw== X-Received: by 2002:a1c:8a04:: with SMTP id m4-v6mr3264208wmd.137.1534972262790; Wed, 22 Aug 2018 14:11:02 -0700 (PDT) Received: from scaer.bzh.lan (2a01cb08861073009c77937a9139bfde.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:9c77:937a:9139:bfde]) by smtp.gmail.com with ESMTPSA id z14-v6sm2691971wma.18.2018.08.22.14.11.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Aug 2018 14:11:02 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Wed, 22 Aug 2018 23:10:57 +0200 Message-Id: X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 4/5] core/download: add per-download timeout X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Hollis Blanchard , "Yann E. MORIN" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In case an remote sitre is slow, or hangs for whatever reasons, of the backend is somehow stuck (e.g. locally waiting on a lock that is never released), the whole build can be stuck forever. But in such circumstances, it may happen that another download location (e.g. a mirror on the LAN) may already have the required tarball, and downloading from there would be faster and would succeed. Add an optional, configurable, per-backend timeout. Note: all the FDs of a backend will be forcibly closed by the kernel when the backend 15-terminates or is 9-killed; any lock held on those FDs would be automatically released by the kernel, thus releasing any concurrent download. Signed-off-by: "Yann E. MORIN" Cc: Hollis Blanchard Cc: Maxime Hadjinlian --- Note: with the follow-up patch, this would probably fix build issues like the ones on Hollis' autobuilder: http://autobuild.buildroot.org/results/ddb/ddbc96b24017f2a2b06c6091dea3e19520bf2dd1/ --- Config.in | 14 ++++++++++++++ package/pkg-download.mk | 1 + support/download/dl-wrapper | 15 ++++++++++++--- 3 files changed, 27 insertions(+), 3 deletions(-) diff --git a/Config.in b/Config.in index 6b5b2b043c..654734855a 100644 --- a/Config.in +++ b/Config.in @@ -312,6 +312,20 @@ endif endmenu +config BR2_DL_TIMEOUT + string "Download timeout" + help + Timeout after which a non-completed download will be considered + failed. Suffix with 's' (or nothing), 'm', 'h', or 'd' for, + respectively, seconds, minutes, hours, or days. + + When a download times out, the next download location, if any, is + attempted, which means that, if the primary site (if any) is too + slow, then upstream will be used; if that is then still too slow, + then the backup mirror, if any, is used. + + Leave empty for no timeout (the default). + config BR2_JLEVEL int "Number of jobs to run simultaneously (0 for auto)" default "0" diff --git a/package/pkg-download.mk b/package/pkg-download.mk index cd77ca5394..832346d3d5 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -92,6 +92,7 @@ endif define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(EXTRA_ENV) $(DL_WRAPPER) \ + -t '$(call qstrip,$(BR2_DL_TIMEOUT))' \ -c '$($(PKG)_DL_VERSION)' \ -d '$($(PKG)_DL_DIR)' \ -D '$(DL_DIR)' \ diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 723c89b7d5..fff27db497 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -23,11 +23,11 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG - local backend output hfile recurse quiet rc + local backend output hfile recurse quiet rc timeout timeout_cmd local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":hc:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":hc:d:D:o:n:N:H:rf:u:t:q" OPT; do case "${OPT}" in h) help; exit 0;; c) cset="${OPTARG}";; @@ -40,6 +40,7 @@ main() { r) recurse="-r";; f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; + t) timeout="${OPTARG}";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; @@ -53,6 +54,14 @@ main() { error "no output specified, use -o\n" fi + if [ -n "${timeout}" ]; then + # Timeout after the specified delay; additionaly, leave + # 30 more seconds for the backend to properly terminate + # (e.g. to cleanup behind itself), after which forcibly + # kill the backend. + timeout_cmd="timeout --kill-after=30s ${timeout}" + fi + # Legacy handling: check if the file already exists in the global # download directory. If it does, hard-link it. If it turns out it # was an incorrect download, we'd still check it below anyway. @@ -123,7 +132,7 @@ main() { # directory to remove all the cruft it may have left behind, and try # the next URI until it succeeds. Once out of URI to try, we need to # cleanup and exit. - if ! "${OLDPWD}/support/download/${backend}" \ + if ! ${timeout_cmd} "${OLDPWD}/support/download/${backend}" \ $([ -n "${urlencode}" ] && printf %s '-e') \ -c "${cset}" \ -d "${dl_dir}" \