From patchwork Tue Feb 5 20:24:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1037185 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="SJqLBvRo"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43vGJy51NJz9sDL for ; Wed, 6 Feb 2019 07:24:49 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 7D4BA87C5C; Tue, 5 Feb 2019 20:24:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SNyhPK1HNtGk; Tue, 5 Feb 2019 20:24:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 74C1B87BFD; Tue, 5 Feb 2019 20:24:44 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 748D51BF398 for ; Tue, 5 Feb 2019 20:24:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 71A048207A for ; Tue, 5 Feb 2019 20:24:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yt_RV7DylTI7 for ; Tue, 5 Feb 2019 20:24:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f43.google.com (mail-ed1-f43.google.com [209.85.208.43]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 09BAC80335 for ; Tue, 5 Feb 2019 20:24:41 +0000 (UTC) Received: by mail-ed1-f43.google.com with SMTP id a20so4015161edc.8 for ; Tue, 05 Feb 2019 12:24:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qpB50accHCPF3Xay92Ib49/ZgiBHVBBLJH42/wRrNRk=; b=SJqLBvRoDjnp2LBQj/LoX2SvBtynzzEEq1Vshyb415W8pjH/yMd5W0g9AUxB8wRJ0y oDlyQMXp4R/UjxbjbtTpFvSv6ReQH6fnVsPMYoh3JYdPzEzUOOoZnocSqNC/gR4Ee1TZ JJn2eApEtQHA63sIjBvbGJqIs2xKfzy7lopLOVOq0Zh3r9tk7i8Jk7xWX+6x3eQoDTX0 NZLmJKw58UB1TodITkg0rzN7/4GGPT2NqP/uzExPnPpKG+fovpwGd4RGnDf1P8oxUVaz Q7aCBhW519MeiHotOszVjiWza9XeRIPgfp175jtdgWcePO0vU9VzQFKHDkzaqRya9+SJ wcow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=qpB50accHCPF3Xay92Ib49/ZgiBHVBBLJH42/wRrNRk=; b=oafgMZCROsd0YtK+zZQw2Y12p5Kjod03imnwLwlOlwCXXi0CoEvxMOO4uBKVT6UG4j 6XKs8ScqkXVUoBTpc/Iz7UxTdE/ITUx+aPCNDzbxg165utYQKjDZt6O9IId2dyJ8hlYk p+ClY57RBvTR4kacO4gvM47RkOb9H+g+8uRqs5ciMxVu+pfLu3r6KRnR9p3mMZhqK1vi wvexBoHAo8sHlEU0SyZiwXFm7z6CM6tWPRxT0Wkz2t/uiGbIT6Il7ReHUdPok/Eihk6D l10xP8n0PTTBw9uJJIlW+pccopXIvgCNbqUDaeiKsDzn/55D007vYq2bRq/i5D6a8nks A90w== X-Gm-Message-State: AHQUAuY5E0rJ+laitF8qiBnksz4GmP1FgMdtlL7dKZsMmQ4ZQANt9Sbu h0mcr99WG1KHkF7fjPjTwYjmYhAgoTo= X-Google-Smtp-Source: AHgI3IZJg2mci6w4scz8B5I+P+ixc8JqpxTgCPEFD4qy+Eue/SHk9oChNub/5wEH4JIdJBRms6Qvnw== X-Received: by 2002:aa7:d35a:: with SMTP id m26mr5267573edr.244.1549398279114; Tue, 05 Feb 2019 12:24:39 -0800 (PST) Received: from localhost.localdomain (ptr-5ltycmajy3605r3p7a.18120a2.ip6.access.telenet.be. [2a02:1810:530:b600:aa50:f99f:a32c:5156]) by smtp.gmail.com with ESMTPSA id t23sm1795425ejo.63.2019.02.05.12.24.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 05 Feb 2019 12:24:38 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Tue, 5 Feb 2019 21:24:33 +0100 Message-Id: <20190205202433.25292-1-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.19.2 MIME-Version: 1.0 Subject: [Buildroot] [TO-BE-TESTED] support/download/hg: implement repository cache X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Similar to the git download helper, implement a repository cache for Mercurial repositories. The code is mostly guided by the implementation for git, with certain parts copied almost verbatim. Signed-off-by: Thomas De Schampheleire --- support/download/hg | 81 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 78 insertions(+), 3 deletions(-) Note: this patch only got limited testing so far and needs to be tested further. But I already wanted to send it out for feedback and for those of you that want to help test it. diff --git a/support/download/hg b/support/download/hg index efb515fca5..bb5cc87969 100755 --- a/support/download/hg +++ b/support/download/hg @@ -1,7 +1,7 @@ #!/usr/bin/env bash # We want to catch any unexpected failure, and exit immediately -set -e +set -E # Download helper for hg, to be called from the download wrapper script # @@ -10,11 +10,39 @@ set -e # -o FILE Generate archive in FILE. # -u URI Clone from repository at URI. # -c CSET Use changeset (or revision) CSET. +# -d DLDIR Download directory path. # -n NAME Use basename NAME. # # Environment: # HG : the hg command to call +# Save our path and options in case we need to call ourselves again +myname="${0}" +declare -a OPTS=("${@}") + +# This function is called when an error occurs. Its job is to attempt a clone +# from scratch (only once!) in case the hg tree is borked, or in case an +# unexpected and unsupported situation arises with uncommitted stuff (e.g. if +# the user manually mucked around in the hg cache). +# Note that this function would also be triggered by connection errors during +# the clone/pull. +_on_error() { + local ret=${?} + + printf "Detected a possibly corrupted hg cache.\n" >&2 + if ${BR_HG_BACKEND_FIRST_FAULT:-false}; then + printf "This is the second time in a row; bailing out\n" >&2 + exit ${ret} + fi + export BR_HG_BACKEND_FIRST_FAULT=true + + printf "Removing it and starting afresh.\n" >&2 + + rm -rf "${hg_cache}" + + exec "${myname}" "${OPTS[@]}" || exit ${ret} +} + verbose= while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do case "${OPT}" in @@ -22,6 +50,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";; + d) dl_dir="${OPTARG}";; n) basename="${OPTARG}";; :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;; \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; @@ -36,8 +65,54 @@ _hg() { eval ${HG} "${@}" } -_hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${basename}'" +# Create and cd into the directory that will contain the local hg cache +hg_cache="${dl_dir}/hg" +mkdir -p "${hg_cache}" + +# Any error now should try to recover +trap _on_error ERR + +# Create a warning file, that the user should not use the hg cache. +# It's ours. Our precious. +cat <<-_EOF_ >"${dl_dir}/hg.readme" + IMPORTANT NOTE! + + The hg tree located in this directory is for the exclusive use + by Buildroot, which uses it as a local cache to reduce bandwidth + usage. + + Buildroot *will* trash any changes in that tree whenever it needs + to use it. Buildroot may even remove it in case it detects the + repository may have been damaged or corrupted. + + Do *not* work in that directory; your changes will eventually get + lost. Do *not* even use it as a remote, or as the source for new + worktrees; your commits will eventually get lost. +_EOF_ + +if ! [ -d "${hg_cache}/.hg" ]; then + # While it would be possible to create an empty repo and use pull + # subsequently, we intentionally use clone the first time as it could be + # faster than pull thanks to clonebundles, if enabled on the server. + printf "Cloning repository from '${uri}' into '${hg_cache}'\n" + _hg clone ${verbose} "${@}" --noupdate "'${uri}'" "'${hg_cache}'" +else + printf "Pulling repository from '${uri}' into '${hg_cache}'\n" + _hg pull ${verbose} "${@}" --repository "'${hg_cache}'" "'${uri}'" +fi + +# Check that the changeset does exist. If it does not, re-cloning from +# scratch won't help, so we don't want to trash the repository for a +# missing commit. We just exit without going through the ERR trap. +if ! _hg identify --repository "'${hg_cache}'" --rev "'${cset}'" >/dev/null 2>&1; then + printf "Commit '%s' does not exist in this repository\n." "${cset}" + exit 1 +fi + +# Make sure that there is no working directory. This will clear any user +# temptation to work in this directory. +_hg update --repository "'${hg_cache}'" null >/dev/null 2>&1 -_hg archive ${verbose} --repository "'${basename}'" --type tgz \ +_hg archive ${verbose} --repository "'${hg_cache}'" --type tgz \ --prefix "'${basename}'" --rev "'${cset}'" \ - >"${output}"