From patchwork Thu Feb 20 16:01:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Havelange X-Patchwork-Id: 1241519 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=essensium.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=essensium.com header.i=@essensium.com header.a=rsa-sha256 header.s=google header.b=RN9VrxfE; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48NfVD2xH7z9sPK for ; Fri, 21 Feb 2020 03:01:56 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 8605E87DC5; Thu, 20 Feb 2020 16:01:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ih2JdPWDlEdO; Thu, 20 Feb 2020 16:01:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 6C3AE87E54; Thu, 20 Feb 2020 16:01:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id D62751BF9B3 for ; Thu, 20 Feb 2020 16:01:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id D1C1285FD4 for ; Thu, 20 Feb 2020 16:01:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JoQfBNjShfgP for ; Thu, 20 Feb 2020 16:01:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by fraxinus.osuosl.org (Postfix) with ESMTPS id CB9EB84D2E for ; Thu, 20 Feb 2020 16:01:43 +0000 (UTC) Received: by mail-wr1-f44.google.com with SMTP id c9so5206946wrw.8 for ; Thu, 20 Feb 2020 08:01:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=essensium.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CYu46s8vFRBgDtpzey+ASxHaya/eJy9CB7ZtqYSUIPI=; b=RN9VrxfEomUaWa0AwfyJ/Qe1LmgoRSwkQfj9gVj1hDjrZV0KmRw1iaGJlQEuQSAtRp PU8N37wMwe76Qvj7zqlNrc42BkG3HaqrFyuh4JyvHwK3cXWGkNL1zed5IMYx8j1eSaLO gxapPw6yTcPQSezOFGAQ9rVZhIC0c1/xvwbS9U7FCwx0nqFfstmSdU9O2Nl0SOLI44xB ENWOHL3OUsfpQRubF/2yfKKHUGILZM9U3TOrGRuQE9622ueAjXE0Xwj+Rm7kcBA091g4 9Grh+mlQLfdTTUljVtFcOY79HYzm74CTJIJT+eoMe8ZwyteeAoY3cKkMrgKBcK9Sdcs3 n/Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CYu46s8vFRBgDtpzey+ASxHaya/eJy9CB7ZtqYSUIPI=; b=W3FWuZPmMab44TWCrs4fZJAfjfcErmfqFeBEwSpDDNOY58Nr0gFFpvfClpHdNybUIm cv1g6S6BE4oH4vtZIo/jjyWjDP9Gl+W41e4dUpCnlosKdn5jbkC6/QgzsEcIBwQ6ExIA 0zPZuhIBS2xEhxpaF8aYSE9bTMaFXoDfYAee+mCfJEu9fDO6pSLEKZ/A2oEIEJQLSSmD QPN2ZjF2gxo1Zt4brY2nocquNyfQOzYmSmhT0eeaFdCrrfcN8mTmhAgDI7wMN2DwQegi KuHm4nI0T1xckOgNcY+DACmQOW/HGnwhI8fLilEFiGmFLvcvHIca65lvAUNYZoYjeO8C zeUA== X-Gm-Message-State: APjAAAXAJbkwDVF4pPk6IrkNuophJMnSOKww5t6JTuq3b1Y9opbL+8gP 4p00uPCxA3iJpnQAnndHoB6d24iwigSzcg== X-Google-Smtp-Source: APXvYqzp6vL7uXKc+z0RSkSWmZEyMCwF7QCT1Ra3q1yn/Qcspr4CA3tYFX33y7fHcWRpkpnm41ky2g== X-Received: by 2002:adf:ea85:: with SMTP id s5mr42221745wrm.75.1582214501582; Thu, 20 Feb 2020 08:01:41 -0800 (PST) Received: from ph-ThinkPad-E560.local.ess-mail.com (ip-188-118-3-185.reverse.destiny.be. [188.118.3.185]) by smtp.gmail.com with ESMTPSA id h5sm5483691wmf.8.2020.02.20.08.01.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2020 08:01:41 -0800 (PST) From: Patrick Havelange To: buildroot@buildroot.org Date: Thu, 20 Feb 2020 17:01:15 +0100 Message-Id: <20200220160119.3407-6-patrick.havelange@essensium.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200220160119.3407-1-patrick.havelange@essensium.com> References: <20200220160119.3407-1-patrick.havelange@essensium.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v3 06/10] package/pkg-cargo.mk: Introduce the cargo dl backend X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire , Patrick Havelange , Sam Voss , "Yann E . MORIN" , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Cargo is now a fully supported PKGMGR, automatically set for any package using the cargo infrastructure. This effectively splits the download phase and the build phase. The cargo backend will set CARGO_HOME inside DL_DIR during download. The cached files in CARGO_HOME permits to download only once the crates index and each dependencies. During download phase, it will call cargo vendor to copy the dependencies inside the VENDOR directory inside the source archive. A local cargo config is also inserted inside the archive in order to use the VENDOR dir during the build phase. The build phase is forced to not query the online repository anymore and thus will be using the vendored dependencies from the tarball. This also permits to have offline builds. Signed-off-by: Patrick Havelange --- package/pkg-cargo.mk | 10 ++++-- package/ripgrep/ripgrep.hash | 2 +- support/download/cargo | 65 ++++++++++++++++++++++++++++++++++++ support/download/dl-wrapper | 2 +- 4 files changed, 75 insertions(+), 4 deletions(-) create mode 100755 support/download/cargo diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk index 35f7c15ad9..c084f7b35e 100644 --- a/package/pkg-cargo.mk +++ b/package/pkg-cargo.mk @@ -39,6 +39,8 @@ define inner-cargo-package # We need host-rustc to run cargo $(2)_DEPENDENCIES += host-rustc +$(2)_PKGMGR = cargo\| + $(2)_CARGO_ENV = \ CARGO_HOME=$(HOST_DIR)/share/cargo \ $(TARGET_CONFIGURE_OPTS) @@ -55,11 +57,13 @@ endif # ifndef $(2)_BUILD_CMDS define $(2)_BUILD_CMDS + cd $$(@D) && \ $(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ cargo build \ --$$($(2)_CARGO_MODE) \ $$($(2)_CARGO_TARGET_OPT) \ - --manifest-path $$(@D)/Cargo.toml + --manifest-path Cargo.toml \ + --offline endef endif @@ -69,12 +73,14 @@ endif # ifndef $(2)_INSTALL_TARGET_CMDS define $(2)_INSTALL_TARGET_CMDS + cd $$(@D) && \ $(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ cargo install \ --root $(TARGET_DIR)/usr/ \ --bins \ - --path $$(@D) \ + --path ./ \ $$($(2)_CARGO_TARGET_OPT) \ + --offline \ --force endef endif diff --git a/package/ripgrep/ripgrep.hash b/package/ripgrep/ripgrep.hash index 0841c0185c..8c48458cd8 100644 --- a/package/ripgrep/ripgrep.hash +++ b/package/ripgrep/ripgrep.hash @@ -1,3 +1,3 @@ # Locally calculated -sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz +sha256 cb895cff182740c219fefbbaaf903e60f005a4ac4688cac1e43e5fbbbccc5a94 ripgrep-0.8.1.tar.gz sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT diff --git a/support/download/cargo b/support/download/cargo new file mode 100755 index 0000000000..0ce94cf16b --- /dev/null +++ b/support/download/cargo @@ -0,0 +1,65 @@ +#!/usr/bin/env bash + +# We want to catch any unexpected failure, and exit immediately +set -e + +# Download helper for cargo package. It will populate the VENDOR directory +# inside the archive with the dependencies of the package +# +# Arguments are in this order: +# $1 mandatory fullpath to an already downloaded source file of a cargo package +# $2 mandatory fullpath to a temp dir +# $3 mandatory fullpath to the DL_DIR +# +# Environment: +# cargo in PATH + +tmpd="" + +do_clean() { + if [ -n "${tmpd}" ]; then + rm -rf "${tmpd}" + fi +} + +trap do_clean EXIT + +if [ $# -le 2 ] ; then + echo 'Need at least 3 arguments' >&2 + exit 1 +fi; + +tmpd="$(mktemp -d -p "$2")" +cd "${tmpd}" + +tar xf "${1}" +cd ./*/ +echo "Running cargo vendor." +CARGO_HOME="${3}"/cargo_home cargo vendor -q --locked VENDOR +# Create the local .cargo/config with vendor info +mkdir -p .cargo/ +cat <.cargo/config +[source.crates-io] +replace-with = "vendored-sources" + +[source.vendored-sources] +directory = "VENDOR" +EOF + +cd .. + +# Generate the archive, sort with the C locale so that it is reproducible. +find "$(basename "$OLDPWD")" -not -type d -print0 >files.list +LC_ALL=C sort -z files.list.sorted +# let's use a fixed hardcoded date to be reproducible +date="2020-02-06 01:02:03" + +# Create GNU-format tarballs, since that's the format of the tarballs on +# sources.buildroot.org and used in the *.hash files +echo "Creating final archive." +tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \ + --owner=0 --group=0 --mtime="${date}" -T files.list.sorted +gzip -6 -n new.tar.gz +mv "${1}" "${1}".old +mv new.tar.gz "${1}" +rm "${1}".old diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 3f613bb622..5e52b3e60f 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -98,7 +98,7 @@ main() { case "${b}" in urlencode) urlencode="${b}" ;; git|svn|cvs|bzr|file|scp|hg) backend="${b}" ;; - # insert here supported second backends) backend2="${b}" ;; + cargo) backend2="${b}" ;; esac done uri=${uri#*+}