From patchwork Thu Nov 19 21:36:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403289 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1C6WVsz9sVH for ; Fri, 20 Nov 2020 08:37:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 3F83E86CE0; Thu, 19 Nov 2020 21:37:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Xk1Njt99ZDHf; Thu, 19 Nov 2020 21:37:16 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 12ADE86D23; Thu, 19 Nov 2020 21:37:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 1A0871BF32B for ; Thu, 19 Nov 2020 21:37:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1761386C4F for ; Thu, 19 Nov 2020 21:37:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id U1iIPD1IhSfc for ; Thu, 19 Nov 2020 21:37:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 25A1B86C5A for ; Thu, 19 Nov 2020 21:37:09 +0000 (UTC) X-Originating-IP: 86.201.80.25 Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 48AF31BF20A; Thu, 19 Nov 2020 21:37:05 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:46 +0100 Message-Id: <20201119213658.1232531-2-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 01/12] support/download/dl-wrapper: add concept of download post-processing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In order to support package managers such as Cargo (Rust) or Go, we want to run some custom logic after the main download, but before packing the tarball and checking the hash. To implement this, this commit introduces a concept of download post-processing: if -p is passed to the dl-wrapper, then support/download/-post-process will be called. Signed-off-by: Thomas Petazzoni --- support/download/dl-wrapper | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 3315bd410e..2d74554213 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -25,7 +25,7 @@ main() { local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";; @@ -37,6 +37,7 @@ main() { r) recurse="-r";; f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; + p) post_process="${OPTARG}";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; @@ -135,6 +136,12 @@ main() { continue fi + if [ -n "${post_process}" ] ; then + ${OLDPWD}/support/download/${post_process}-post-process \ + -o "${tmpf}" \ + -n "${raw_base_name}" + fi + # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}" From patchwork Thu Nov 19 21:36:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403287 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY190z7Gz9sTR for ; Fri, 20 Nov 2020 08:37:17 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 852C9874BB; Thu, 19 Nov 2020 21:37:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uCoZzseHvHva; Thu, 19 Nov 2020 21:37:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id DDEB2874BC; Thu, 19 Nov 2020 21:37:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 0DB5C1BF32B for ; Thu, 19 Nov 2020 21:37:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0734486C5D for ; Thu, 19 Nov 2020 21:37:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mfBUb-yNCASi for ; Thu, 19 Nov 2020 21:37:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by fraxinus.osuosl.org (Postfix) with ESMTPS id F3C4286C4F for ; Thu, 19 Nov 2020 21:37:09 +0000 (UTC) X-Originating-IP: 86.201.80.25 Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id BC19DFF808; Thu, 19 Nov 2020 21:37:07 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:47 +0100 Message-Id: <20201119213658.1232531-3-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 02/12] package/pkg-download.mk: add _DOWNLOAD_POST_PROCESS variable X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This will allow packages to register than a download post-processing is needed. Note that this variable is intentionally not documented: it is an internal variable meant to be set by package infrastructures, not directly by packages. Signed-off-by: Thomas Petazzoni --- package/pkg-download.mk | 1 + 1 file changed, 1 insertion(+) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 951d2fb554..175b9d7dba 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \ $(QUIET) \ From patchwork Thu Nov 19 21:36:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403290 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1K3JzRz9sTR for ; Fri, 20 Nov 2020 08:37:25 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 4528727E4C; Thu, 19 Nov 2020 21:37:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lxzTgRbW4NhY; Thu, 19 Nov 2020 21:37:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 38E622754C; Thu, 19 Nov 2020 21:37:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 9D2A71BF32B for ; Thu, 19 Nov 2020 21:37:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 95A22874C1 for ; Thu, 19 Nov 2020 21:37:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uQplnjqccoZI for ; Thu, 19 Nov 2020 21:37:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by hemlock.osuosl.org (Postfix) with ESMTPS id 69293874B2 for ; Thu, 19 Nov 2020 21:37:13 +0000 (UTC) Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay11.mail.gandi.net (Postfix) with ESMTPSA id 3D9D3100003; Thu, 19 Nov 2020 21:37:08 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:48 +0100 Message-Id: <20201119213658.1232531-4-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 03/12] package/pkg-download.mk: add _DL_ENV variable X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This variable can be used by package to pass extra environment variables to the download logic. It will be used for the Go/Cargo vendoring. The _DL_ENV variable is intentionally not documented: at this point, it is not meant to be used by packages directly, but only by package infrastructures. Suggested-by: Yann E. MORIN Signed-off-by: Thomas Petazzoni --- package/pkg-download.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 175b9d7dba..c914d016e2 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -99,7 +99,9 @@ endif define DOWNLOAD $(Q)mkdir -p $($(2)_DL_DIR) - $(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ + $(Q)$(EXTRA_ENV) \ + $($(2)_DL_ENV) \ + flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ -c '$($(2)_DL_VERSION)' \ -d '$($(2)_DL_DIR)' \ -D '$(DL_DIR)' \ From patchwork Thu Nov 19 21:36:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403294 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1V4cfYz9sVJ for ; Fri, 20 Nov 2020 08:37:34 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 117EC86C63; Thu, 19 Nov 2020 21:37:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rPfwU9IOpA0l; Thu, 19 Nov 2020 21:37:30 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 45FB486C71; Thu, 19 Nov 2020 21:37:29 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id AFD8C1BF32B for ; Thu, 19 Nov 2020 21:37:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9CC4127E4C for ; Thu, 19 Nov 2020 21:37:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fa6LTjIgi8oJ for ; Thu, 19 Nov 2020 21:37:17 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by silver.osuosl.org (Postfix) with ESMTPS id 57C08275A5 for ; Thu, 19 Nov 2020 21:37:15 +0000 (UTC) Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 1B5BF200009; Thu, 19 Nov 2020 21:37:10 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:49 +0100 Message-Id: <20201119213658.1232531-5-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 04/12] support/download/post-process-helpers: add helper function for post process scripts X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function. Signed-off-by: Thomas Petazzoni --- support/download/post-process-helpers | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 support/download/post-process-helpers diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers new file mode 100644 index 0000000000..bed8df2577 --- /dev/null +++ b/support/download/post-process-helpers @@ -0,0 +1,30 @@ + +unpack() { + dest="$1" + tarball="$2" + + mkdir ${dest} + tar -C ${dest} --strip-components=1 -xf ${tarball} +} + +repack() { + src="$1" + tarball="$2" + + # Generate the archive, sort with the C locale so that it is reproducible. + find "$(basename ${src})" -not -type d -print0 >files.list + LC_ALL=C sort -z files.list.sorted + + # let's use a fixed hardcoded date to be reproducible + date="2020-02-06 01:02:03 +0000" + + # Create GNU-format tarballs, since that's the format of the tarballs on + # sources.buildroot.org and used in the *.hash files + tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \ + --owner=0 --group=0 --mtime="${date}" -T files.list.sorted + gzip -6 -n new.tar.gz + mv "${tarball}" "${tarball}".old + mv new.tar.gz "${tarball}" + rm "${tarball}".old + rm -rf ${src} +} From patchwork Thu Nov 19 21:36:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403291 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1R6vKpz9sTR for ; Fri, 20 Nov 2020 08:37:31 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 3A03327A49; Thu, 19 Nov 2020 21:37:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sUj+XcNd9oRo; Thu, 19 Nov 2020 21:37:25 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id E376127E5E; Thu, 19 Nov 2020 21:37:22 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 022351BF32B for ; Thu, 19 Nov 2020 21:37:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id F29BB86D08 for ; Thu, 19 Nov 2020 21:37:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yQlifnIBLZXC for ; Thu, 19 Nov 2020 21:37:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay11.mail.gandi.net (relay11.mail.gandi.net [217.70.178.231]) by whitealder.osuosl.org (Postfix) with ESMTPS id B565E86CE0 for ; Thu, 19 Nov 2020 21:37:14 +0000 (UTC) Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay11.mail.gandi.net (Postfix) with ESMTPSA id 9833E100006; Thu, 19 Nov 2020 21:37:12 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:50 +0100 Message-Id: <20201119213658.1232531-6-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 05/12] support/download/go-post-process: implement Go vendoring support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anisse Astier , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure. Signed-off-by: Thomas Petazzoni --- package/pkg-golang.mk | 7 ++++++- support/download/go-post-process | 29 +++++++++++++++++++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) create mode 100755 support/download/go-post-process diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk index 3813e1c406..150445bf17 100644 --- a/package/pkg-golang.mk +++ b/package/pkg-golang.mk @@ -47,7 +47,7 @@ $(2)_BUILD_OPTS += \ -p $(PARALLEL_JOBS) # Target packages need the Go compiler on the host. -$(2)_DEPENDENCIES += host-go +$(2)_DOWNLOAD_DEPENDENCIES += host-go $(2)_BUILD_TARGETS ?= . @@ -72,6 +72,11 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) # If the go.mod file does not exist, one is written with this root path. $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) +$(2)_DOWNLOAD_POST_PROCESS = go +$(2)_DL_ENV = \ + $(HOST_GO_COMMON_ENV) \ + GOPROXY=direct + # Generate a go.mod file if it doesn't exist. Note: Go is configured # to use the "vendor" dir and not make network calls. define $(2)_GEN_GOMOD diff --git a/support/download/go-post-process b/support/download/go-post-process new file mode 100755 index 0000000000..01073aee8b --- /dev/null +++ b/support/download/go-post-process @@ -0,0 +1,29 @@ +#!/usr/bin/env bash + +set -e + +. $(dirname $0)/post-process-helpers + +# Parse our options +while getopts "n:o:" OPT; do + case "${OPT}" in + o) output="${OPTARG}";; + n) base_name="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac +done + +# Already vendored tarball, nothing to do +if tar tf ${output} | grep -q "^[^/]*/vendor" ; then + exit 0 +fi + +unpack ${base_name} ${output} + +# Do the Go vendoring +pushd ${base_name} > /dev/null +go mod vendor -v +popd > /dev/null + +repack ${base_name} ${output} From patchwork Thu Nov 19 21:36:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403292 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1S12rjz9sVH for ; Fri, 20 Nov 2020 08:37:31 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id DB2EA86C4F; Thu, 19 Nov 2020 21:37:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1CRYDQ_Zn3vr; Thu, 19 Nov 2020 21:37:28 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4608986C5A; Thu, 19 Nov 2020 21:37:28 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 607A01BF32B for ; Thu, 19 Nov 2020 21:37:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5D79586C5A for ; Thu, 19 Nov 2020 21:37:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Do0BU_WWa9ZO for ; Thu, 19 Nov 2020 21:37:16 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 3C9D586C4F for ; Thu, 19 Nov 2020 21:37:16 +0000 (UTC) Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id E069A200004; Thu, 19 Nov 2020 21:37:13 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:51 +0100 Message-Id: <20201119213658.1232531-7-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 06/12] package/tinifier: new package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This is a Go package that needs vendor modules to be downloaded at build time. Signed-off-by: Thomas Petazzoni --- DEVELOPERS | 1 + package/Config.in | 1 + package/tinifier/Config.in | 10 ++++++++++ package/tinifier/tinifier.hash | 3 +++ package/tinifier/tinifier.mk | 13 +++++++++++++ 5 files changed, 28 insertions(+) create mode 100644 package/tinifier/Config.in create mode 100644 package/tinifier/tinifier.hash create mode 100644 package/tinifier/tinifier.mk diff --git a/DEVELOPERS b/DEVELOPERS index d0c9c16423..eb97cfeac0 100644 --- a/DEVELOPERS +++ b/DEVELOPERS @@ -2590,6 +2590,7 @@ F: package/rtc-tools/ F: package/sam-ba/ F: package/scons/ F: package/squashfs/ +F: package/tinifier/ F: package/wayland/ F: package/weston/ F: support/testing/tests/boot/test_syslinux.py diff --git a/package/Config.in b/package/Config.in index d32a271113..a93959ada8 100644 --- a/package/Config.in +++ b/package/Config.in @@ -298,6 +298,7 @@ comment "Graphic applications" source "package/rrdtool/Config.in" source "package/stellarium/Config.in" source "package/tesseract-ocr/Config.in" + source "package/tinifier/Config.in" comment "Graphic libraries" source "package/cegui/Config.in" diff --git a/package/tinifier/Config.in b/package/tinifier/Config.in new file mode 100644 index 0000000000..fbadfe6bd9 --- /dev/null +++ b/package/tinifier/Config.in @@ -0,0 +1,10 @@ +config BR2_PACKAGE_TINIFIER + bool "tinifier" + depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS + help + CLI tool for images compressing + + This tool uses tinypng.com API endpoint for compressing your + local jpg/png images (it supports parallel jobs). + + https://github.com/tarampampam/tinifier diff --git a/package/tinifier/tinifier.hash b/package/tinifier/tinifier.hash new file mode 100644 index 0000000000..146700817b --- /dev/null +++ b/package/tinifier/tinifier.hash @@ -0,0 +1,3 @@ +# Locally calculated +sha256 707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c tinifier-2.1.0.tar.gz +sha256 791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6 LICENSE diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk new file mode 100644 index 0000000000..b47d265a8e --- /dev/null +++ b/package/tinifier/tinifier.mk @@ -0,0 +1,13 @@ +################################################################################ +# +# tinifier +# +################################################################################ + +TINIFIER_VERSION = 2.1.0 +TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION)) +TINIFIER_LICENSE = MIT +TINIFIER_LICENSE_FILES = LICENSE +TINIFIER_GOMOD = tinifier + +$(eval $(golang-package)) From patchwork Thu Nov 19 21:36:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403296 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1f1cvsz9sTR for ; Fri, 20 Nov 2020 08:37:42 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id B170886F99; Thu, 19 Nov 2020 21:37:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WjPJ7uxczBVG; Thu, 19 Nov 2020 21:37:34 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 855BE86EC6; Thu, 19 Nov 2020 21:37:32 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id B38F61BF32B for ; Thu, 19 Nov 2020 21:37:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id AE87286EC6 for ; Thu, 19 Nov 2020 21:37:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LUY+tyMRKP8t for ; Thu, 19 Nov 2020 21:37:20 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by whitealder.osuosl.org (Postfix) with ESMTPS id 84BCF86D44 for ; Thu, 19 Nov 2020 21:37:19 +0000 (UTC) X-Originating-IP: 86.201.80.25 Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 49912E000C; Thu, 19 Nov 2020 21:37:15 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:52 +0100 Message-Id: <20201119213658.1232531-8-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Patrick Havelange In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo. Signed-off-by: Patrick Havelange [Thomas: add support for host-cargo-package] Signed-off-by: Thomas Petazzoni --- package/Makefile.in | 1 + package/pkg-cargo.mk | 123 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 package/pkg-cargo.mk diff --git a/package/Makefile.in b/package/Makefile.in index 51f5cbce4f..2af123f36d 100644 --- a/package/Makefile.in +++ b/package/Makefile.in @@ -427,3 +427,4 @@ include package/pkg-waf.mk include package/pkg-golang.mk include package/pkg-meson.mk include package/pkg-qmake.mk +include package/pkg-cargo.mk diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk new file mode 100644 index 0000000000..52c237aa3c --- /dev/null +++ b/package/pkg-cargo.mk @@ -0,0 +1,123 @@ +################################################################################ +# Cargo package infrastructure +# +# This file implements an infrastructure that eases development of package +# .mk files for Cargo packages. It should be used for all packages that use +# Cargo as their build system. +# +# See the Buildroot documentation for details on the usage of this +# infrastructure +# +# In terms of implementation, this Cargo infrastructure requires the .mk file +# to only specify metadata information about the package: name, version, +# download URL, etc. +# +# We still allow the package .mk file to override what the different steps +# are doing, if needed. For example, if _BUILD_CMDS is already defined, +# it is used as the list of commands to perform to build the package, +# instead of the default Cargo behaviour. The package can also define some +# post operation hooks. +# +################################################################################ + +################################################################################ +# inner-cargo-package -- defines how the configuration, compilation and +# installation of a cargo package should be done, implements a few hooks +# to tune the build process for cargo specifities and calls the generic +# package infrastructure to generate the necessary make targets +# +# argument 1 is the lowercase package name +# argument 2 is the uppercase package name, including a HOST_ prefix +# for host packages +# argument 3 is the uppercase package name, without the HOST_ prefix +# for host packages +# argument 4 is the type (target or host) +################################################################################ + +define inner-cargo-package + +# We need host-rustc to run cargo +$(2)_DEPENDENCIES += host-rustc + +$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo + +# +# Build step. Only define it if not already defined by the package .mk +# file. +# +ifndef $(2)_BUILD_CMDS +ifeq ($(4),target) +define $(2)_BUILD_CMDS + $$(TARGET_MAKE_ENV) \ + $$(TARGET_CONFIGURE_OPTS) \ + $$($(2)_CARGO_ENV) \ + cargo build \ + --offline \ + --target $$(RUSTC_TARGET_NAME) \ + $$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \ + --manifest-path $$(@D)/Cargo.toml \ + --locked \ + $$($(2)_CARGO_BUILD_OPTS) +endef +else # ifeq ($(4),target) +define $(2)_BUILD_CMDS + $$(HOST_MAKE_ENV) \ + RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \ + $$($(2)_CARGO_ENV) \ + cargo build \ + --offline \ + --release \ + --manifest-path $$(@D)/Cargo.toml \ + --locked \ + $$($(2)_CARGO_BUILD_OPTS) +endef +endif # ifeq ($(4),target) +endif # ifndef $(2)_BUILD_CMDS + +# +# Target installation step. Only define it if not already defined by +# the package .mk file. +# +ifndef $(2)_INSTALL_TARGET_CMDS +define $(2)_INSTALL_TARGET_CMDS + $$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ + cargo install \ + --target $$(RUSTC_TARGET_NAME) \ + --offline \ + --root $$(TARGET_DIR)/usr/ \ + --bins \ + --path $$(@D) \ + --force \ + --locked \ + $$($(2)_CARGO_INSTALL_OPTS) +endef +endif + +ifndef $(2)_INSTALL_CMDS +define $(2)_INSTALL_CMDS + $$(HOST_MAKE_ENV) \ + RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \ + $$($(2)_CARGO_ENV) \ + cargo install \ + --offline \ + --root $$(HOST_DIR) \ + --bins \ + --path $$(@D) \ + --force \ + --locked \ + $$($(2)_CARGO_INSTALL_OPTS) +endef +endif + +# Call the generic package infrastructure to generate the necessary +# make targets +$(call inner-generic-package,$(1),$(2),$(3),$(4)) + +endef + +################################################################################ +# cargo-package -- the target generator macro for Cargo packages +################################################################################ + +cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target) +host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host) From patchwork Thu Nov 19 21:36:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403293 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1V2Z27z9sVH for ; Fri, 20 Nov 2020 08:37:34 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id CDE0786EE2; Thu, 19 Nov 2020 21:37:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O51Bj8vR4SXM; Thu, 19 Nov 2020 21:37:31 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 76AB986D26; Thu, 19 Nov 2020 21:37:31 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 402451BF32B for ; Thu, 19 Nov 2020 21:37:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 3733386EC6 for ; Thu, 19 Nov 2020 21:37:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WkJtRIvO5Hy2 for ; Thu, 19 Nov 2020 21:37:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by whitealder.osuosl.org (Postfix) with ESMTPS id 253DC86D14 for ; Thu, 19 Nov 2020 21:37:18 +0000 (UTC) Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id C9CC6200005; Thu, 19 Nov 2020 21:37:16 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:53 +0100 Message-Id: <20201119213658.1232531-9-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 08/12] docs/manual/cargo: document the cargo-package infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Patrick Havelange The Buildroot manual was already providing some details on how to integrate Cargo packages, and those details now need to be updated with a proper documentation for the cargo-package infrastructure. Signed-off-by: Patrick Havelange [Thomas: numerous updates and extensions.] Signed-off-by: Thomas Petazzoni --- docs/manual/adding-packages-cargo.txt | 88 ++++++++++++--------------- 1 file changed, 39 insertions(+), 49 deletions(-) diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt index 8fcc80bcc6..c65a32f017 100644 --- a/docs/manual/adding-packages-cargo.txt +++ b/docs/manual/adding-packages-cargo.txt @@ -1,7 +1,7 @@ // -*- mode:doc; -*- // vim: set syntax=asciidoc: -=== Integration of Cargo-based packages +=== Infrastructure for Cargo-based packages Cargo is the package manager for the Rust programming language. It allows the user to build programs or libraries written in Rust, but it also downloads and @@ -10,7 +10,7 @@ called "crates". [[cargo-package-tutorial]] -==== Cargo-based package's +Config.in+ file +==== +cargo-package+ tutorial The +Config.in+ file of Cargo-based package 'foo' should contain: @@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain: 08: http://foosoftware.org/foo/ --------------------------- -==== Cargo-based package's +.mk+ file - -Buildroot does not (yet) provide a dedicated package infrastructure for -Cargo-based packages. So, we will explain how to write a +.mk+ file for such a -package. Let's start with an example: +And the +.mk+ file for this package should contain: ------------------------------ 01: ################################################################################ @@ -44,52 +40,48 @@ package. Let's start with an example: 10: FOO_LICENSE = GPL-3.0+ 11: FOO_LICENSE_FILES = COPYING 12: -13: FOO_DEPENDENCIES = host-rustc -14: -15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo -16: -17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE) -18: -19: FOO_CARGO_OPTS = \ -20: $(if $(BR2_ENABLE_DEBUG),,--release) \ -21: --target=$(RUSTC_TARGET_NAME) \ -22: --manifest-path=$(@D)/Cargo.toml -23: -24: define FOO_BUILD_CMDS -25: $(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \ -26: cargo build $(FOO_CARGO_OPTS) -27: endef -28: -29: define FOO_INSTALL_TARGET_CMDS -30: $(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \ -31: $(TARGET_DIR)/usr/bin/foo -32: endef -33: -34: $(eval $(generic-package)) +13: $(eval $(cargo-package)) -------------------------------- -The Makefile starts with the definition of the standard variables for package -declaration (lines 7 to 11). +The Makefile starts with the definition of the standard variables for +package declaration (lines 7 to 11). + +As seen in line 13, it is based on the +cargo-package+ +infrastructure. Cargo will be invoked automatically by this +infrastructure to build and install the package. + +It is still possible to define custom build commands or install +commands (i.e. with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS). +Those will then replace the commands from the cargo infrastructure. + +==== +cargo-package+ reference + +The main macros for the Cargo package infrastructure are ++cargo-package+ for target packages and +host-cargo-package+ for host +packages. + +Just like the generic infrastructure, the Cargo infrastructure works +by defining a number of variables before calling the +cargo-package+ +or +host-cargo-package+ macros. -As seen in line 34, it is based on the -xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines -the variables required by this particular infrastructure, where Cargo is -invoked: +First, all the package metadata information variables that exist in +the generic infrastructure also exist in the Cargo infrastructure: ++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+, ++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc. -* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required - to configure the cross-compilation of the package are passed via - +FOO_CONF_OPTS+. +A few additional variables, specific to the Cargo infrastructure, can +also be defined. Many of them are only useful in very specific cases, +typical packages will therefore only use a few of them. -* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on - the target. +* +FOO_CARGO_ENV+ can be used to pass additional variables in the + environment of +cargo+ invocations. It used at both build and + installation time -In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to -contain +host-cargo+. +* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to + +cargo+ at build time. -To sum it up, to add a new Cargo-based package, the Makefile example can be -copied verbatim then edited to replace all occurences of +FOO+ with the -uppercase name of the new package and update the values of the standard -variables. +* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to + +cargo+ at install time. ==== About Dependencies Management @@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the +cargo fetch+ command. Cargo maintains a local cache of the registry index and of git checkouts of the -crates, whose location is given by +$CARGO_HOME+. As seen in the package -Makefile example at line 15, this environment variable is set to -+$(HOST_DIR)/share/cargo+. +crates, whose location is given by +$CARGO_HOME+. This dependency download mechanism is not convenient when performing an offline build, as Cargo will fail to fetch the dependencies. In that case, it is advised From patchwork Thu Nov 19 21:36:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403299 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY2p08kPz9sTR for ; Fri, 20 Nov 2020 08:38:42 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 24F91275B0; Thu, 19 Nov 2020 21:38:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id j9TDV3KFklHD; Thu, 19 Nov 2020 21:38:35 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 52BA52D059; Thu, 19 Nov 2020 21:37:36 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id BF7921BF32B for ; Thu, 19 Nov 2020 21:37:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 9C4242D3FC for ; Thu, 19 Nov 2020 21:37:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eivKubOc56XF for ; Thu, 19 Nov 2020 21:37:20 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay4-d.mail.gandi.net (relay4-d.mail.gandi.net [217.70.183.196]) by silver.osuosl.org (Postfix) with ESMTPS id 6520E27A6E for ; Thu, 19 Nov 2020 21:37:20 +0000 (UTC) X-Originating-IP: 86.201.80.25 Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay4-d.mail.gandi.net (Postfix) with ESMTPSA id 2A3F7E000B; Thu, 19 Nov 2020 21:37:18 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:54 +0100 Message-Id: <20201119213658.1232531-10-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 09/12] package/ripgrep: convert to cargo infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Patrick Havelange Signed-off-by: Patrick Havelange Signed-off-by: Thomas Petazzoni --- package/ripgrep/ripgrep.mk | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk index 9dd8d58de1..97e9e2ce5f 100644 --- a/package/ripgrep/ripgrep.mk +++ b/package/ripgrep/ripgrep.mk @@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT -RIPGREP_DEPENDENCIES = host-rustc -RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo - -RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR) - -RIPGREP_CARGO_OPTS = \ - --target=$(RUSTC_TARGET_NAME) \ - --manifest-path=$(@D)/Cargo.toml - -ifeq ($(BR2_ENABLE_DEBUG),y) -RIPGREP_CARGO_BIN_SUBDIR = debug -else -RIPGREP_CARGO_OPTS += --release -RIPGREP_CARGO_BIN_SUBDIR = release -endif - -define RIPGREP_BUILD_CMDS - $(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \ - cargo build $(RIPGREP_CARGO_OPTS) -endef - -define RIPGREP_INSTALL_TARGET_CMDS - $(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \ - $(TARGET_DIR)/usr/bin/rg -endef - -$(eval $(generic-package)) +$(eval $(cargo-package)) From patchwork Thu Nov 19 21:36:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403295 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1Z5tVbz9sTR for ; Fri, 20 Nov 2020 08:37:38 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4324D86C71; Thu, 19 Nov 2020 21:37:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id COifTc-6qHvT; Thu, 19 Nov 2020 21:37:33 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id AA20686C73; Thu, 19 Nov 2020 21:37:33 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 43EFD1BF32B for ; Thu, 19 Nov 2020 21:37:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 3FE0486D44 for ; Thu, 19 Nov 2020 21:37:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gYU3i7Ic6Ssf for ; Thu, 19 Nov 2020 21:37:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by whitealder.osuosl.org (Postfix) with ESMTPS id 6E95D86D14 for ; Thu, 19 Nov 2020 21:37:21 +0000 (UTC) Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 820BF200004; Thu, 19 Nov 2020 21:37:19 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:55 +0100 Message-Id: <20201119213658.1232531-11-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 10/12] package/sentry-cli: convert to host-cargo-package infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Joseph Kogut , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Thomas Petazzoni --- package/sentry-cli/sentry-cli.mk | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk index 7e53f16026..40ca22fdf9 100644 --- a/package/sentry-cli/sentry-cli.mk +++ b/package/sentry-cli/sentry-cli.mk @@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION)) SENTRY_CLI_LICENSE = BSD-3-clause SENTRY_CLI_LICENSE_FILES = LICENSE -HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib +HOST_SENTRY_CLI_DEPENDENCIES = host-zlib -HOST_SENTRY_CLI_CARGO_ENV = \ - CARGO_HOME=$(HOST_DIR)/share/cargo \ - RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))" - -HOST_SENTRY_CLI_CARGO_OPTS = \ - --release \ - --manifest-path=$(@D)/Cargo.toml - -define HOST_SENTRY_CLI_BUILD_CMDS - $(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \ - cargo build $(HOST_SENTRY_CLI_CARGO_OPTS) -endef - -define HOST_SENTRY_CLI_INSTALL_CMDS - $(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \ - $(HOST_DIR)/bin/sentry-cli -endef - -$(eval $(host-generic-package)) +$(eval $(host-cargo-package)) From patchwork Thu Nov 19 21:36:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403298 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1m11Xxz9sTR for ; Fri, 20 Nov 2020 08:37:48 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id A3CF186D14; Thu, 19 Nov 2020 21:37:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id iPkhvw+aPPTz; Thu, 19 Nov 2020 21:37:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id ADCB886CE0; Thu, 19 Nov 2020 21:37:38 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 696E01BF32B for ; Thu, 19 Nov 2020 21:37:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6616F86D14 for ; Thu, 19 Nov 2020 21:37:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DjJLKMZd+5iM for ; Thu, 19 Nov 2020 21:37:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay1-d.mail.gandi.net (relay1-d.mail.gandi.net [217.70.183.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id 0185F86D26 for ; Thu, 19 Nov 2020 21:37:24 +0000 (UTC) X-Originating-IP: 86.201.80.25 Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay1-d.mail.gandi.net (Postfix) with ESMTPSA id EB78D240002; Thu, 19 Nov 2020 21:37:20 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:56 +0100 Message-Id: <20201119213658.1232531-12-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 11/12] support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Joseph Kogut , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds support/download/cargo-post-process to perform the vendoring on Cargo packages, and enables it in package/pkg-cargo.mk. Since it changes the contents of the tarballs for ripgrep and sentry-cli, it changes their hashes. To not have a different hash for the same version of ripgrep and sentry-cli, we bump their versions. It has to be done in the same commit as the Cargo vendoring to make the series bisectable. Signed-off-by: Thomas Petazzoni --- package/pkg-cargo.mk | 5 +++- package/ripgrep/ripgrep.hash | 2 +- package/ripgrep/ripgrep.mk | 2 +- package/sentry-cli/sentry-cli.hash | 2 +- package/sentry-cli/sentry-cli.mk | 2 +- support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++ 6 files changed, 46 insertions(+), 5 deletions(-) create mode 100755 support/download/cargo-post-process diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk index 52c237aa3c..6653f64594 100644 --- a/package/pkg-cargo.mk +++ b/package/pkg-cargo.mk @@ -37,10 +37,13 @@ define inner-cargo-package # We need host-rustc to run cargo -$(2)_DEPENDENCIES += host-rustc +$(2)_DOWNLOAD_DEPENDENCIES += host-rustc $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo +$(2)_DOWNLOAD_POST_PROCESS = cargo +$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo + # # Build step. Only define it if not already defined by the package .mk # file. diff --git a/package/ripgrep/ripgrep.hash b/package/ripgrep/ripgrep.hash index 0841c0185c..81ac905d3d 100644 --- a/package/ripgrep/ripgrep.hash +++ b/package/ripgrep/ripgrep.hash @@ -1,3 +1,3 @@ # Locally calculated -sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz +sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk index 97e9e2ce5f..d21f88a6e8 100644 --- a/package/ripgrep/ripgrep.mk +++ b/package/ripgrep/ripgrep.mk @@ -4,7 +4,7 @@ # ################################################################################ -RIPGREP_VERSION = 0.8.1 +RIPGREP_VERSION = 12.1.1 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT diff --git a/package/sentry-cli/sentry-cli.hash b/package/sentry-cli/sentry-cli.hash index 3b0733a276..63b0c812bc 100644 --- a/package/sentry-cli/sentry-cli.hash +++ b/package/sentry-cli/sentry-cli.hash @@ -1,3 +1,3 @@ # locally calculated -sha256 5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd sentry-cli-1.57.0.tar.gz +sha256 a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4 sentry-cli-1.59.0.tar.gz sha256 9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14 LICENSE diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk index 40ca22fdf9..58f5f1e325 100644 --- a/package/sentry-cli/sentry-cli.mk +++ b/package/sentry-cli/sentry-cli.mk @@ -4,7 +4,7 @@ # ################################################################################ -SENTRY_CLI_VERSION = 1.57.0 +SENTRY_CLI_VERSION = 1.59.0 SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION)) SENTRY_CLI_LICENSE = BSD-3-clause SENTRY_CLI_LICENSE_FILES = LICENSE diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process new file mode 100755 index 0000000000..5081476385 --- /dev/null +++ b/support/download/cargo-post-process @@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +set -e + +. $(dirname $0)/post-process-helpers + +while getopts "n:o:" OPT; do + case "${OPT}" in + o) output="${OPTARG}";; + n) base_name="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac +done + +# Already vendored tarball, nothing to do +if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then + exit 0 +fi + +unpack ${base_name} ${output} + +# Do the Cargo vendoring +pushd ${base_name} > /dev/null +cargo vendor --locked VENDOR +echo $? +# Create the local .cargo/config with vendor info +mkdir -p .cargo/ +cat <.cargo/config +[source.crates-io] +replace-with = "vendored-sources" + +[source.vendored-sources] +directory = "VENDOR" +EOF +popd > /dev/null + +repack ${base_name} ${output} From patchwork Thu Nov 19 21:36:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1403297 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CcY1f2XnHz9sVH for ; Fri, 20 Nov 2020 08:37:42 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4084486C7A; Thu, 19 Nov 2020 21:37:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id MMQ_C4-sNXQb; Thu, 19 Nov 2020 21:37:37 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9026886CD7; Thu, 19 Nov 2020 21:37:37 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 644B51BF32B for ; Thu, 19 Nov 2020 21:37:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 6063786D54 for ; Thu, 19 Nov 2020 21:37:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E7rDI21JXGH0 for ; Thu, 19 Nov 2020 21:37:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by whitealder.osuosl.org (Postfix) with ESMTPS id B1D3986D14 for ; Thu, 19 Nov 2020 21:37:26 +0000 (UTC) Received: from localhost (lfbn-tou-1-236-25.w86-201.abo.wanadoo.fr [86.201.80.25]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 74B0E240005; Thu, 19 Nov 2020 21:37:22 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List , "Yann E. MORIN" , "Arnout Vandecappelle (Essensium/Mind)" , Christian Stewart , Patrick Havelange , Matt Weber , Sam Voss Date: Thu, 19 Nov 2020 22:36:57 +0100 Message-Id: <20201119213658.1232531-13-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> References: <20201119213658.1232531-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH next 12/12] docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire , Thomas Petazzoni Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Now that we have vendoring support for Cargo packages, let's rewrite the dependency management section in a more accurate way. We drop the part about the local cache of the registry, because +CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is not shared between builds nor preserved accross builds, so its effect as a cache is limited. Signed-off-by: Thomas Petazzoni --- docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt index c65a32f017..fb3e7d0780 100644 --- a/docs/manual/adding-packages-cargo.txt +++ b/docs/manual/adding-packages-cargo.txt @@ -85,15 +85,13 @@ typical packages will therefore only use a few of them. ==== About Dependencies Management -A crate can depend on other libraries from crates.io or git repositories, listed -in its Cargo.toml file. Before starting a build, Cargo usually downloads -automatically them. This step can also be performed independently, via the -+cargo fetch+ command. - -Cargo maintains a local cache of the registry index and of git checkouts of the -crates, whose location is given by +$CARGO_HOME+. - -This dependency download mechanism is not convenient when performing an offline -build, as Cargo will fail to fetch the dependencies. In that case, it is advised -to generate a tarball of the dependencies using the +cargo vendor+ and add it to -+FOO_EXTRA_DOWNLOADS+. +A crate can depend on other libraries from crates.io or git +repositories, listed in its Cargo.toml file. Buildroot automatically +takes care of downloading such dependencies as part of the download +step of packages that use the +cargo-package+ infrastructure. Such +dependencies are then kept together with the package source code in +the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of +the package's tarball includes such dependencies. + +This mechanism ensures that any change in the dependencies will be +detected, and allows the build to be performed completely offline.