From patchwork Sat Dec 19 15:35:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418719 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZB74FTz9sVM for ; Sun, 20 Dec 2020 02:35:46 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1F192876D6; Sat, 19 Dec 2020 15:35:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TaluI9rEn4X6; Sat, 19 Dec 2020 15:35:40 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 6626B87649; Sat, 19 Dec 2020 15:35:40 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 5AF851BF94D for ; Sat, 19 Dec 2020 15:35:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 4F1FB203BF for ; Sat, 19 Dec 2020 15:35:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NtSFzZyqJ5Ow for ; Sat, 19 Dec 2020 15:35:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by silver.osuosl.org (Postfix) with ESMTPS id AC0D220390 for ; Sat, 19 Dec 2020 15:35:33 +0000 (UTC) X-Originating-IP: 90.2.82.147 Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 1A3001BF203; Sat, 19 Dec 2020 15:35:30 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:14 +0100 Message-Id: <20201219153525.1361175-2-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 01/12] support/download/dl-wrapper: add concept of download post-processing X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In order to support package managers such as Cargo (Rust) or Go, we want to run some custom logic after the main download, but before packing the tarball and checking the hash. To implement this, this commit introduces a concept of download post-processing: if -p is passed to the dl-wrapper, then support/download/-post-process will be called. Signed-off-by: Thomas Petazzoni --- support/download/dl-wrapper | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 3315bd410e..2d74554213 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -25,7 +25,7 @@ main() { local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";; @@ -37,6 +37,7 @@ main() { r) recurse="-r";; f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; + p) post_process="${OPTARG}";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";; @@ -135,6 +136,12 @@ main() { continue fi + if [ -n "${post_process}" ] ; then + ${OLDPWD}/support/download/${post_process}-post-process \ + -o "${tmpf}" \ + -n "${raw_base_name}" + fi + # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}" From patchwork Sat Dec 19 15:35:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418717 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZC5G5Kz9sWK for ; Sun, 20 Dec 2020 02:35:47 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 77AE686FE4; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dDn-fc7k7QeQ; Sat, 19 Dec 2020 15:35:39 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 5881A86FE3; Sat, 19 Dec 2020 15:35:39 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 555901BF2C0 for ; Sat, 19 Dec 2020 15:35:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 4CF5686FE3 for ; Sat, 19 Dec 2020 15:35:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Fj4hQy2DP0iY for ; Sat, 19 Dec 2020 15:35:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by whitealder.osuosl.org (Postfix) with ESMTPS id 87AB286F78 for ; Sat, 19 Dec 2020 15:35:34 +0000 (UTC) Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 63E9220000D; Sat, 19 Dec 2020 15:35:32 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:15 +0100 Message-Id: <20201219153525.1361175-3-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 02/12] package/pkg-download.mk: add _DOWNLOAD_POST_PROCESS variable X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This will allow packages to register than a download post-processing is needed. Note that this variable is intentionally not documented: it is an internal variable meant to be set by package infrastructures, not directly by packages. Signed-off-by: Thomas Petazzoni --- package/pkg-download.mk | 1 + 1 file changed, 1 insertion(+) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 951d2fb554..175b9d7dba 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \ $(QUIET) \ From patchwork Sat Dec 19 15:35:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418718 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZC1b8Rz9sWH for ; Sun, 20 Dec 2020 02:35:46 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B356687074; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 84m5wjmiQwSJ; Sat, 19 Dec 2020 15:35:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id D972C86EDC; Sat, 19 Dec 2020 15:35:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id C970D1BF2C0 for ; Sat, 19 Dec 2020 15:35:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C6A1986ED1 for ; Sat, 19 Dec 2020 15:35:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WY2Ja-UXvIHj for ; Sat, 19 Dec 2020 15:35:36 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by fraxinus.osuosl.org (Postfix) with ESMTPS id BB2F786EBA for ; Sat, 19 Dec 2020 15:35:35 +0000 (UTC) X-Originating-IP: 90.2.82.147 Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id B61001BF207; Sat, 19 Dec 2020 15:35:33 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:16 +0100 Message-Id: <20201219153525.1361175-4-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 03/12] package/pkg-download.mk: add _DL_ENV variable X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This variable can be used by package to pass extra environment variables to the download logic. It will be used for the Go/Cargo vendoring. The _DL_ENV variable is intentionally not documented: at this point, it is not meant to be used by packages directly, but only by package infrastructures. Suggested-by: Yann E. MORIN Signed-off-by: Thomas Petazzoni --- package/pkg-download.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 175b9d7dba..c914d016e2 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -99,7 +99,9 @@ endif define DOWNLOAD $(Q)mkdir -p $($(2)_DL_DIR) - $(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ + $(Q)$(EXTRA_ENV) \ + $($(2)_DL_ENV) \ + flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ -c '$($(2)_DL_VERSION)' \ -d '$($(2)_DL_DIR)' \ -D '$(DL_DIR)' \ From patchwork Sat Dec 19 15:35:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418721 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZF2cccz9sWH for ; Sun, 20 Dec 2020 02:35:49 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id D18828715A; Sat, 19 Dec 2020 15:35:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8uhPLD0158_C; Sat, 19 Dec 2020 15:35:44 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id CE24C87148; Sat, 19 Dec 2020 15:35:44 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 0A4BF1BF2C0 for ; Sat, 19 Dec 2020 15:35:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0059120470 for ; Sat, 19 Dec 2020 15:35:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Rk98AvgEZgXk for ; Sat, 19 Dec 2020 15:35:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by silver.osuosl.org (Postfix) with ESMTPS id 27972203EC for ; Sat, 19 Dec 2020 15:35:36 +0000 (UTC) Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 01CD320000B; Sat, 19 Dec 2020 15:35:34 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:17 +0100 Message-Id: <20201219153525.1361175-5-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 04/12] support/download/post-process-helpers: add helper function for post process scripts X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function. Signed-off-by: Thomas Petazzoni --- support/download/post-process-helpers | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 support/download/post-process-helpers diff --git a/support/download/post-process-helpers b/support/download/post-process-helpers new file mode 100644 index 0000000000..bed8df2577 --- /dev/null +++ b/support/download/post-process-helpers @@ -0,0 +1,30 @@ + +unpack() { + dest="$1" + tarball="$2" + + mkdir ${dest} + tar -C ${dest} --strip-components=1 -xf ${tarball} +} + +repack() { + src="$1" + tarball="$2" + + # Generate the archive, sort with the C locale so that it is reproducible. + find "$(basename ${src})" -not -type d -print0 >files.list + LC_ALL=C sort -z files.list.sorted + + # let's use a fixed hardcoded date to be reproducible + date="2020-02-06 01:02:03 +0000" + + # Create GNU-format tarballs, since that's the format of the tarballs on + # sources.buildroot.org and used in the *.hash files + tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \ + --owner=0 --group=0 --mtime="${date}" -T files.list.sorted + gzip -6 -n new.tar.gz + mv "${tarball}" "${tarball}".old + mv new.tar.gz "${tarball}" + rm "${tarball}".old + rm -rf ${src} +} From patchwork Sat Dec 19 15:35:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418722 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZG0DYpz9sVM for ; Sun, 20 Dec 2020 02:35:50 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 5E2FF87039; Sat, 19 Dec 2020 15:35:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DntJ1GebHUbP; Sat, 19 Dec 2020 15:35:46 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 26CB986FEC; Sat, 19 Dec 2020 15:35:46 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 2D3BD1BF2C0 for ; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2A94D87072 for ; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JBRCm3xGdLwN for ; Sat, 19 Dec 2020 15:35:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay10.mail.gandi.net (relay10.mail.gandi.net [217.70.178.230]) by fraxinus.osuosl.org (Postfix) with ESMTPS id A724086EBA for ; Sat, 19 Dec 2020 15:35:40 +0000 (UTC) Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay10.mail.gandi.net (Postfix) with ESMTPSA id 8917B240007; Sat, 19 Dec 2020 15:35:36 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:18 +0100 Message-Id: <20201219153525.1361175-6-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Anisse Astier , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure. The -modcacherw flag is added to ensure that the Go cache is read/write, and can be deleted properly upon "make clean". The _LICENSE variable of golang packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account. The logic to generate go.mod when not available is moved to the download post-process helper, as it must be done before "go mod vendor" is executed. Also, "go mod init" is used instead of manually crafting go.mod. This was suggested by Christian Stewart . The Go module name is passed down to go-post-process using the BR_GOMOD environment variable. Signed-off-by: Thomas Petazzoni --- package/pkg-golang.mk | 20 ++++++++++-------- support/download/go-post-process | 35 ++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 9 deletions(-) create mode 100755 support/download/go-post-process diff --git a/package/pkg-golang.mk b/package/pkg-golang.mk index d07242310d..00a74a4d18 100644 --- a/package/pkg-golang.mk +++ b/package/pkg-golang.mk @@ -42,12 +42,13 @@ define inner-golang-package $(2)_BUILD_OPTS += \ -ldflags "$$($(2)_LDFLAGS)" \ + -modcacherw \ -tags "$$($(2)_TAGS)" \ -trimpath \ -p $(PARALLEL_JOBS) # Target packages need the Go compiler on the host. -$(2)_DEPENDENCIES += host-go +$(2)_DOWNLOAD_DEPENDENCIES += host-go $(2)_BUILD_TARGETS ?= . @@ -72,14 +73,15 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) # If the go.mod file does not exist, one is written with this root path. $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) -# Generate a go.mod file if it doesn't exist. Note: Go is configured -# to use the "vendor" dir and not make network calls. -define $(2)_GEN_GOMOD - if [ ! -f $$(@D)/go.mod ]; then \ - printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \ - fi -endef -$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD +$(2)_DOWNLOAD_POST_PROCESS = go +$(2)_DL_ENV = \ + $(HOST_GO_COMMON_ENV) \ + GOPROXY=direct \ + BR_GOMOD=$$($(2)_GOMOD) + +# Due to vendoring, it is pretty likely that not all licenses are +# listed in _LICENSE. +$(2)_LICENSE += , vendored dependencies licenses probably not listed # Build step. Only define it if not already defined by the package .mk # file. diff --git a/support/download/go-post-process b/support/download/go-post-process new file mode 100755 index 0000000000..4ca3e9a710 --- /dev/null +++ b/support/download/go-post-process @@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +set -e + +. $(dirname $0)/post-process-helpers + +# Parse our options +while getopts "n:o:" OPT; do + case "${OPT}" in + o) output="${OPTARG}";; + n) base_name="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac +done + +# Already vendored tarball, nothing to do +if tar tf ${output} | grep -q "^[^/]*/vendor" ; then + exit 0 +fi + +unpack ${base_name} ${output} + +# Do the Go vendoring +pushd ${base_name} > /dev/null + +# Generate go.mod if it doesn't exist +if [ ! -f go.mod ] && [ -n "${BR_GOMOD}" ]; then + go mod init ${BR_GOMOD} +fi + +go mod vendor -v -modcacherw +popd > /dev/null + +repack ${base_name} ${output} From patchwork Sat Dec 19 15:35:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418724 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZM1dG5z9sWF for ; Sun, 20 Dec 2020 02:35:54 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id B077E203C3; Sat, 19 Dec 2020 15:35:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TYOEmQ95Mr9x; Sat, 19 Dec 2020 15:35:47 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 688FD20485; Sat, 19 Dec 2020 15:35:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 8BC781BF2C0 for ; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 8810187649 for ; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OvcvmvE1fMr4 for ; Sat, 19 Dec 2020 15:35:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [217.70.183.198]) by hemlock.osuosl.org (Postfix) with ESMTPS id 634E087648 for ; Sat, 19 Dec 2020 15:35:42 +0000 (UTC) X-Originating-IP: 90.2.82.147 Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay6-d.mail.gandi.net (Postfix) with ESMTPSA id 4F76EC0003; Sat, 19 Dec 2020 15:35:38 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:19 +0100 Message-Id: <20201219153525.1361175-7-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 06/12] package/tinifier: new package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This is a Go package that needs vendor modules to be downloaded at build time. Signed-off-by: Thomas Petazzoni --- DEVELOPERS | 1 + package/Config.in | 1 + package/tinifier/Config.in | 10 ++++++++++ package/tinifier/tinifier.hash | 3 +++ package/tinifier/tinifier.mk | 13 +++++++++++++ 5 files changed, 28 insertions(+) create mode 100644 package/tinifier/Config.in create mode 100644 package/tinifier/tinifier.hash create mode 100644 package/tinifier/tinifier.mk diff --git a/DEVELOPERS b/DEVELOPERS index 9406f30cd5..1a7c22b718 100644 --- a/DEVELOPERS +++ b/DEVELOPERS @@ -2590,6 +2590,7 @@ F: package/rtc-tools/ F: package/sam-ba/ F: package/scons/ F: package/squashfs/ +F: package/tinifier/ F: package/wayland/ F: package/weston/ F: support/testing/tests/boot/test_syslinux.py diff --git a/package/Config.in b/package/Config.in index dc7139a49a..d5d8caef47 100644 --- a/package/Config.in +++ b/package/Config.in @@ -298,6 +298,7 @@ comment "Graphic applications" source "package/rrdtool/Config.in" source "package/stellarium/Config.in" source "package/tesseract-ocr/Config.in" + source "package/tinifier/Config.in" comment "Graphic libraries" source "package/cegui/Config.in" diff --git a/package/tinifier/Config.in b/package/tinifier/Config.in new file mode 100644 index 0000000000..fbadfe6bd9 --- /dev/null +++ b/package/tinifier/Config.in @@ -0,0 +1,10 @@ +config BR2_PACKAGE_TINIFIER + bool "tinifier" + depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS + help + CLI tool for images compressing + + This tool uses tinypng.com API endpoint for compressing your + local jpg/png images (it supports parallel jobs). + + https://github.com/tarampampam/tinifier diff --git a/package/tinifier/tinifier.hash b/package/tinifier/tinifier.hash new file mode 100644 index 0000000000..146700817b --- /dev/null +++ b/package/tinifier/tinifier.hash @@ -0,0 +1,3 @@ +# Locally calculated +sha256 707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c tinifier-2.1.0.tar.gz +sha256 791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6 LICENSE diff --git a/package/tinifier/tinifier.mk b/package/tinifier/tinifier.mk new file mode 100644 index 0000000000..b47d265a8e --- /dev/null +++ b/package/tinifier/tinifier.mk @@ -0,0 +1,13 @@ +################################################################################ +# +# tinifier +# +################################################################################ + +TINIFIER_VERSION = 2.1.0 +TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION)) +TINIFIER_LICENSE = MIT +TINIFIER_LICENSE_FILES = LICENSE +TINIFIER_GOMOD = tinifier + +$(eval $(golang-package)) From patchwork Sat Dec 19 15:35:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418725 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZM4Wrcz9sWH for ; Sun, 20 Dec 2020 02:35:55 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 30DC787146; Sat, 19 Dec 2020 15:35:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GHGqQJ1o8WdL; Sat, 19 Dec 2020 15:35:52 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 85A4787002; Sat, 19 Dec 2020 15:35:52 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 069A31BF2C0 for ; Sat, 19 Dec 2020 15:35:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 035DE87648 for ; Sat, 19 Dec 2020 15:35:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3vSmq2JTOsH3 for ; Sat, 19 Dec 2020 15:35:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay9-d.mail.gandi.net (relay9-d.mail.gandi.net [217.70.183.199]) by hemlock.osuosl.org (Postfix) with ESMTPS id 1CEC787649 for ; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) X-Originating-IP: 90.2.82.147 Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay9-d.mail.gandi.net (Postfix) with ESMTPSA id 0FDECFF807; Sat, 19 Dec 2020 15:35:39 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:20 +0100 Message-Id: <20201219153525.1361175-8-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 07/12] package/pkg-cargo.mk: introduce the cargo package infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Patrick Havelange In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo. Signed-off-by: Patrick Havelange [Thomas: add support for host-cargo-package] Signed-off-by: Thomas Petazzoni --- package/Makefile.in | 1 + package/pkg-cargo.mk | 140 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) create mode 100644 package/pkg-cargo.mk diff --git a/package/Makefile.in b/package/Makefile.in index 51f5cbce4f..2af123f36d 100644 --- a/package/Makefile.in +++ b/package/Makefile.in @@ -427,3 +427,4 @@ include package/pkg-waf.mk include package/pkg-golang.mk include package/pkg-meson.mk include package/pkg-qmake.mk +include package/pkg-cargo.mk diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk new file mode 100644 index 0000000000..058c6756bb --- /dev/null +++ b/package/pkg-cargo.mk @@ -0,0 +1,140 @@ +################################################################################ +# Cargo package infrastructure +# +# This file implements an infrastructure that eases development of package +# .mk files for Cargo packages. It should be used for all packages that use +# Cargo as their build system. +# +# See the Buildroot documentation for details on the usage of this +# infrastructure +# +# In terms of implementation, this Cargo infrastructure requires the .mk file +# to only specify metadata information about the package: name, version, +# download URL, etc. +# +# We still allow the package .mk file to override what the different steps +# are doing, if needed. For example, if _BUILD_CMDS is already defined, +# it is used as the list of commands to perform to build the package, +# instead of the default Cargo behaviour. The package can also define some +# post operation hooks. +# +################################################################################ + +################################################################################ +# inner-cargo-package -- defines how the configuration, compilation and +# installation of a cargo package should be done, implements a few hooks +# to tune the build process for cargo specifities and calls the generic +# package infrastructure to generate the necessary make targets +# +# argument 1 is the lowercase package name +# argument 2 is the uppercase package name, including a HOST_ prefix +# for host packages +# argument 3 is the uppercase package name, without the HOST_ prefix +# for host packages +# argument 4 is the type (target or host) +################################################################################ + +define inner-cargo-package + +# We need host-rustc to run cargo +$(2)_DEPENDENCIES += host-rustc + +$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo + +# Note: in all the steps below, we "cd" into the build directory to +# execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the +# manifest-path. Indeed while the latter seems to work, it in fact +# breaks in subtle ways as the way cargo searches for its +# configuration file is based (among other rules) on the current +# directory. This means that if cargo is started outside of a package +# directory, its configuration file will not be taken into account. +# +# Also, we pass: +# * --offline to prevent cargo from downloading anything +# * --locked to force cargo to use the Cargo.lock file, which ensures +# that a fixed set of dependency versions is used + +# +# Build step. Only define it if not already defined by the package .mk +# file. +# +ifndef $(2)_BUILD_CMDS +ifeq ($(4),target) +define $(2)_BUILD_CMDS + cd $$(@D) && \ + $$(TARGET_MAKE_ENV) \ + $$(TARGET_CONFIGURE_OPTS) \ + $$($(2)_CARGO_ENV) \ + cargo build \ + --offline \ + --target $$(RUSTC_TARGET_NAME) \ + $$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \ + --manifest-path Cargo.toml \ + --locked \ + $$($(2)_CARGO_BUILD_OPTS) +endef +else # ifeq ($(4),target) +define $(2)_BUILD_CMDS + cd $$(@D) && \ + $$(HOST_MAKE_ENV) \ + RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \ + $$($(2)_CARGO_ENV) \ + cargo build \ + --offline \ + --release \ + --manifest-path Cargo.toml \ + --locked \ + $$($(2)_CARGO_BUILD_OPTS) +endef +endif # ifeq ($(4),target) +endif # ifndef $(2)_BUILD_CMDS + +# +# Target installation step. Only define it if not already defined by +# the package .mk file. +# +ifndef $(2)_INSTALL_TARGET_CMDS +define $(2)_INSTALL_TARGET_CMDS + cd $$(@D) && \ + $$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ + cargo install \ + --target $$(RUSTC_TARGET_NAME) \ + --offline \ + --root $$(TARGET_DIR)/usr/ \ + --bins \ + --path ./ \ + --force \ + --locked \ + $$($(2)_CARGO_INSTALL_OPTS) +endef +endif + +ifndef $(2)_INSTALL_CMDS +define $(2)_INSTALL_CMDS + cd $$(@D) && \ + $$(HOST_MAKE_ENV) \ + RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \ + $$($(2)_CARGO_ENV) \ + cargo install \ + --offline \ + --root $$(HOST_DIR) \ + --bins \ + --path ./ \ + --force \ + --locked \ + $$($(2)_CARGO_INSTALL_OPTS) +endef +endif + +# Call the generic package infrastructure to generate the necessary +# make targets +$(call inner-generic-package,$(1),$(2),$(3),$(4)) + +endef + +################################################################################ +# cargo-package -- the target generator macro for Cargo packages +################################################################################ + +cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target) +host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host) From patchwork Sat Dec 19 15:35:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418723 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZK5GQrz9sWF for ; Sun, 20 Dec 2020 02:35:53 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1B7CB8706A; Sat, 19 Dec 2020 15:35:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3I59cx5CEBhs; Sat, 19 Dec 2020 15:35:51 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 4004C870E3; Sat, 19 Dec 2020 15:35:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 870C31BF2C0 for ; Sat, 19 Dec 2020 15:35:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 8369087655 for ; Sat, 19 Dec 2020 15:35:45 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GNiW-24G9m4c for ; Sat, 19 Dec 2020 15:35:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by hemlock.osuosl.org (Postfix) with ESMTPS id A905387648 for ; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) X-Originating-IP: 90.2.82.147 Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id 8E77C1C0002; Sat, 19 Dec 2020 15:35:41 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:21 +0100 Message-Id: <20201219153525.1361175-9-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 08/12] docs/manual/cargo: document the cargo-package infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Patrick Havelange The Buildroot manual was already providing some details on how to integrate Cargo packages, and those details now need to be updated with a proper documentation for the cargo-package infrastructure. Signed-off-by: Patrick Havelange [Thomas: numerous updates and extensions.] Signed-off-by: Thomas Petazzoni --- docs/manual/adding-packages-cargo.txt | 88 ++++++++++++--------------- 1 file changed, 39 insertions(+), 49 deletions(-) diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt index 8fcc80bcc6..c65a32f017 100644 --- a/docs/manual/adding-packages-cargo.txt +++ b/docs/manual/adding-packages-cargo.txt @@ -1,7 +1,7 @@ // -*- mode:doc; -*- // vim: set syntax=asciidoc: -=== Integration of Cargo-based packages +=== Infrastructure for Cargo-based packages Cargo is the package manager for the Rust programming language. It allows the user to build programs or libraries written in Rust, but it also downloads and @@ -10,7 +10,7 @@ called "crates". [[cargo-package-tutorial]] -==== Cargo-based package's +Config.in+ file +==== +cargo-package+ tutorial The +Config.in+ file of Cargo-based package 'foo' should contain: @@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain: 08: http://foosoftware.org/foo/ --------------------------- -==== Cargo-based package's +.mk+ file - -Buildroot does not (yet) provide a dedicated package infrastructure for -Cargo-based packages. So, we will explain how to write a +.mk+ file for such a -package. Let's start with an example: +And the +.mk+ file for this package should contain: ------------------------------ 01: ################################################################################ @@ -44,52 +40,48 @@ package. Let's start with an example: 10: FOO_LICENSE = GPL-3.0+ 11: FOO_LICENSE_FILES = COPYING 12: -13: FOO_DEPENDENCIES = host-rustc -14: -15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo -16: -17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE) -18: -19: FOO_CARGO_OPTS = \ -20: $(if $(BR2_ENABLE_DEBUG),,--release) \ -21: --target=$(RUSTC_TARGET_NAME) \ -22: --manifest-path=$(@D)/Cargo.toml -23: -24: define FOO_BUILD_CMDS -25: $(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \ -26: cargo build $(FOO_CARGO_OPTS) -27: endef -28: -29: define FOO_INSTALL_TARGET_CMDS -30: $(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \ -31: $(TARGET_DIR)/usr/bin/foo -32: endef -33: -34: $(eval $(generic-package)) +13: $(eval $(cargo-package)) -------------------------------- -The Makefile starts with the definition of the standard variables for package -declaration (lines 7 to 11). +The Makefile starts with the definition of the standard variables for +package declaration (lines 7 to 11). + +As seen in line 13, it is based on the +cargo-package+ +infrastructure. Cargo will be invoked automatically by this +infrastructure to build and install the package. + +It is still possible to define custom build commands or install +commands (i.e. with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS). +Those will then replace the commands from the cargo infrastructure. + +==== +cargo-package+ reference + +The main macros for the Cargo package infrastructure are ++cargo-package+ for target packages and +host-cargo-package+ for host +packages. + +Just like the generic infrastructure, the Cargo infrastructure works +by defining a number of variables before calling the +cargo-package+ +or +host-cargo-package+ macros. -As seen in line 34, it is based on the -xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines -the variables required by this particular infrastructure, where Cargo is -invoked: +First, all the package metadata information variables that exist in +the generic infrastructure also exist in the Cargo infrastructure: ++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+, ++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc. -* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required - to configure the cross-compilation of the package are passed via - +FOO_CONF_OPTS+. +A few additional variables, specific to the Cargo infrastructure, can +also be defined. Many of them are only useful in very specific cases, +typical packages will therefore only use a few of them. -* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on - the target. +* +FOO_CARGO_ENV+ can be used to pass additional variables in the + environment of +cargo+ invocations. It used at both build and + installation time -In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to -contain +host-cargo+. +* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to + +cargo+ at build time. -To sum it up, to add a new Cargo-based package, the Makefile example can be -copied verbatim then edited to replace all occurences of +FOO+ with the -uppercase name of the new package and update the values of the standard -variables. +* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to + +cargo+ at install time. ==== About Dependencies Management @@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the +cargo fetch+ command. Cargo maintains a local cache of the registry index and of git checkouts of the -crates, whose location is given by +$CARGO_HOME+. As seen in the package -Makefile example at line 15, this environment variable is set to -+$(HOST_DIR)/share/cargo+. +crates, whose location is given by +$CARGO_HOME+. This dependency download mechanism is not convenient when performing an offline build, as Cargo will fail to fetch the dependencies. In that case, it is advised From patchwork Sat Dec 19 15:35:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418726 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZP606Hz9sVM for ; Sun, 20 Dec 2020 02:35:57 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2FF9D87753; Sat, 19 Dec 2020 15:35:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ze+-DHpAZYZO; Sat, 19 Dec 2020 15:35:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 7C34B87649; Sat, 19 Dec 2020 15:35:53 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id DAC761BF2C0 for ; Sat, 19 Dec 2020 15:35:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id D700F87649 for ; Sat, 19 Dec 2020 15:35:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2yNCizlITCsL for ; Sat, 19 Dec 2020 15:35:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay5-d.mail.gandi.net (relay5-d.mail.gandi.net [217.70.183.197]) by hemlock.osuosl.org (Postfix) with ESMTPS id B290687652 for ; Sat, 19 Dec 2020 15:35:44 +0000 (UTC) X-Originating-IP: 90.2.82.147 Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay5-d.mail.gandi.net (Postfix) with ESMTPSA id D89021C0003; Sat, 19 Dec 2020 15:35:42 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:22 +0100 Message-Id: <20201219153525.1361175-10-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 09/12] package/ripgrep: convert to cargo infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Sam Voss , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Patrick Havelange Signed-off-by: Patrick Havelange Signed-off-by: Thomas Petazzoni --- package/ripgrep/ripgrep.mk | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-) diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk index 9dd8d58de1..97e9e2ce5f 100644 --- a/package/ripgrep/ripgrep.mk +++ b/package/ripgrep/ripgrep.mk @@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT -RIPGREP_DEPENDENCIES = host-rustc -RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo - -RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR) - -RIPGREP_CARGO_OPTS = \ - --target=$(RUSTC_TARGET_NAME) \ - --manifest-path=$(@D)/Cargo.toml - -ifeq ($(BR2_ENABLE_DEBUG),y) -RIPGREP_CARGO_BIN_SUBDIR = debug -else -RIPGREP_CARGO_OPTS += --release -RIPGREP_CARGO_BIN_SUBDIR = release -endif - -define RIPGREP_BUILD_CMDS - $(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \ - cargo build $(RIPGREP_CARGO_OPTS) -endef - -define RIPGREP_INSTALL_TARGET_CMDS - $(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \ - $(TARGET_DIR)/usr/bin/rg -endef - -$(eval $(generic-package)) +$(eval $(cargo-package)) From patchwork Sat Dec 19 15:35:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418727 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZR09kBz9sVM for ; Sun, 20 Dec 2020 02:35:59 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id CC4A487652; Sat, 19 Dec 2020 15:35:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XvwBfM-rjrGB; Sat, 19 Dec 2020 15:35:55 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id E64ED8770E; Sat, 19 Dec 2020 15:35:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 162AB1BF94D for ; Sat, 19 Dec 2020 15:35:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 0B3348706A for ; Sat, 19 Dec 2020 15:35:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cWpVer4HWLjZ for ; Sat, 19 Dec 2020 15:35:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by fraxinus.osuosl.org (Postfix) with ESMTPS id D5759870E3 for ; Sat, 19 Dec 2020 15:35:45 +0000 (UTC) Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 045FE200009; Sat, 19 Dec 2020 15:35:43 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:23 +0100 Message-Id: <20201219153525.1361175-11-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 10/12] package/sentry-cli: convert to host-cargo-package infrastructure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Joseph Kogut , Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Thomas Petazzoni --- package/sentry-cli/sentry-cli.mk | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-) diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk index 7e53f16026..40ca22fdf9 100644 --- a/package/sentry-cli/sentry-cli.mk +++ b/package/sentry-cli/sentry-cli.mk @@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION)) SENTRY_CLI_LICENSE = BSD-3-clause SENTRY_CLI_LICENSE_FILES = LICENSE -HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib +HOST_SENTRY_CLI_DEPENDENCIES = host-zlib -HOST_SENTRY_CLI_CARGO_ENV = \ - CARGO_HOME=$(HOST_DIR)/share/cargo \ - RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))" - -HOST_SENTRY_CLI_CARGO_OPTS = \ - --release \ - --manifest-path=$(@D)/Cargo.toml - -define HOST_SENTRY_CLI_BUILD_CMDS - $(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \ - cargo build $(HOST_SENTRY_CLI_CARGO_OPTS) -endef - -define HOST_SENTRY_CLI_INSTALL_CMDS - $(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \ - $(HOST_DIR)/bin/sentry-cli -endef - -$(eval $(host-generic-package)) +$(eval $(host-cargo-package)) From patchwork Sat Dec 19 15:35:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418729 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZV4fKVz9sVM for ; Sun, 20 Dec 2020 02:36:02 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 348B287648; Sat, 19 Dec 2020 15:35:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id a9r0Z8b2O2yb; Sat, 19 Dec 2020 15:35:57 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 44E6B8776A; Sat, 19 Dec 2020 15:35:57 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 2A9B31BF2C0 for ; Sat, 19 Dec 2020 15:35:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 2434687649 for ; Sat, 19 Dec 2020 15:35:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Wg+z2g8yg1yW for ; Sat, 19 Dec 2020 15:35:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay8-d.mail.gandi.net (relay8-d.mail.gandi.net [217.70.183.201]) by hemlock.osuosl.org (Postfix) with ESMTPS id 8DED787648 for ; Sat, 19 Dec 2020 15:35:47 +0000 (UTC) X-Originating-IP: 90.2.82.147 Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 315761BF204; Sat, 19 Dec 2020 15:35:45 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:24 +0100 Message-Id: <20201219153525.1361175-12-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 11/12] support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Joseph Kogut , Matt Weber , Patrick Havelange , Ryan Barnett , Sam Voss , Thomas Petazzoni , "Yann E. MORIN" Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This commit adds support/download/cargo-post-process to perform the vendoring on Cargo packages, and enables it in package/pkg-cargo.mk. Since it changes the contents of the tarballs for ripgrep and sentry-cli, it changes their hashes. To not have a different hash for the same version of ripgrep and sentry-cli, we bump their versions. It has to be done in the same commit as the Cargo vendoring to make the series bisectable. The _LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account. Signed-off-by: Thomas Petazzoni --- package/pkg-cargo.mk | 13 ++++++++-- package/ripgrep/ripgrep.hash | 2 +- package/ripgrep/ripgrep.mk | 2 +- package/sentry-cli/sentry-cli.hash | 2 +- package/sentry-cli/sentry-cli.mk | 2 +- support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++ 6 files changed, 53 insertions(+), 6 deletions(-) create mode 100755 support/download/cargo-post-process diff --git a/package/pkg-cargo.mk b/package/pkg-cargo.mk index 058c6756bb..371b8dd424 100644 --- a/package/pkg-cargo.mk +++ b/package/pkg-cargo.mk @@ -37,10 +37,17 @@ define inner-cargo-package # We need host-rustc to run cargo -$(2)_DEPENDENCIES += host-rustc +$(2)_DOWNLOAD_DEPENDENCIES += host-rustc $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo +$(2)_DOWNLOAD_POST_PROCESS = cargo +$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo + +# Due to vendoring, it is pretty likely that not all licenses are +# listed in _LICENSE. +$(2)_LICENSE += , vendored dependencies licenses probably not listed + # Note: in all the steps below, we "cd" into the build directory to # execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the # manifest-path. Indeed while the latter seems to work, it in fact @@ -50,7 +57,9 @@ $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo # directory, its configuration file will not be taken into account. # # Also, we pass: -# * --offline to prevent cargo from downloading anything +# * --offline to prevent cargo from downloading anything: all +# dependencies should have been built by the download post +# process logic # * --locked to force cargo to use the Cargo.lock file, which ensures # that a fixed set of dependency versions is used diff --git a/package/ripgrep/ripgrep.hash b/package/ripgrep/ripgrep.hash index 0841c0185c..81ac905d3d 100644 --- a/package/ripgrep/ripgrep.hash +++ b/package/ripgrep/ripgrep.hash @@ -1,3 +1,3 @@ # Locally calculated -sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz +sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT diff --git a/package/ripgrep/ripgrep.mk b/package/ripgrep/ripgrep.mk index 97e9e2ce5f..d21f88a6e8 100644 --- a/package/ripgrep/ripgrep.mk +++ b/package/ripgrep/ripgrep.mk @@ -4,7 +4,7 @@ # ################################################################################ -RIPGREP_VERSION = 0.8.1 +RIPGREP_VERSION = 12.1.1 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT diff --git a/package/sentry-cli/sentry-cli.hash b/package/sentry-cli/sentry-cli.hash index 3b0733a276..63b0c812bc 100644 --- a/package/sentry-cli/sentry-cli.hash +++ b/package/sentry-cli/sentry-cli.hash @@ -1,3 +1,3 @@ # locally calculated -sha256 5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd sentry-cli-1.57.0.tar.gz +sha256 a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4 sentry-cli-1.59.0.tar.gz sha256 9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14 LICENSE diff --git a/package/sentry-cli/sentry-cli.mk b/package/sentry-cli/sentry-cli.mk index 40ca22fdf9..58f5f1e325 100644 --- a/package/sentry-cli/sentry-cli.mk +++ b/package/sentry-cli/sentry-cli.mk @@ -4,7 +4,7 @@ # ################################################################################ -SENTRY_CLI_VERSION = 1.57.0 +SENTRY_CLI_VERSION = 1.59.0 SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION)) SENTRY_CLI_LICENSE = BSD-3-clause SENTRY_CLI_LICENSE_FILES = LICENSE diff --git a/support/download/cargo-post-process b/support/download/cargo-post-process new file mode 100755 index 0000000000..5081476385 --- /dev/null +++ b/support/download/cargo-post-process @@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +set -e + +. $(dirname $0)/post-process-helpers + +while getopts "n:o:" OPT; do + case "${OPT}" in + o) output="${OPTARG}";; + n) base_name="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac +done + +# Already vendored tarball, nothing to do +if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then + exit 0 +fi + +unpack ${base_name} ${output} + +# Do the Cargo vendoring +pushd ${base_name} > /dev/null +cargo vendor --locked VENDOR +echo $? +# Create the local .cargo/config with vendor info +mkdir -p .cargo/ +cat <.cargo/config +[source.crates-io] +replace-with = "vendored-sources" + +[source.vendored-sources] +directory = "VENDOR" +EOF +popd > /dev/null + +repack ${base_name} ${output} From patchwork Sat Dec 19 15:35:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Petazzoni X-Patchwork-Id: 1418728 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=bootlin.com Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CyqZT14Mfz9sVM for ; Sun, 20 Dec 2020 02:36:01 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B204686EDC; Sat, 19 Dec 2020 15:35:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OIXBjzZ0UNBN; Sat, 19 Dec 2020 15:35:59 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 164768716A; Sat, 19 Dec 2020 15:35:59 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 1B7201BF2C0 for ; Sat, 19 Dec 2020 15:35:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id E2F4A20498 for ; Sat, 19 Dec 2020 15:35:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id reGvtu9gYYvK for ; Sat, 19 Dec 2020 15:35:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from relay12.mail.gandi.net (relay12.mail.gandi.net [217.70.178.232]) by silver.osuosl.org (Postfix) with ESMTPS id 7E17E20467 for ; Sat, 19 Dec 2020 15:35:48 +0000 (UTC) Received: from localhost (aputeaux-654-1-223-147.w90-2.abo.wanadoo.fr [90.2.82.147]) (Authenticated sender: thomas.petazzoni@bootlin.com) by relay12.mail.gandi.net (Postfix) with ESMTPSA id 8598120000D; Sat, 19 Dec 2020 15:35:46 +0000 (UTC) From: Thomas Petazzoni To: Buildroot List Date: Sat, 19 Dec 2020 16:35:25 +0100 Message-Id: <20201219153525.1361175-13-thomas.petazzoni@bootlin.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> References: <20201219153525.1361175-1-thomas.petazzoni@bootlin.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 12/12] docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber , Patrick Havelange , Ryan Barnett , Thomas Petazzoni , "Yann E. MORIN" , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Now that we have vendoring support for Cargo packages, let's rewrite the dependency management section in a more accurate way. We drop the part about the local cache of the registry, because +CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is not shared between builds nor preserved accross builds, so its effect as a cache is limited. Signed-off-by: Thomas Petazzoni --- docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/docs/manual/adding-packages-cargo.txt b/docs/manual/adding-packages-cargo.txt index c65a32f017..fb3e7d0780 100644 --- a/docs/manual/adding-packages-cargo.txt +++ b/docs/manual/adding-packages-cargo.txt @@ -85,15 +85,13 @@ typical packages will therefore only use a few of them. ==== About Dependencies Management -A crate can depend on other libraries from crates.io or git repositories, listed -in its Cargo.toml file. Before starting a build, Cargo usually downloads -automatically them. This step can also be performed independently, via the -+cargo fetch+ command. - -Cargo maintains a local cache of the registry index and of git checkouts of the -crates, whose location is given by +$CARGO_HOME+. - -This dependency download mechanism is not convenient when performing an offline -build, as Cargo will fail to fetch the dependencies. In that case, it is advised -to generate a tarball of the dependencies using the +cargo vendor+ and add it to -+FOO_EXTRA_DOWNLOADS+. +A crate can depend on other libraries from crates.io or git +repositories, listed in its Cargo.toml file. Buildroot automatically +takes care of downloading such dependencies as part of the download +step of packages that use the +cargo-package+ infrastructure. Such +dependencies are then kept together with the package source code in +the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of +the package's tarball includes such dependencies. + +This mechanism ensures that any change in the dependencies will be +detected, and allows the build to be performed completely offline.