From patchwork Wed May 13 19:19:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthias Schiffer X-Patchwork-Id: 1289655 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.openwrt.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=universe-factory.net Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20170209 header.b=UBSlyGuM; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49MkzY6pfJz9sSd for ; Thu, 14 May 2020 05:20:57 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Subject:MIME-Version:References: In-Reply-To:Message-Id:Date:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=T2gakAfBk7IieZLB1rYHcICdlTBEptPTE4348DoAOxg=; b=UBSlyGuMGQ72nh KaZerdUKNgkD3mCbFXqhhfEQRkNtpHJt4O5h/N0a3ATBHRfFOzX+vWC3BVS8sA42+Bt/a5TwaEACj fW3a8/FieMjX9OrS4STKN8b6/jFjkfxBhbRMbGrKFMdaSO4sKpHQqjtRYeegcMq3l3ebmCtrPwl0/ 6LlcItzqzxD0WMByllVZ9NQrTxA+BJilbuXFXSUICyz07l61fCMZfniWPOU5lf6lkKQQeUSkbgaNj rSDiiC2Ianj5pGaey2jyzjeDwb6FuejK/OrMdlx87+RpzT7A0CkauLNUwFMSDbuS1LHY+7OKNal11 kfwLrpZK44rIWIwnAFng==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYww2-00089c-Hz; Wed, 13 May 2020 19:20:54 +0000 Received: from orthanc.universe-factory.net ([2001:19f0:6c01:100::1]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYwvf-0007hn-2c for openwrt-devel@lists.openwrt.org; Wed, 13 May 2020 19:20:33 +0000 Received: from localhost.localdomain (unknown [IPv6:2001:19f0:6c01:100::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orthanc.universe-factory.net (Postfix) with ESMTPSA id 8276B251F2; Wed, 13 May 2020 21:20:24 +0200 (CEST) From: Matthias Schiffer To: openwrt-devel@lists.openwrt.org Date: Wed, 13 May 2020 21:19:44 +0200 Message-Id: <96937b728d9ddd55df3cfdb6e3816cff3f6fd392.1589396871.git.mschiffer@universe-factory.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200513_122031_427877_A4BFD230 X-CRM114-Status: GOOD ( 10.34 ) X-Spam-Score: -0.0 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record Subject: [OpenWrt-Devel] [PATCH 1/2] tools: add zstd X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mschiffer@universe-factory.net Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org Signed-off-by: Matthias Schiffer --- tools/zstd/Makefile | 20 ++++++ ...re-portable-header-prefix-usage-1987.patch | 61 +++++++++++++++++++ 2 files changed, 81 insertions(+) create mode 100644 tools/zstd/Makefile create mode 100644 tools/zstd/patches/0001-build-issue-More-portable-header-prefix-usage-1987.patch diff --git a/tools/zstd/Makefile b/tools/zstd/Makefile new file mode 100644 index 000000000000..dc0bf8ffaff8 --- /dev/null +++ b/tools/zstd/Makefile @@ -0,0 +1,20 @@ +include $(TOPDIR)/rules.mk + +PKG_NAME:=zstd +PKG_VERSION:=1.4.4 + +PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz +PKG_SOURCE_URL:=@GITHUB/facebook/zstd/releases/download/v$(PKG_VERSION) +PKG_HASH:=a364f5162c7d1a455cc915e8e3cf5f4bd8b75d09bc0f53965b0c9ca1383c52c8 + +PKG_LICENSE:=BSD-3-Clause +PKG_LICENSE_FILES:=LICENSE +PKG_CPE_ID:=cpe:/a:facebook:zstandard + +HOST_BUILD_PARALLEL:=1 + +include $(INCLUDE_DIR)/host-build.mk + +HOST_MAKE_FLAGS = PREFIX=$(HOST_BUILD_PREFIX) + +$(eval $(call HostBuild)) diff --git a/tools/zstd/patches/0001-build-issue-More-portable-header-prefix-usage-1987.patch b/tools/zstd/patches/0001-build-issue-More-portable-header-prefix-usage-1987.patch new file mode 100644 index 000000000000..6d743aa38552 --- /dev/null +++ b/tools/zstd/patches/0001-build-issue-More-portable-header-prefix-usage-1987.patch @@ -0,0 +1,61 @@ +From 06a57cf57e3c4e887cadcf688e3081154f3f6db4 Mon Sep 17 00:00:00 2001 +Message-Id: <06a57cf57e3c4e887cadcf688e3081154f3f6db4.1589392463.git.mschiffer@universe-factory.net> +From: Bimba Shrestha +Date: Thu, 6 Feb 2020 14:10:51 -0800 +Subject: [PATCH] [build-issue] More portable header prefix usage (#) (#1987) + +* make 4.3 build issue fix + +* Changing header name and adding comment +--- + programs/Makefile | 11 +++++++---- + 1 file changed, 7 insertions(+), 4 deletions(-) + +diff --git a/programs/Makefile b/programs/Makefile +index b75314a83f43..a9ee3cb5311b 100644 +--- a/programs/Makefile ++++ b/programs/Makefile +@@ -94,9 +94,12 @@ endif + + VOID = /dev/null + ++# Make 4.3 doesn't support '\#' anymore (https://lwn.net/Articles/810071/) ++NUM_SYMBOL := \# ++ + # thread detection + NO_THREAD_MSG := ==> no threads, building without multithreading support +-HAVE_PTHREAD := $(shell printf '\#include \nint main(void) { return 0; }' > have_pthread.c && $(CC) $(FLAGS) -o have_pthread$(EXT) have_pthread.c -pthread 2> $(VOID) && rm have_pthread$(EXT) && echo 1 || echo 0; rm have_pthread.c) ++HAVE_PTHREAD := $(shell printf '$(NUM_SYMBOL)include \nint main(void) { return 0; }' > have_pthread.c && $(CC) $(FLAGS) -o have_pthread$(EXT) have_pthread.c -pthread 2> $(VOID) && rm have_pthread$(EXT) && echo 1 || echo 0; rm have_pthread.c) + HAVE_THREAD := $(shell [ "$(HAVE_PTHREAD)" -eq "1" -o -n "$(filter Windows%,$(OS))" ] && echo 1 || echo 0) + ifeq ($(HAVE_THREAD), 1) + THREAD_MSG := ==> building with threading support +@@ -108,7 +111,7 @@ endif + + # zlib detection + NO_ZLIB_MSG := ==> no zlib, building zstd without .gz support +-HAVE_ZLIB := $(shell printf '\#include \nint main(void) { return 0; }' > have_zlib.c && $(CC) $(FLAGS) -o have_zlib$(EXT) have_zlib.c -lz 2> $(VOID) && rm have_zlib$(EXT) && echo 1 || echo 0; rm have_zlib.c) ++HAVE_ZLIB := $(shell printf '$(NUM_SYMBOL)include \nint main(void) { return 0; }' > have_zlib.c && $(CC) $(FLAGS) -o have_zlib$(EXT) have_zlib.c -lz 2> $(VOID) && rm have_zlib$(EXT) && echo 1 || echo 0; rm have_zlib.c) + ifeq ($(HAVE_ZLIB), 1) + ZLIB_MSG := ==> building zstd with .gz compression support + ZLIBCPP = -DZSTD_GZCOMPRESS -DZSTD_GZDECOMPRESS +@@ -119,7 +122,7 @@ endif + + # lzma detection + NO_LZMA_MSG := ==> no liblzma, building zstd without .xz/.lzma support +-HAVE_LZMA := $(shell printf '\#include \nint main(void) { return 0; }' > have_lzma.c && $(CC) $(FLAGS) -o have_lzma$(EXT) have_lzma.c -llzma 2> $(VOID) && rm have_lzma$(EXT) && echo 1 || echo 0; rm have_lzma.c) ++HAVE_LZMA := $(shell printf '$(NUM_SYMBOL)include \nint main(void) { return 0; }' > have_lzma.c && $(CC) $(FLAGS) -o have_lzma$(EXT) have_lzma.c -llzma 2> $(VOID) && rm have_lzma$(EXT) && echo 1 || echo 0; rm have_lzma.c) + ifeq ($(HAVE_LZMA), 1) + LZMA_MSG := ==> building zstd with .xz/.lzma compression support + LZMACPP = -DZSTD_LZMACOMPRESS -DZSTD_LZMADECOMPRESS +@@ -130,7 +133,7 @@ endif + + # lz4 detection + NO_LZ4_MSG := ==> no liblz4, building zstd without .lz4 support +-HAVE_LZ4 := $(shell printf '\#include \n\#include \nint main(void) { return 0; }' > have_lz4.c && $(CC) $(FLAGS) -o have_lz4$(EXT) have_lz4.c -llz4 2> $(VOID) && rm have_lz4$(EXT) && echo 1 || echo 0; rm have_lz4.c) ++HAVE_LZ4 := $(shell printf '$(NUM_SYMBOL)include \n\#include \nint main(void) { return 0; }' > have_lz4.c && $(CC) $(FLAGS) -o have_lz4$(EXT) have_lz4.c -llz4 2> $(VOID) && rm have_lz4$(EXT) && echo 1 || echo 0; rm have_lz4.c) + ifeq ($(HAVE_LZ4), 1) + LZ4_MSG := ==> building zstd with .lz4 compression support + LZ4CPP = -DZSTD_LZ4COMPRESS -DZSTD_LZ4DECOMPRESS +-- +2.26.2 + From patchwork Wed May 13 19:19:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthias Schiffer X-Patchwork-Id: 1289653 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.openwrt.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=universe-factory.net Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20170209 header.b=FLStTTwI; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49MkzM3BYnz9sSf for ; Thu, 14 May 2020 05:20:47 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Subject:MIME-Version:References: In-Reply-To:Message-Id:Date:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=g96ebZHRPc4WWjCstylsw/bLTKaGkA68JpnHzzI+IHo=; b=FLStTTwIsFo9gi Y7NHryOsry5mmJsu7JKOZiHsJhcDHUx0NilaP7649SwM2O8DqTagpdqS5FfyXH3CvaDH8JBaE1v9U DHjxojCmxSmnURAXSSe7jCCJy0I9dz4wQD7FT1ouAbuuo1Amekq+fNxWHTnXRSVu2xxLcogeWPOn/ bb7K8G82DAXTsjlVzPVidZWOO5nu89dgC7Sl+TCi4dUCaNtRJKw8RITh5BEuiwtOj0+5IgnDFSlPZ IuCel/t3DZDI0KJlUcwEkqoa99dE4DaY0B9sTaI0dgJVISQnYbjImYOoTXQ1DpVVcspg89040yMF2 fWQNVSF9gSkdJ06hdNBg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYwvh-0007jJ-TS; Wed, 13 May 2020 19:20:33 +0000 Received: from orthanc.universe-factory.net ([104.238.176.138]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jYwve-0007ho-Ow for openwrt-devel@lists.openwrt.org; Wed, 13 May 2020 19:20:32 +0000 Received: from localhost.localdomain (unknown [IPv6:2001:19f0:6c01:100::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orthanc.universe-factory.net (Postfix) with ESMTPSA id A2520251F3; Wed, 13 May 2020 21:20:24 +0200 (CEST) From: Matthias Schiffer To: openwrt-devel@lists.openwrt.org Date: Wed, 13 May 2020 21:19:45 +0200 Message-Id: X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200513_122030_972696_F2C8F321 X-CRM114-Status: GOOD ( 10.19 ) X-Spam-Score: -0.0 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record Subject: [OpenWrt-Devel] [PATCH 2/2] build: compress kernel debuginfo using zstd X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mschiffer@universe-factory.net Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org zstd with its default settings (compression level -3) compresses better than bzip2 -9 (which is the default setting), and is an order of magnitude faster. I made the following measurements for the most common compression tools (all standard Debian Buster versions, default flags unless noted otherwise), using the debug information of a large x86-64 kernel with ALL_KMODS: * kernel-debug.tar: 376M * kernel-debug.tar.gz: 101M, compressed in ~12s * kernel-debug.tar.bz2: 91M, compressed in ~15s * kernel-debug.tar.xz: 57M, compressed in ~101s * kernel-debug.tar.zst: 86M, compressed in ~1s With zstd, there is still some room for improvement by increasing the compression, but the slight increase in compression ratio (22.83% -> 19.46%) does not justify the significant increase in compression time (about 5 times on my machine) in my opinion. Note that multithreaded compression (-T argument) does not affect reproducibility with zstd. Signed-off-by: Matthias Schiffer --- include/kernel-build.mk | 2 +- tools/Makefile | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/include/kernel-build.mk b/include/kernel-build.mk index c371e78ab9b0..32c91a5b8359 100644 --- a/include/kernel-build.mk +++ b/include/kernel-build.mk @@ -70,7 +70,7 @@ ifdef CONFIG_COLLECT_KERNEL_DEBUG $(FIND) $(KERNEL_BUILD_DIR)/debug -type f | $(XARGS) $(KERNEL_CROSS)strip --only-keep-debug $(TAR) c -C $(KERNEL_BUILD_DIR) debug \ $(if $(SOURCE_DATE_EPOCH),--mtime="@$(SOURCE_DATE_EPOCH)") \ - | bzip2 -c -9 > $(BIN_DIR)/kernel-debug.tar.bz2 + | zstd -T0 -f -o $(BIN_DIR)/kernel-debug.tar.zst endef endif diff --git a/tools/Makefile b/tools/Makefile index 02556d4b83f3..169f36c5bb69 100644 --- a/tools/Makefile +++ b/tools/Makefile @@ -37,6 +37,7 @@ tools-y += lzma squashfskit4 zip tools-$(BUILD_B43_TOOLS) += b43-tools tools-$(BUILD_ISL) += isl tools-$(CONFIG_USE_SPARSE) += sparse +tools-$(CONFIG_COLLECT_KERNEL_DEBUG) += zstd tools-$(CONFIG_TARGET_apm821xx)$(CONFIG_TARGET_gemini) += genext2fs tools-$(CONFIG_TARGET_tegra) += cbootimage cbootimage-configs From patchwork Sun May 17 11:51:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthias Schiffer X-Patchwork-Id: 1292122 X-Patchwork-Delegate: mschiffer@universe-factory.net Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.openwrt.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=universe-factory.net Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20170209 header.b=IHk4gDpS; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49Q0qw6f38z9sTD for ; Sun, 17 May 2020 21:52:12 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Subject:MIME-Version:References: In-Reply-To:Message-Id:Date:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ALtVAESNoBYDf1eP2tKKd4UO0D+/Q4LpHlAw7WesQuw=; b=IHk4gDpSuPUH3R FZ4kHHlD89UK8FjsfY76bvWD2YrC4UgHqDlotk3Sp1LM8PGJvX1c77TX5mwrbEpajL5DTbTV5QAoQ IFq9gY7DydrQYgLwuttTHb1A1lFy953gLMcgj4WR25ypaFnarVkzstbndZx8WnLm8USBtr4cVRtlQ W6agEgMz7tGmx+CHN7mCZal44/PJmUkkvAhaVfthDZb0WT5joM2WKzkcaZ8SVjHR27j0E6/njj3Ju k0BFga9zmRsqtXoxaPzhhw7D+gQDoVcZkVR4DgCxV+VNR8eiJ1lO7VFdfURRyoyiKbG6m7O2Ebc0y kP4YB1eIRXTWIUPplspg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jaHpv-0006HW-Vr; Sun, 17 May 2020 11:52:07 +0000 Received: from orthanc.universe-factory.net ([2001:19f0:6c01:100::1]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jaHpl-00067g-35 for openwrt-devel@lists.openwrt.org; Sun, 17 May 2020 11:51:58 +0000 Received: from localhost.localdomain (unknown [IPv6:2001:19f0:6c01:100::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orthanc.universe-factory.net (Postfix) with ESMTPSA id 2802125782; Sun, 17 May 2020 13:51:53 +0200 (CEST) From: Matthias Schiffer To: openwrt-devel@lists.openwrt.org Date: Sun, 17 May 2020 13:51:42 +0200 Message-Id: <2d9ce4d894da492ed79b6a2d3ba51d6e859e81a2.1589716209.git.mschiffer@universe-factory.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200517_045157_276303_A8145EAD X-CRM114-Status: GOOD ( 10.51 ) X-Spam-Score: -0.0 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record Subject: [OpenWrt-Devel] [PATCH v2 3/4] build: compress kernel debuginfo using zstd X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matthias Schiffer Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org zstd with its default settings (compression level -3) compresses better than bzip2 -9 (which is the default setting), and is an order of magnitude faster. I made the following measurements for the most common compression tools (all standard Debian Buster versions, default flags unless noted otherwise), using the debug information of a large x86-64 kernel with ALL_KMODS: * kernel-debug.tar: 376M * kernel-debug.tar.gz: 101M, compressed in ~12s * kernel-debug.tar.bz2: 91M, compressed in ~15s * kernel-debug.tar.xz: 57M, compressed in ~101s * kernel-debug.tar.zst: 86M, compressed in ~1s With zstd, there is still some room for improvement by increasing the compression, but the slight increase in compression ratio (22.83% -> 19.46%) does not justify the significant increase in compression time (about 5 times on my machine) in my opinion. Note that multithreaded compression (-T argument) does not affect reproducibility with zstd. Signed-off-by: Matthias Schiffer --- include/kernel-build.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/kernel-build.mk b/include/kernel-build.mk index c371e78ab9b0..32c91a5b8359 100644 --- a/include/kernel-build.mk +++ b/include/kernel-build.mk @@ -70,7 +70,7 @@ ifdef CONFIG_COLLECT_KERNEL_DEBUG $(FIND) $(KERNEL_BUILD_DIR)/debug -type f | $(XARGS) $(KERNEL_CROSS)strip --only-keep-debug $(TAR) c -C $(KERNEL_BUILD_DIR) debug \ $(if $(SOURCE_DATE_EPOCH),--mtime="@$(SOURCE_DATE_EPOCH)") \ - | bzip2 -c -9 > $(BIN_DIR)/kernel-debug.tar.bz2 + | zstd -T0 -f -o $(BIN_DIR)/kernel-debug.tar.zst endef endif From patchwork Sun May 17 11:51:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthias Schiffer X-Patchwork-Id: 1292123 X-Patchwork-Delegate: mschiffer@universe-factory.net Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.openwrt.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=universe-factory.net Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20170209 header.b=L1+lY6X5; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49Q0rL1hmdz9sT8 for ; Sun, 17 May 2020 21:52:34 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Subject:MIME-Version:References: In-Reply-To:Message-Id:Date:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ASGZCAT7OyBwDZbPQH4F1ja/fsWrYP2r6sWgr8nSGVM=; b=L1+lY6X5Aq89zC cG96a9+flZ9XF5WS2WKIKq4EWsM8L2X8+62Qti3AAT4DVd1ZlmXuxUQNnAnf1j4Yxz9EBcaD68LjA tudE1rCanwvmqPOFZ3VcmMrZkPDUhLeC/xoTNKvTFZdauHNEuJJ2NFysbFJaFbG6EtcmM+zoWUqBj BxZwOu1PuRpVUWp08bUxWECo/RSNC0xlgHDPwTpMhBgkrD40jSQlqn+2qYDznUPkEjdNM3POA0qVJ YCUfhC7380jQgbNrrLZ5168WFGQxkm+AepVhZQJZ8SL7u+VPrdU48idy4JmjPMMYwzl5Xb6aUrA/1 VRqrSzQlvQqMHNWJGVXg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jaHq8-0006X2-1w; Sun, 17 May 2020 11:52:20 +0000 Received: from orthanc.universe-factory.net ([104.238.176.138]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jaHpl-00067h-CL for openwrt-devel@lists.openwrt.org; Sun, 17 May 2020 11:51:59 +0000 Received: from localhost.localdomain (unknown [IPv6:2001:19f0:6c01:100::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by orthanc.universe-factory.net (Postfix) with ESMTPSA id 4AB5825783; Sun, 17 May 2020 13:51:53 +0200 (CEST) From: Matthias Schiffer To: openwrt-devel@lists.openwrt.org Date: Sun, 17 May 2020 13:51:43 +0200 Message-Id: <0f48abf2ac872957d6a4a150ead39564053f2afc.1589716209.git.mschiffer@universe-factory.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: References: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200517_045157_576171_7889959B X-CRM114-Status: UNSURE ( 6.95 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -0.0 (/) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-0.0 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record Subject: [OpenWrt-Devel] [PATCH v2 4/4] build: use zstd for SDK and ImageBuilder tarballs X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matthias Schiffer Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org Comression level -19 was chosen as it provides a very good tradeoff between compression ratio and performance, especially in multithreaded operation. Signed-off-by: Matthias Schiffer --- target/imagebuilder/Makefile | 8 ++++---- target/sdk/Makefile | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/target/imagebuilder/Makefile b/target/imagebuilder/Makefile index b463feb456ee..5c09109150a8 100644 --- a/target/imagebuilder/Makefile +++ b/target/imagebuilder/Makefile @@ -21,7 +21,7 @@ IB_IDIR:=$(patsubst $(TOPDIR)/%,$(PKG_BUILD_DIR)/%,$(STAGING_DIR_IMAGE)) all: compile -$(BIN_DIR)/$(IB_NAME).tar.xz: clean +$(BIN_DIR)/$(IB_NAME).tar.zst: clean rm -rf $(PKG_BUILD_DIR) mkdir -p $(IB_KDIR) $(IB_LDIR) $(PKG_BUILD_DIR)/staging_dir/host/lib \ $(PKG_BUILD_DIR)/target $(PKG_BUILD_DIR)/scripts $(IB_DTSDIR) @@ -86,12 +86,12 @@ endif (cd $(PKG_BUILD_DIR); find staging_dir/host/bin/ $(IB_LDIR)/scripts/dtc/ -type f | \ $(XARGS) $(SCRIPT_DIR)/bundle-libraries.sh $(PKG_BUILD_DIR)/staging_dir/host) STRIP=sstrip $(SCRIPT_DIR)/rstrip.sh $(PKG_BUILD_DIR)/staging_dir/host/bin/ - $(TAR) -cf - -C $(BUILD_DIR) $(IB_NAME) | xz -T$(if $(filter 1,$(NPROC)),2,0) -zc -7e > $@ + $(TAR) -cf - -C $(BUILD_DIR) $(IB_NAME) | zstd -T0 -19 -f -o $@ download: prepare: -compile: $(BIN_DIR)/$(IB_NAME).tar.xz +compile: $(BIN_DIR)/$(IB_NAME).tar.zst install: compile clean: FORCE - rm -rf $(PKG_BUILD_DIR) $(BIN_DIR)/$(IB_NAME).tar.xz + rm -rf $(PKG_BUILD_DIR) $(BIN_DIR)/$(IB_NAME).tar.zst diff --git a/target/sdk/Makefile b/target/sdk/Makefile index 6d818347204a..13389c849958 100644 --- a/target/sdk/Makefile +++ b/target/sdk/Makefile @@ -81,7 +81,7 @@ KERNEL_FILES := $(patsubst $(TOPDIR)/%,%,$(wildcard $(addprefix $(LINUX_DIR)/,$( all: compile -$(BIN_DIR)/$(SDK_NAME).tar.xz: clean +$(BIN_DIR)/$(SDK_NAME).tar.zst: clean mkdir -p $(SDK_BUILD_DIR)/dl $(SDK_BUILD_DIR)/package $(CP) -L $(INCLUDE_DIR) $(SCRIPT_DIR) $(SDK_BUILD_DIR)/ $(TAR) -cf - -C $(TOPDIR) \ @@ -156,13 +156,13 @@ $(BIN_DIR)/$(SDK_NAME).tar.xz: clean find $(SDK_BUILD_DIR) -name CVS | $(XARGS) rm -rf -make -C $(SDK_BUILD_DIR)/scripts/config clean (cd $(BUILD_DIR); \ - tar -I 'xz -7e' -cf $@ $(SDK_NAME); \ + tar -I 'zstd -T0 -19' -cf $@ $(SDK_NAME); \ ) download: prepare: -compile: $(BIN_DIR)/$(SDK_NAME).tar.xz +compile: $(BIN_DIR)/$(SDK_NAME).tar.zst install: compile clean: - rm -rf $(SDK_BUILD_DIR) $(BIN_DIR)/$(SDK_NAME).tar.xz + rm -rf $(SDK_BUILD_DIR) $(BIN_DIR)/$(SDK_NAME).tar.zst