| Message ID | d9a8cdcbfe046467dcb5e19e2bedcc773092333d.1589396871.git.mschiffer@universe-factory.net |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Switch to zstd for kernel debuginfo compression | expand |
śr., 13 maj 2020 o 21:19 Matthias Schiffer <mschiffer@universe-factory.net> napisał(a): > > zstd with its default settings (compression level -3) compresses better > than bzip2 -9 (which is the default setting), and is an order of magnitude > faster. > > I made the following measurements for the most common compression tools > (all standard Debian Buster versions, default flags unless noted > otherwise), using the debug information of a large x86-64 kernel with > ALL_KMODS: > > * kernel-debug.tar: 376M > * kernel-debug.tar.gz: 101M, compressed in ~12s > * kernel-debug.tar.bz2: 91M, compressed in ~15s > * kernel-debug.tar.xz: 57M, compressed in ~101s > * kernel-debug.tar.zst: 86M, compressed in ~1s >... Impressive results. Perhaps it should also be used for SquashFS instead of LZMA //Thomas
Could we use that for the SDK/ImageBuilder as well? Best, Paul
On 5/16/20 3:38 AM, Paul Spooren wrote: > Could we use that for the SDK/ImageBuilder as well? > > Best, > Paul Sure, have some benchmark results with the current ath79 snapshot IB: * openwrt-imagebuilder-ath79-generic.Linux-x86_64.tar: 515M * xz -7e (current implementation): 96M, ~100s * zstd -3 (default setting): 248M, ~1s * zstd -19: 112M, ~73s * zstd --ultra -20: 93M, ~97s * zstd --ultra -22: 78M, ~136s So it seems at --ultra -20 we have a sweet spot where we surpass xz -7e in both compression ratio and speed. --ultra just unlocks the highest compression levels >19, at the cost of addional memory use for both compression and decompression. All of the above measurements were made without multithreading. The multithreaded mode of xz (as used in OpenWrt master) makes compression worse, while zstd produces an idential file with and without multithreading: * xz -7e -T32: 113M, ~17s * zstd -19 -T32: 112M: ~12s * zstd --ultra -20 -T32: 93M, ~30s IMO the fact that xz compresses differently depending on the number of threads used is also a big problem, as it makes it harder to make builds fully reproducible. If we switch to zstd for SDK/IB, I would go for level -19: it's significantly faster than --ultra -20, especially in multithreaded mode, and the size difference seems acceptable to me (in fact, our current snapshot builds use xz in multithreaded mode, so the current IB tar.xz on the download server is 113M, not 96M. Matthias
diff --git a/include/kernel-build.mk b/include/kernel-build.mk index c371e78ab9b0..32c91a5b8359 100644 --- a/include/kernel-build.mk +++ b/include/kernel-build.mk @@ -70,7 +70,7 @@ ifdef CONFIG_COLLECT_KERNEL_DEBUG $(FIND) $(KERNEL_BUILD_DIR)/debug -type f | $(XARGS) $(KERNEL_CROSS)strip --only-keep-debug $(TAR) c -C $(KERNEL_BUILD_DIR) debug \ $(if $(SOURCE_DATE_EPOCH),--mtime="@$(SOURCE_DATE_EPOCH)") \ - | bzip2 -c -9 > $(BIN_DIR)/kernel-debug.tar.bz2 + | zstd -T0 -f -o $(BIN_DIR)/kernel-debug.tar.zst endef endif diff --git a/tools/Makefile b/tools/Makefile index 02556d4b83f3..169f36c5bb69 100644 --- a/tools/Makefile +++ b/tools/Makefile @@ -37,6 +37,7 @@ tools-y += lzma squashfskit4 zip tools-$(BUILD_B43_TOOLS) += b43-tools tools-$(BUILD_ISL) += isl tools-$(CONFIG_USE_SPARSE) += sparse +tools-$(CONFIG_COLLECT_KERNEL_DEBUG) += zstd tools-$(CONFIG_TARGET_apm821xx)$(CONFIG_TARGET_gemini) += genext2fs tools-$(CONFIG_TARGET_tegra) += cbootimage cbootimage-configs
zstd with its default settings (compression level -3) compresses better than bzip2 -9 (which is the default setting), and is an order of magnitude faster. I made the following measurements for the most common compression tools (all standard Debian Buster versions, default flags unless noted otherwise), using the debug information of a large x86-64 kernel with ALL_KMODS: * kernel-debug.tar: 376M * kernel-debug.tar.gz: 101M, compressed in ~12s * kernel-debug.tar.bz2: 91M, compressed in ~15s * kernel-debug.tar.xz: 57M, compressed in ~101s * kernel-debug.tar.zst: 86M, compressed in ~1s With zstd, there is still some room for improvement by increasing the compression, but the slight increase in compression ratio (22.83% -> 19.46%) does not justify the significant increase in compression time (about 5 times on my machine) in my opinion. Note that multithreaded compression (-T argument) does not affect reproducibility with zstd. Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net> --- include/kernel-build.mk | 2 +- tools/Makefile | 1 + 2 files changed, 2 insertions(+), 1 deletion(-)