From patchwork Fri Dec 4 09:57:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1410914 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=bxN7fzPh; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CnSmn5ngcz9sWq for ; Fri, 4 Dec 2020 20:57:29 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 2246988277; Fri, 4 Dec 2020 09:57:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id apzjIITEXbcO; Fri, 4 Dec 2020 09:57:21 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 23DFE88268; Fri, 4 Dec 2020 09:57:21 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 0B0821BF3A3 for ; Fri, 4 Dec 2020 09:57:19 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 0444888263 for ; Fri, 4 Dec 2020 09:57:19 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SHUnvdqGWjd0 for ; Fri, 4 Dec 2020 09:57:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by whitealder.osuosl.org (Postfix) with ESMTPS id 5B1C788268 for ; Fri, 4 Dec 2020 09:57:18 +0000 (UTC) Received: by mail-pl1-f172.google.com with SMTP id s2so2812140plr.9 for ; Fri, 04 Dec 2020 01:57:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=+NTFTsyZ6Fu4shjumeBW8CWmsF4rqk+fe6N0v7oroMI=; b=bxN7fzPhbrj9wanAoGenW0kg4EDiVLBK9mUR7VbKxWGhkfzLJGo6Jjm3NmujtFO6LP ikZP9fV8PMCtW3wj2N7IQ5taY+KHnlizu0xTOQLeOKfOgJlqdVdEGrNPKzNTVRHrD9fX USrwbZyGY4zALHatF2DCJ13F88lwi7TzkGScYiMw4Wl48CUKY8s2v23vMUhtchXm7Tmm FfUDT3hTMH4lpTwhKIUYjtPcuigEF5rYb3CIdZInF3JXlkWcvkPqHp0i5NfdIv2PQNnH Ull9H48Y87iafLdlfJ9fJT25dz8Qfk1SoNGZAG+iMhzNwa/V5EvKGc3iYrIVS0IIY0IO nFDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=+NTFTsyZ6Fu4shjumeBW8CWmsF4rqk+fe6N0v7oroMI=; b=KjlhZHrcZ+T4TlcXHzFubLiHch4oZon2S6U0BRM1tWOgpx15d+2dKMZp2sT9BYwMqO 7TiJvgjSGyDyghCnzWCJBbzAIM0kBz9AEGpoAEICsQjdB4q26Cjd49vlKYJxlxKtwZ3Y N8vyFU04NIG9YD5R+eM/f2PSKxMQGV9wAjBc+CfQFGjFW1FcSl5dKFXdply89kJneh62 dT1E8WLPBY4AzqL2NohFnB2+6U0CP5PJNjQiK0/aNKXPjCw3a7wc09RsLSjcqB9bB4WT ULV8Xk4dzUYBfxeako2fXFF8vVre8rCSlS3zFYKv8rBvoE7/f76IZudK9j5GkNvPpBrJ zjfA== X-Gm-Message-State: AOAM530G5tECf2GucI2H2XW76s4QSq0r1yOQW5i8y2T+8N993csXfLSH +ZKmb4vTyvbEuC0LqNCx9bjPvGC0cul36g== X-Google-Smtp-Source: ABdhPJz4mThfoq1IxgDr05yQRlPMW3NygrdQfb8ASw2qeqCcDx0kfoSPO9G0HcU5eCFh8s0DqvJp2A== X-Received: by 2002:a17:90a:202:: with SMTP id c2mr3379836pjc.164.1607075837711; Fri, 04 Dec 2020 01:57:17 -0800 (PST) Received: from localhost (ptr-5gw9txf6g5tndjqqqz.18120a2.ip6.access.telenet.be. [2a02:1810:510:3800:1b15:18e4:57d1:eb]) by smtp.gmail.com with ESMTPSA id h12sm3451503pgf.49.2020.12.04.01.57.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Dec 2020 01:57:17 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Fri, 4 Dec 2020 10:57:01 +0100 Message-Id: <20201204095703.4714-1-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 Subject: [Buildroot] [PATCHv2 1/2] package/zstd: build multithreaded library if supported X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrey Smirnov , yann.morin.1998@free.fr, Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire libzstd.so is built without multi-threading support by default. The 'HAVE_THREAD' flag is not respected by lib/Makefile, only by programs/Makefile. Use the %-mt recipe in lib/Makefile to enable multithreading. Signed-off-by: Thomas De Schampheleire --- v2: split off of next patch package/zstd/zstd.mk | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk index 35002da332..37c1a8d687 100644 --- a/package/zstd/zstd.mk +++ b/package/zstd/zstd.mk @@ -48,6 +48,14 @@ ZSTD_BUILD_LIBS = libzstd.a libzstd ZSTD_INSTALL_LIBS = install-static install-shared endif +# The HAVE_THREAD flag is read by the 'programs' makefile but not by the 'lib' +# one. Building a multi-threaded binary with a library (which defaults to +# single-threaded) gives a runtime error when compressing files. +# The 'lib' makefile provides specific '%-mt' targets for this purpose. +ifeq ($(BR2_TOOLCHAIN_HAS_THREADS),y) +ZSTD_BUILD_LIBS := $(addsuffix -mt,$(ZSTD_BUILD_LIBS)) +endif + define ZSTD_BUILD_CMDS $(TARGET_MAKE_ENV) $(TARGET_CONFIGURE_OPTS) $(MAKE) $(ZSTD_OPTS) \ -C $(@D)/lib $(ZSTD_BUILD_LIBS) From patchwork Fri Dec 4 09:57:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas De Schampheleire X-Patchwork-Id: 1410915 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=GwgaoznI; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CnSms38Gbz9sWq for ; Fri, 4 Dec 2020 20:57:32 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id E9CA287C28; Fri, 4 Dec 2020 09:57:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QMg2fKL5cDBr; Fri, 4 Dec 2020 09:57:30 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 2C08887BD3; Fri, 4 Dec 2020 09:57:30 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 163001BF9C1 for ; Fri, 4 Dec 2020 09:57:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 1256087BD3 for ; Fri, 4 Dec 2020 09:57:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Wus3H4AtKoIh for ; Fri, 4 Dec 2020 09:57:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by hemlock.osuosl.org (Postfix) with ESMTPS id 1A8BD87B08 for ; Fri, 4 Dec 2020 09:57:27 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id q22so3326705pfk.12 for ; Fri, 04 Dec 2020 01:57:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=jEzF7PT28t/PKPdQD9JPS73wlj1QfdpfDMJWdxE6JWc=; b=GwgaoznIg20R3kmx3FVFL7Uko1a9XQGfjLyYQV2l03ldqPAle6gpuWyU+abUC4iG1o Jsv3K9I9n63v2ys1XVycfhVk/GukIActRC6O5ph8DedO4NJCkd2tU6+e6eAWjvzV9S3q 6WE6WRUG3YZuW+ZdOir4jJj/kLuV/tMWy4KNK8i/IkicNaqymc4WZ7ziygFQ4hBbQ5QS bd/hWCS2I5+AWGpokVt0qMiC388D0pOoBE8iZdtdx5pgUdVYWXmWDubg+QTdk/9G6iFY 536hJ71DvTHx/2X/qASN89/3fFKs8RzlCFWMT+JwAheCQxmLEeRyo2fpecS2ZPwQUcWb hV2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=jEzF7PT28t/PKPdQD9JPS73wlj1QfdpfDMJWdxE6JWc=; b=VEYpbmPYWYNKrBFaKmV/REn1XmEMjO7FjIc1OmXjaoyIjb479N3/LO9lljZLC7M47+ 0BktFjTQrzmLhK22oNuAiZn8bkcKIybb2uOhbOKLcj6hbwg7O7YqSUvNz9pWur14jVJx /ftVoT9PuzTdRd3tzUz2psY+mPizuLR0fA5aWV2THigi19ZQLATO+0K4dC/EaTy1LTTW fltIIzDdl1Kw+9htCxIJHFNEPESYAao++9mE0tusMR/z8Ya/mM0YE69NF0FBynw62All H7AN6GzaMTcw99MvAsDHhPG/Ea4g+zrXvh0zKcB+vsePPUVkiIBViv2cSU1zrnrnBkMs hiTA== X-Gm-Message-State: AOAM530iLgnrf0bCR1d6cZR4/GsUd/EYpZdK/JL0fzqER2cU6AGGZ4L9 zyGnIToA1ClcxaPf8zVojsGhJgN5Gi6GEw== X-Google-Smtp-Source: ABdhPJwu8W6pIig+bxwDMCdPaXqIU4owxwgnDZ/UFOr5cTnZeoAR5crwkNSFokGIAKSiwod8DXgyAA== X-Received: by 2002:aa7:8c12:0:b029:18b:9939:9798 with SMTP id c18-20020aa78c120000b029018b99399798mr3189440pfd.44.1607075846200; Fri, 04 Dec 2020 01:57:26 -0800 (PST) Received: from localhost (ptr-5gw9txf6g5tndjqqqz.18120a2.ip6.access.telenet.be. [2a02:1810:510:3800:1b15:18e4:57d1:eb]) by smtp.gmail.com with ESMTPSA id l10sm1687864pjg.3.2020.12.04.01.57.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Dec 2020 01:57:25 -0800 (PST) From: Thomas De Schampheleire To: buildroot@buildroot.org Date: Fri, 4 Dec 2020 10:57:02 +0100 Message-Id: <20201204095703.4714-2-patrickdepinguin@gmail.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20201204095703.4714-1-patrickdepinguin@gmail.com> References: <20201204095703.4714-1-patrickdepinguin@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCHv2 2/2] package/zstd: link programs dynamically with libzstd to save space X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Andrey Smirnov , yann.morin.1998@free.fr, Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas De Schampheleire Even when a shared libzstd is built, the zstd command-line programs are statically linked with libzstd, causing a large rootfs footprint. While the cmake backend in zstd already supported a flag ZSTD_PROGRAMS_LINK_SHARED, the make backend did not. This commit adds support for ZSTD_PROGRAMS_LINK_SHARED in the make system and applies it for the target compilation, unless only static libs are supported. See https://github.com/facebook/zstd/issues/2261 . Signed-off-by: Thomas De Schampheleire --- v2: no functional changes other than splitting off part to a separate patch ...D_PROGRAMS_LINK_SHARED-to-link-zstd-.patch | 80 +++++++++++++++++++ package/zstd/zstd.mk | 2 + 2 files changed, 82 insertions(+) create mode 100644 package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch diff --git a/package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch b/package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch new file mode 100644 index 0000000000..100cfaba95 --- /dev/null +++ b/package/zstd/0002-make-support-ZSTD_PROGRAMS_LINK_SHARED-to-link-zstd-.patch @@ -0,0 +1,80 @@ +From afc368e7247abfe89250def5b7673a9ccb79e0b1 Mon Sep 17 00:00:00 2001 +From: Thomas De Schampheleire +Date: Wed, 5 Aug 2020 15:35:01 +0200 +Subject: [PATCH] make: support ZSTD_PROGRAMS_LINK_SHARED to link zstd with + libzstd + +zstd allows building via make, cmake, meson, and others, but unfortunately +the features and flags used for these build systems are inconsistent. + +The cmake version respects an option 'ZSTD_PROGRAMS_LINK_SHARED' to let the +zstd binary link dynamically with its own libzstd. Without it, the zstd +program uses static linking and thus has a big size. + +This option is not provided in the 'make' system. There does exist a target +'zstd-dll' which intends to do exactly this, but it does not work out of the +box due to linking issues. These in turn are caused because the lib makefile +passes '-fvisibility=hidden' to the linker, which hides certain symbols that +the zstd program needs. The cmake-based compilation does not pass this +visibility flag, due to which all symbols are visible. + +Unfortunately, there is no 'install' rule that will install zstd-dll and +create the derived programs like zstdless etc. + +With the above in mind, when ZSTD_PROGRAMS_LINK_SHARED is set in make +context: +- remove '-fvisibility=hidden' from the lib compilation +- alter the 'zstd' target to not include the lib objects directly but link + dynamically with libzstd. + +See also https://github.com/facebook/zstd/issues/2261 which reported these +inconsistencies between make and cmake compilation. + +Signed-off-by: Thomas De Schampheleire + +--- + lib/Makefile | 5 ++++- + programs/Makefile | 8 +++++++- + 2 files changed, 11 insertions(+), 2 deletions(-) + +diff --git a/lib/Makefile b/lib/Makefile +index 7c6dff02..8f9f36a6 100644 +--- a/lib/Makefile ++++ b/lib/Makefile +@@ -204,7 +204,10 @@ $(LIBZSTD): $(ZSTD_FILES) + else + + LIBZSTD = libzstd.$(SHARED_EXT_VER) +-$(LIBZSTD): LDFLAGS += -shared -fPIC -fvisibility=hidden ++$(LIBZSTD): LDFLAGS += -shared -fPIC ++ifndef ZSTD_PROGRAMS_LINK_SHARED ++$(LIBZSTD): LDFLAGS += -fvisibility=hidden ++endif + $(LIBZSTD): $(ZSTD_FILES) + @echo compiling dynamic library $(LIBVER) + $(Q)$(CC) $(FLAGS) $^ $(LDFLAGS) $(SONAME_FLAGS) -o $@ +diff --git a/programs/Makefile b/programs/Makefile +index 418ad4e6..8897dcf9 100644 +--- a/programs/Makefile ++++ b/programs/Makefile +@@ -169,10 +169,16 @@ $(ZSTDDECOMP_O): CFLAGS += $(ALIGN_LOOP) + zstd : CPPFLAGS += $(THREAD_CPP) $(ZLIBCPP) $(LZMACPP) $(LZ4CPP) + zstd : LDFLAGS += $(THREAD_LD) $(ZLIBLD) $(LZMALD) $(LZ4LD) $(DEBUGFLAGS_LD) + zstd : CPPFLAGS += -DZSTD_LEGACY_SUPPORT=$(ZSTD_LEGACY_SUPPORT) ++ifdef ZSTD_PROGRAMS_LINK_SHARED ++# link dynamically against own library ++zstd : LDFLAGS += -L$(ZSTDDIR) -lzstd ++else ++zstd : $(ZSTDLIB_FILES) ++endif + ifneq (,$(filter Windows%,$(OS))) + zstd : $(RES_FILE) + endif +-zstd : $(ZSTDLIB_FILES) $(ZSTD_CLI_OBJ) ++zstd : $(ZSTD_CLI_OBJ) + @echo "$(THREAD_MSG)" + @echo "$(ZLIB_MSG)" + @echo "$(LZMA_MSG)" +-- +2.26.2 + diff --git a/package/zstd/zstd.mk b/package/zstd/zstd.mk index 37c1a8d687..c7b224b002 100644 --- a/package/zstd/zstd.mk +++ b/package/zstd/zstd.mk @@ -43,9 +43,11 @@ ZSTD_INSTALL_LIBS = install-static else ifeq ($(BR2_SHARED_LIBS),y) ZSTD_BUILD_LIBS = libzstd ZSTD_INSTALL_LIBS = install-shared +ZSTD_OPTS += ZSTD_PROGRAMS_LINK_SHARED=1 else ZSTD_BUILD_LIBS = libzstd.a libzstd ZSTD_INSTALL_LIBS = install-static install-shared +ZSTD_OPTS += ZSTD_PROGRAMS_LINK_SHARED=1 endif # The HAVE_THREAD flag is read by the 'programs' makefile but not by the 'lib'