From patchwork Sun Jan 3 17:15:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gleb Mazovetskiy X-Patchwork-Id: 1421894 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=KoswO25T; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4D854X424jz9sVq for ; Mon, 4 Jan 2021 04:15:40 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4293D87014; Sun, 3 Jan 2021 17:15:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vJcmqKP-Ep0L; Sun, 3 Jan 2021 17:15:33 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 04DA587011; Sun, 3 Jan 2021 17:15:33 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id C6EC81BF337 for ; Sun, 3 Jan 2021 17:15:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id C2D0185A1D for ; Sun, 3 Jan 2021 17:15:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ki47e68RZEIg for ; Sun, 3 Jan 2021 17:15:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by whitealder.osuosl.org (Postfix) with ESMTPS id 97E86858AE for ; Sun, 3 Jan 2021 17:15:30 +0000 (UTC) Received: by mail-wm1-f54.google.com with SMTP id g185so15707655wmf.3 for ; Sun, 03 Jan 2021 09:15:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=B1CMbQJWSU5dG8itRp4b+fqYtMD3dJH+Y9Bz2t6N+Dw=; b=KoswO25THkXTrlZLr1WUHNHPp9GzYAUip7pryoaWqBeHpuqoCHJLNmU9zLZU98COeU jWD8wZZfIkrawcTUbXv7pSTkoIuT4dUlPcMTjEYues+GXnjomJr950QkDD741xlXR0Wm g2aVM0NQbZTFA57UFIxWwtVm1gg6+nFo8K05/A/eZqLYRPlqwhY7MbhfSS/Mb7qc6wUK wBH4Vwag8TJUHSTNFl//cASaESkIreps3enJtmPoRfpqYKHBxc/eiIqAFyufVrUpDR+6 59txBdwS6sPwS5S+AKnQIhjgA4Rn0sFtzTQXlV5lr9Nt6jLMG/J4tQYXISH/HkX0KWbp yXTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=B1CMbQJWSU5dG8itRp4b+fqYtMD3dJH+Y9Bz2t6N+Dw=; b=QT+rY6fU46pmYaMigWjwl0Kn3wWQPFfc1MBePNrnpJ+niyO2tFN/fxXClgBS0O/aKx UbtN5su0at6x5TTh17ATDlaLFCrDU35TT+KG77wTYeKwX6MRxSismPCGq7/BOYKOzJV8 9lPTP0wjcYcdceXxzFmrrCyRp0Galx1b5YubG1VKNCvprdsc4BrYBrGIX1p87Hco8kXW XhMgVXxvvmPft81V4s8o0TbMQZKot/wPcIIcQQ6jSyWkbHx1FJ/lmN6IvIQ62hfoyxFd H1E+niV4jmoOS/xsanL6cCgXgOe35ePW1IXU4wdQ6mJoHrfxSwCi5K7A2qWuZYuNQIhv V5YQ== X-Gm-Message-State: AOAM530Vy0pli4AJFyQ2I6W4Rpll4Akqs7k7/Rqxi50u/ta9xzHzzRCC vWgbqI1BPsd0YMGT4zQ87pOyGjOmucNYQQ== X-Google-Smtp-Source: ABdhPJyixc9vGzBFeDKVxaR598UU2+BIiBdBUQbx4KJPK9+fcbOzMcMd0oHokKqDeaVhxpNgphQVzg== X-Received: by 2002:a1c:68d6:: with SMTP id d205mr23855895wmc.154.1609694128829; Sun, 03 Jan 2021 09:15:28 -0800 (PST) Received: from localhost (cpc76906-dals22-2-0-cust400.20-2.cable.virginm.net. [81.106.81.145]) by smtp.gmail.com with ESMTPSA id h16sm30272772wmb.41.2021.01.03.09.15.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Sun, 03 Jan 2021 09:15:28 -0800 (PST) From: Gleb Mazovetskiy To: buildroot@buildroot.org Date: Sun, 3 Jan 2021 17:15:25 +0000 Message-Id: <20210103171525.2323323-1-glex.spb@gmail.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Subject: [Buildroot] [PATCH v5 1/1] Makefile: Parallelize glibc locale generation X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gleb Mazovetskiy Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Parallelizes locale generation based on `BR2_JLEVEL` setting. Locale generation always runs during the finalize stage and can consume a significant amount of time. Parallelizing it greatly reduces that time on multi-core machines. To parallelize it, we first invoke `localedef` for every locale in parallel with the `--no-archive` option. This creates the intermediate locale data instead of writing to the finally archive directly. Then, we invoke `localedef` again once to create the archive from the intermediate compiled locale data files. We have to do it this way because `localedef` does not do any locking when writing to the archive file, so calling it without `--no-archive` concurrently could result in a corrupt archive file or an archive file that is missing some locales. Signed-off-by: Gleb Mazovetskiy --- Makefile | 29 +++++++--------------- support/misc/gen-glibc-locales.mk | 41 +++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 20 deletions(-) create mode 100644 support/misc/gen-glibc-locales.mk diff --git a/Makefile b/Makefile index 4d334adcd6..dbeabf9e61 100644 --- a/Makefile +++ b/Makefile @@ -661,32 +661,21 @@ endef TARGET_FINALIZE_HOOKS += TOOLCHAIN_ECLIPSE_REGISTER endif -# Generate locale data. Basically, we call the localedef program -# (built by the host-localedef package) for each locale. The input -# data comes preferably from the toolchain, or if the toolchain does -# not have them (Linaro toolchains), we use the ones available on the -# host machine. +# Generate locale data. ifeq ($(BR2_TOOLCHAIN_USES_GLIBC),y) GLIBC_GENERATE_LOCALES = $(call qstrip,$(BR2_GENERATE_LOCALE)) ifneq ($(GLIBC_GENERATE_LOCALES),) PACKAGES += host-localedef define GENERATE_GLIBC_LOCALES - $(Q)mkdir -p $(TARGET_DIR)/usr/lib/locale/ - $(Q)for locale in $(GLIBC_GENERATE_LOCALES) ; do \ - inputfile=`echo $${locale} | cut -f1 -d'.'` ; \ - charmap=`echo $${locale} | cut -f2 -d'.' -s` ; \ - if test -z "$${charmap}" ; then \ - charmap="UTF-8" ; \ - fi ; \ - echo "Generating locale $${inputfile}.$${charmap}" ; \ - I18NPATH=$(STAGING_DIR)/usr/share/i18n:/usr/share/i18n \ - $(HOST_DIR)/bin/localedef \ - --prefix=$(TARGET_DIR) \ - --$(call LOWERCASE,$(BR2_ENDIAN))-endian \ - -i $${inputfile} -f $${charmap} \ - $${locale} ; \ - done + $(TARGET_CONFIGURE_OPTS) \ + $(MAKE) -j$(PARALLEL_JOBS) -f support/misc/gen-glibc-locales.mk \ + HOST_DIR=$(HOST_DIR) \ + TARGET_DIR=$(TARGET_DIR) \ + STAGING_DIR=$(STAGING_DIR) \ + ENDIAN=$(call LOWERCASE,$(BR2_ENDIAN)) \ + LOCALES="$(GLIBC_GENERATE_LOCALES)" \ + Q=$(Q) endef TARGET_FINALIZE_HOOKS += GENERATE_GLIBC_LOCALES endif diff --git a/support/misc/gen-glibc-locales.mk b/support/misc/gen-glibc-locales.mk new file mode 100644 index 0000000000..3db1e0dc11 --- /dev/null +++ b/support/misc/gen-glibc-locales.mk @@ -0,0 +1,41 @@ +# Generates glibc locale data for target. + +inputfile = $(firstword $(subst ., ,$(1))) +charmap = $(or $(word 2,$(subst ., ,$(1))),UTF-8) + +# Packages all the generated locale data into the final archive. +# +# We sort the file names to produce consistent output regardless of +# the `find` outputs order. +$(TARGET_DIR)/usr/lib/locale/locale-archive: $(LOCALES) + $(Q)rm -f $(@) + $(Q)find $(TARGET_DIR)/usr/lib/locale/ -maxdepth 1 -mindepth 1 -type d -print0 \ + | sort -z \ + | xargs -0 \ + $(HOST_DIR)/bin/localedef \ + --prefix=$(TARGET_DIR) \ + --$(ENDIAN)-endian \ + --add-to-archive + +# Generates locale data for each locale. +# +# The input data comes preferably from the toolchain, or if the toolchain +# does not have them (Linaro toolchains), we use the ones available on the +# host machine. +# +# Uses `localedef`, which is built by the `host-localedef` package. +$(LOCALES): | $(TARGET_DIR)/usr/lib/locale/ + $(Q)echo "Generating locale $(@)" + $(Q)I18NPATH=$(STAGING_DIR)/usr/share/i18n:/usr/share/i18n \ + $(HOST_DIR)/bin/localedef \ + --prefix=$(TARGET_DIR) \ + --$(ENDIAN)-endian \ + --no-archive \ + -i $(call inputfile,$(@)) \ + -f $(call charmap,$(@)) \ + $(@) + +.PHONY: $(LOCALES) + +$(TARGET_DIR)/usr/lib/locale/: + $(Q)mkdir -p $(TARGET_DIR)/usr/lib/locale/