From patchwork Tue Sep 20 19:46:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1680195 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=buildroot.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=buildroot-bounces@buildroot.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MXBrd6QV1z1yp7 for ; Wed, 21 Sep 2022 05:46:57 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 344DF40C83; Tue, 20 Sep 2022 19:46:55 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 344DF40C83 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8vp1v1Y4Tpwf; Tue, 20 Sep 2022 19:46:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp2.osuosl.org (Postfix) with ESMTP id 1DF8A40C79; Tue, 20 Sep 2022 19:46:53 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 1DF8A40C79 X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 04DF71BF616 for ; Tue, 20 Sep 2022 19:46:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id D3DE660E32 for ; Tue, 20 Sep 2022 19:46:50 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org D3DE660E32 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dNM0Sgb2o47a for ; Tue, 20 Sep 2022 19:46:50 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org D4EBE60DC6 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by smtp3.osuosl.org (Postfix) with ESMTPS id D4EBE60DC6 for ; Tue, 20 Sep 2022 19:46:49 +0000 (UTC) Received: by mail-wr1-x42b.google.com with SMTP id y5so6281930wrh.3 for ; Tue, 20 Sep 2022 12:46:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:sender:x-gm-message-state:from:to:cc:subject:date; bh=XnVpUKlCpQIeS2EfcifMj3ctm5Z/XTO9r8AkuIzUKqo=; b=aBcf68PpmWgsx+/09xV4skUv0319iukuA19RssPe8pyLpA34WIVWDQa9aAmOeIS8jc a1yNekje1UDBHons2NThjyCAsMizVdmxYpxHCFebZl7q1sstCHtt97Yw1CGPGdqNy0A8 NRjYuqv9kcsy7ZnGw9qeRiPmyjVTC7RXY01TbcHwxcgQAx92L9CV15qiDrFpywWl57x3 M92eCZ/aXZKQXIy6cZ53n1Q1+qbhaTMI6J1A9U6xFslo+1frpk1XuzsrpdTALPwaDiiZ I20qOl1bPXXIlfd1ozqqu8LkITv5Gdumh0M7/d+kDgO+UwxLHBC3E2ejxRfSMdlcPLxQ O+5A== X-Gm-Message-State: ACrzQf0L2aBnETUQAvWdMl3cKOX7Ishj8B8D5GRjX38+Q/WVnmL88rrK 2dqIXuuZ7upPJsrSW2XU7B5kFaRwvvU= X-Google-Smtp-Source: AMsMyM5r+wFMuo1t92/lwXznpMs726oMDM7tGU0wlG5rupqEEtrOmOsYahK3sYZrYdL0nQV6L/RCeQ== X-Received: by 2002:a5d:64a9:0:b0:22a:c3c3:4943 with SMTP id m9-20020a5d64a9000000b0022ac3c34943mr15899393wrp.655.1663703207813; Tue, 20 Sep 2022 12:46:47 -0700 (PDT) Received: from scaer.home ([2a01:cb19:8b51:cb00:b795:8a2b:1e98:bedc]) by smtp.gmail.com with ESMTPSA id h5-20020a05600c414500b003a608d69a64sm692207wmm.21.2022.09.20.12.46.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Sep 2022 12:46:47 -0700 (PDT) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Tue, 20 Sep 2022 21:46:45 +0200 Message-Id: <20220920194645.670432-1-yann.morin.1998@free.fr> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:sender:from:to:cc:subject:date; bh=XnVpUKlCpQIeS2EfcifMj3ctm5Z/XTO9r8AkuIzUKqo=; b=Wv2DY6JIh8vJdrF98cXSrFMxQu7HP/dS9mn/vO8Gk57+KMapPo9opFoRJ3W2p0oCS6 C1z5DJAFZP5iNYi3QSoBQIKsZ3r14MMj/EphfHS4mKd0O9dy7qm6DVCmdoxtzclL1Qnt O3K1g4EZFCm28innVIEA9HejXxa67nB25FkBxzRbhx/BfXNWQ4UZU/tFt+RmbFAFOJSR rRLiJJ66FprJTkYpEKc91n+ipHNS4aU6G50OjZhRHnqArU7L2OmCZpHci7t/jlGpbdMY 90nkbt0nnReSPnluaMrsfZ2pqiIXgu90wGQDz7o6RFprsq+dR+jrspnsplPq+RjIodlr B73Q== X-Mailman-Original-Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key, unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Wv2DY6JI Subject: [Buildroot] [PATCH v3] Makefile: fix use of many br2-external trees X-BeenThere: buildroot@buildroot.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: David Lawson , "Yann E. MORIN" Errors-To: buildroot-bounces@buildroot.org Sender: "buildroot" The top level Makefile in buildroot has a recursive rule which causes the appearance of a hang as the number of directories in BR2_EXTERNAL increases. When the number of directories in BR2_EXTERNAL is small, the recursion occurs, but make detects the recursion and determines the target does not have to be remade. This allows make to progress. This is the failing rule: define percent_defconfig # Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig %_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile @$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \ $$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN) endef $(eval $(foreach d,$(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)),$(call percent_defconfig,$(d))$(sep))) The rule for %defconfig is created for each directory in BR2_EXTERNAL. When the rule is matched, the stem is 'defconfig_name'. The second prerequisite is expanded to $(1)/configs/defconfig_name_defconfig. The rule, and all of the other rules defined by this macro, are invoked again, but the stem is now $(1)/configs/defconfig_name_defconfig. The second prerequisite is now expanded to $(1)/configs/($1)/configs/defconfig_name_defconfig. This expansion continues until make detects the infinite recursion. With up to 5 br2-external trees, the time is very small, so that it is not noticeable. But starting with 6 br2-external trees, the time is insanely big (so much so that we did not even let it finish after it ran for hours); see timings toward the end of the commit log. One of the rationale behind this code, is that we want the defconfig files from br2-external trees further down the list, to override defconfig files from those earlier in the list, even overriding the defconfig files from Buildroot itself. We fix that by only creating explicit rules for defconfig files. To keep the promise that later defconfig files override previous ones (which we do document in our manual), we need to memorise what defconfig file we already created a rule for, and only create a rule for the first-seen-in-reverse-order (aka the last one) defconfig. Since some people appear to be bold enough (or insane enough?) to use defconfig files that start with a dot, also handle those explictly. Fixes: #14996 Reported-by: David Lawson Signed-off-by: Yann E. MORIN ---- How to test many br2-external trees: $ for i in $(seq 1 1000); do mkdir -p br2-external-${i}/configs touch br2-external-${i}/{Config.in,external.mk} echo "name: BR_TEST_${i}" >br2-external-${i}/external.desc touch br2-external-${i}/configs/foo{,_${i}}_defconfig done $ time make V=1 \ BR2_EXTERNAL="$(for i in $(seq 1 N); do printf '%s ' "$(pwd)/foo²/br2-external-${i}"; done)" \ foo_1_defconfig Timings ('real' as reported by 'time'): N Before After 1 0.325s 0.328s 5 0.957s 0.358s 6 n/a 0.394s 10 n/a 0.432s 100 n/a 1.851s 1000 n/a 15.887s So, not only does that work for a large number of br2-external trees, it is also a little bit faster when using just a few. Tested-by: David Lawson --- Changes v2 -> v3: - fix order of comment documenting $1 and $2 (David) - don't use immediate assignment :=, use simple assignment - add timing information and a way to reproduce Changes v1 -> v2: - keep comment - fix typo --- Makefile | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index ec7c034ac1..9d4b1626ae 100644 --- a/Makefile +++ b/Makefile @@ -1010,13 +1010,27 @@ oldconfig syncconfig olddefconfig: $(BUILD_DIR)/buildroot-config/conf outputmake defconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile @$(COMMON_CONFIG_ENV) $< --defconfig$(if $(DEFCONFIG),=$(DEFCONFIG)) $(CONFIG_CONFIG_IN) -define percent_defconfig -# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the new defconfig -%_defconfig: $(BUILD_DIR)/buildroot-config/conf $(1)/configs/%_defconfig outputmakefile - @$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \ +ALL_DEFCONFIGS = +# $1: br2-external directory, without trailing /configs/ +# $2: defconfig name with trailing _defconfig +define defconfig_rule +ifeq ($$(filter $(2),$$(ALL_DEFCONFIGS)),) +# Override the BR2_DEFCONFIG from COMMON_CONFIG_ENV with the actual defconfig +$(2): $$(BUILD_DIR)/buildroot-config/conf outputmakefile + $$(Q)$$(COMMON_CONFIG_ENV) BR2_DEFCONFIG=$(1)/configs/$$@ \ $$< --defconfig=$(1)/configs/$$@ $$(CONFIG_CONFIG_IN) +ALL_DEFCONFIGS += $(2) +endif endef -$(eval $(foreach d,$(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)),$(call percent_defconfig,$(d))$(sep))) +$(eval \ + $(foreach d, \ + $(call reverse,$(TOPDIR) $(BR2_EXTERNAL_DIRS)), \ + $(foreach c, \ + $(wildcard $(d)/configs/*_defconfig $(d)/configs/.*_defconfig), \ + $(call defconfig_rule,$(d),$(notdir $(c)))$(sep) \ + ) \ + ) \ +) update-defconfig: savedefconfig