From patchwork Mon Jan 7 22:05:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yann E. MORIN" X-Patchwork-Id: 1021617 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=free.fr Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ECQjkDnV"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 43YTx33Q8Yz9sDT for ; Tue, 8 Jan 2019 09:05:59 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 646B687ABF; Mon, 7 Jan 2019 22:05:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pwK1dXAI1pMm; Mon, 7 Jan 2019 22:05:52 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id CF2AA865AA; Mon, 7 Jan 2019 22:05:51 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 5F1561C296D for ; Mon, 7 Jan 2019 22:05:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 5F61C21544 for ; Mon, 7 Jan 2019 22:05:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rV9iaVrvzHyQ for ; Mon, 7 Jan 2019 22:05:49 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ed1-f53.google.com (mail-ed1-f53.google.com [209.85.208.53]) by silver.osuosl.org (Postfix) with ESMTPS id 4A80020762 for ; Mon, 7 Jan 2019 22:05:49 +0000 (UTC) Received: by mail-ed1-f53.google.com with SMTP id p6so2397054eds.0 for ; Mon, 07 Jan 2019 14:05:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:cc:subject:date:message-id:in-reply-to:references; bh=+kKTSsYKYh8a53LzhvNYwPDs2iD4hAg76Ctj5RuymlE=; b=ECQjkDnVx7agCmbHctaEwyMijS8tiy0zSneW9QqHGc4MXa/QAfsxpXQV0lH1y1kiWU VcconxXRsRnR7PfCrTc2spGH5deEPmddD8mMN7kLVHDmV3L7e+2VuCJh1aXqJvSt5dPS Lar1kbUREOlUc9688DfhG4FolNPDUJcchB0I7ZPoEHpua5gMoNugIJXOSKorbIP+0Jri rKDKAQvOwFFBCVgoZldCGVO2Hy8lGK5BziNSpCMMKip72o2t854Xe3aIhYNPALuulbmN sL2aYh7X74A3/TQPogMbQMnWdIGprM8Ofsxl7mNJLMKWDvsEUpIuUPQrvdC/1uORNwhv 1kfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:cc:subject:date:message-id :in-reply-to:references; bh=+kKTSsYKYh8a53LzhvNYwPDs2iD4hAg76Ctj5RuymlE=; b=XYEmJxldM8hNqQtVu1cKVxJyoKX3tofaMTI1jfEaERHtvPV/TtDhUgiP/sB2ogkpmS 3Ob8uzINgaHpFoYnejfpTN5zfeBC3MMoPApSGfXb1zPw/KOD7cpxX0/122zXAGU3Vifq 9h8mU9H+LvO7+fzqE3Kb639J18A+B7jYqybm3fT+Jx6j1pq/BSPJTXlxIpwAEnXv0+Nr u2VnA/jASpG7Rv5k8MeJHJQjEZF0NiiaoGVAfySQO7xGZ5UWEBFJlSloGdbbMUrBpg6a 8T11NWiGJElHdDUuLXY9ZKL2Zk4i+YcOXnGPBmz7CvZTbVcvY5pTLKhOlFCvvCbG/8GQ caPA== X-Gm-Message-State: AA+aEWYDospMMpGdmKUkh2JiqN/1geA6n/xXWi5d4MgDq66yZq2qfk1N lU9xNR7idCT9xU3xN/dpV6kDIXHN X-Google-Smtp-Source: AFSGD/UbAirUx16dSdo+hi3+Gz2kXkIr2vgISjezJG9G6Dze/w0JPMXlfT+Lu3KFXL/to5DpaaZ6ww== X-Received: by 2002:aa7:db0e:: with SMTP id t14mr57778790eds.292.1546898747262; Mon, 07 Jan 2019 14:05:47 -0800 (PST) Received: from scaer.home ([2a01:cb19:829a:2800:68e8:7a61:9bb9:12a]) by smtp.gmail.com with ESMTPSA id d56sm31799589ede.76.2019.01.07.14.05.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Jan 2019 14:05:46 -0800 (PST) From: "Yann E. MORIN" To: buildroot@buildroot.org Date: Mon, 7 Jan 2019 23:05:25 +0100 Message-Id: <61434bbb54154c9020084939907fff50b4539a21.1546898693.git.yann.morin.1998@free.fr> X-Mailer: git-send-email 2.14.1 In-Reply-To: References: Subject: [Buildroot] [PATCH 03/19] infra/pkg-generic: introduce new stampfile at the beginning of all steps X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , "Yann E. MORIN" MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Currently, we use the existing stampfiles as a standard make feature to organise the dependency tree between steps and packages, and so that a step that successfully finished is not re-done in a later build. Additionally, there is one stampfile that gained extra use in the recent past: since 7fb6e782542f (core/instrumentation: shave minutes off the build time) the built stampfile is now used as a reference to detect files installed by a package. However, this falls short during development, when a user may want to re-install a built-early package without rebuilding it (i.e. make foo-reinstall). In this case, the built stampfile is not touched, and is still dated from way back when the package was first built. As such, almost all files in target (or staging or host) are newer than that, and so those files are all now accounted for to that package, when in fact only a minor subset may be accountable to it. So, we need to introduce a new way to memorise the beginning of a step, so that we can properly find files installed by a package, even during development. To that goal, we introduce a new per-step stampfile, which gets touched early in the step, and which can now serve as a starting point in time for that step. This new stampfile is named after the existing stampfile, with an appended '_before' tag to it, and thus is always $@_before (note that, in Makefiles, variables are always 1-char, unless they are in parenthesis, so $@_before is exactly $(@)_before, but since we use $@ everywhere, we keep using it here, even though it can be slightly confusing). Reported-by: Nicolas Cavallari Signed-off-by: "Yann E. MORIN" Cc: Nicolas Cavallari Cc: Thomas Petazzoni --- package/pkg-generic.mk | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 6581ffa5e8..1b8febe8c5 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -141,6 +141,7 @@ $(BUILD_DIR)/%/.stamp_downloaded: fi ; \ done $(Q)mkdir -p $(@D) + $(Q)touch $@_before $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) $(foreach p,$($(PKG)_ALL_DOWNLOADS),$(call DOWNLOAD,$(p))$(sep)) $(foreach hook,$($(PKG)_POST_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) @@ -160,6 +161,7 @@ $(BUILD_DIR)/%/.stamp_extracted: @$(call step_start,extract) @$(call MESSAGE,"Extracting") $(Q)mkdir -p $(@D) + $(Q)touch $@_before $(foreach hook,$($(PKG)_PRE_EXTRACT_HOOKS),$(call $(hook))$(sep)) $($(PKG)_EXTRACT_CMDS) # some packages have messed up permissions inside @@ -174,6 +176,7 @@ $(BUILD_DIR)/%/.stamp_rsynced: @$(call step_start,rsync) @$(call MESSAGE,"Syncing from source dir $(SRCDIR)") @mkdir -p $(@D) + $(Q)touch $@_before $(foreach hook,$($(PKG)_PRE_RSYNC_HOOKS),$(call $(hook))$(sep)) @test -d $(SRCDIR) || (echo "ERROR: $(SRCDIR) does not exist" ; exit 1) rsync -au --chmod=u=rwX,go=rX $(RSYNC_VCS_EXCLUSIONS) $($(PKG)_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) $(call qstrip,$(SRCDIR))/ $(@D) @@ -193,6 +196,7 @@ $(BUILD_DIR)/%/.stamp_patched: PATCH_BASE_DIRS += $(addsuffix /$(RAWNAME),$(call $(BUILD_DIR)/%/.stamp_patched: @$(call step_start,patch) @$(call MESSAGE,"Patching") + $(Q)touch $@_before $(foreach hook,$($(PKG)_PRE_PATCH_HOOKS),$(call $(hook))$(sep)) $(foreach p,$($(PKG)_PATCH),$(APPLY_PATCHES) $(@D) $($(PKG)_DL_DIR) $(notdir $(p))$(sep)) $(Q)( \ @@ -219,6 +223,7 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\ $(BUILD_DIR)/%/.stamp_configured: @$(call step_start,configure) @$(call MESSAGE,"Configuring") + $(Q)touch $@_before $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) @@ -229,6 +234,7 @@ $(BUILD_DIR)/%/.stamp_configured: $(BUILD_DIR)/%/.stamp_built:: @$(call step_start,build) @$(call MESSAGE,"Building") + $(Q)touch $@_before $(foreach hook,$($(PKG)_PRE_BUILD_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_BUILD_CMDS) $(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep)) @@ -239,6 +245,7 @@ $(BUILD_DIR)/%/.stamp_built:: $(BUILD_DIR)/%/.stamp_host_installed: @$(call step_start,install-host) @$(call MESSAGE,"Installing to host directory") + $(Q)touch $@_before @mkdir -p $(HOST_DIR) $(foreach hook,$($(PKG)_PRE_INSTALL_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_CMDS) @@ -269,6 +276,7 @@ $(BUILD_DIR)/%/.stamp_host_installed: $(BUILD_DIR)/%/.stamp_staging_installed: @$(call step_start,install-staging) @$(call MESSAGE,"Installing to staging directory") + $(Q)touch $@_before @mkdir -p $(STAGING_DIR) $(foreach hook,$($(PKG)_PRE_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_STAGING_CMDS) @@ -303,6 +311,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed: $(BUILD_DIR)/%/.stamp_images_installed: @$(call step_start,install-image) @$(call MESSAGE,"Installing to images directory") + $(Q)touch $@_before @mkdir -p $(BINARIES_DIR) $(foreach hook,$($(PKG)_PRE_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_IMAGES_CMDS) @@ -314,6 +323,7 @@ $(BUILD_DIR)/%/.stamp_images_installed: $(BUILD_DIR)/%/.stamp_target_installed: @$(call step_start,install-target) @$(call MESSAGE,"Installing to target") + $(Q)touch $@_before @mkdir -p $(TARGET_DIR) $(foreach hook,$($(PKG)_PRE_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep)) +$($(PKG)_INSTALL_TARGET_CMDS)