| Message ID | 20181114105557.12599-7-thomas.petazzoni@bootlin.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Per-package host/target directory support | expand |
Hi, Am 14.11.18 um 11:55 schrieb Thomas Petazzoni: ... > +# $1: deps > +ifeq ($(BR2_PER_PACKAGE_FOLDERS),y) > +define prepare-per-package-folder > + mkdir -p $(HOST_DIR) $(TARGET_DIR) > + $(foreach pkg,$(1),\ > + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/host/ \ > + $(PER_PACKAGE_DIR)/$(pkg)/host/ \ > + $(HOST_DIR)$(sep)) > + $(foreach pkg,$(1),\ > + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/target/ \ > + $(PER_PACKAGE_DIR)/$(pkg)/target/ \ > + $(TARGET_DIR)$(sep)) > +endef > +endif > + > ################################################################################ > # Implicit targets -- produce a stamp file for each step of a package build > ################################################################################ > @@ -133,6 +148,7 @@ endif > # Retrieve the archive > $(BUILD_DIR)/%/.stamp_downloaded: > @$(call step_start,download) > + $(call prepare-per-package-folder,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) > $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) > # Only show the download message if it isn't already downloaded > $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ > @@ -159,6 +175,7 @@ $(BUILD_DIR)/%/.stamp_actual_downloaded: > $(BUILD_DIR)/%/.stamp_extracted: > @$(call step_start,extract) > @$(call MESSAGE,"Extracting") > + $(call prepare-per-package-folder,$($(PKG)_FINAL_EXTRACT_DEPENDENCIES)) > $(foreach hook,$($(PKG)_PRE_EXTRACT_HOOKS),$(call $(hook))$(sep)) > $(Q)mkdir -p $(@D) > $($(PKG)_EXTRACT_CMDS) > @@ -219,6 +236,7 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\ > $(BUILD_DIR)/%/.stamp_configured: > @$(call step_start,configure) > @$(call MESSAGE,"Configuring") > + $(call prepare-per-package-folder,$($(PKG)_FINAL_DEPENDENCIES)) > $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) > $($(PKG)_CONFIGURE_CMDS) > $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) This is much clearer to me than v3, however there is a problem with kconfig packages. Unfortunately, the kconfig infrastructure is running the 'make xxx_defconfig' step before the configure-step is "officially" started. This is due to the kconfig_fixup target being a dependency of -configure. This in turn requires availability of the toolchain and other prerequisites that are part of _FINAL_DEPENDENCIES earlier as prepared in your above patch. I made a patch which moves the _FINAL_DEPENDENCIES preparation to an additional .stamp-configure-prepare step just before .stamp_configured. That works but is not particularly beautiful. I guess a more proper solution would be to somehow move the kconfig_fixup code into the configure-step. Maybe use the pre-configure-hook, any suggestions? regards, Andreas
Hello, Yann was already in Cc, I've added Thomas De Schampheleire as well. Yann, Thomas, there is some kconfig related discussion below, we need you :-) On Thu, 15 Nov 2018 17:41:35 +0100, Andreas Naumann wrote: > This is much clearer to me than v3, however there is a problem with > kconfig packages. Unfortunately, the kconfig infrastructure is running > the 'make xxx_defconfig' step before the configure-step is "officially" > started. This is due to the kconfig_fixup target being a dependency of > -configure. This in turn requires availability of the toolchain and > other prerequisites that are part of _FINAL_DEPENDENCIES earlier as > prepared in your above patch. > > I made a patch which moves the _FINAL_DEPENDENCIES preparation to an > additional .stamp-configure-prepare step just before .stamp_configured. > That works but is not particularly beautiful. That is not too bad actually. Semantically speaking, preparing the per-package folders is not really part of the configuration step. It could be logical to do it in a "prepare" step. The only thing that I dislike a bit with this is that it would no longer be consistent with what we do for download and extract dependencies: we prepare the per-package folders with download and extract dependencies respectively in the download and extract steps. So it would be quite logical to also do the same for the "configuration dependencies" (which we name just "dependencies" in Buildroot). So, this leaves us with 3 options: - Keep the dependency preparation within the download, extract and configure step, as proposed in this v4. This will require changing the kconfig logic to prepare the configuration file inside the "configure" step and not as a additional step injected before the "configure" step. - Keep the dependency preparation within the download and extract steps, and make an exception for the configure step, with a separate "prepare" step that comes before. Not nice in terms of consistency, as explained above. - Introduce "prepare download", "prepare extract" and "prepare configure" steps that would do the dependency preparation. > I guess a more proper solution would be to somehow move the > kconfig_fixup code into the configure-step. Maybe use the > pre-configure-hook, any suggestions? I don't recall why they need to be done before the configuration step, but I'm pretty sure there is a reason for that. Yann, Thomas, do you remember ? Thanks a lot, Thomas
On Fri, Nov 16, 2018, 14:47 Thomas Petazzoni <thomas.petazzoni@bootlin.com wrote: > Hello, > > Yann was already in Cc, I've added Thomas De Schampheleire as well. > Yann, Thomas, there is some kconfig related discussion below, we > need you :-) > > On Thu, 15 Nov 2018 17:41:35 +0100, Andreas Naumann wrote: > > > This is much clearer to me than v3, however there is a problem with > > kconfig packages. Unfortunately, the kconfig infrastructure is running > > the 'make xxx_defconfig' step before the configure-step is "officially" > > started. This is due to the kconfig_fixup target being a dependency of > > -configure. This in turn requires availability of the toolchain and > > other prerequisites that are part of _FINAL_DEPENDENCIES earlier as > > prepared in your above patch. > > > > I made a patch which moves the _FINAL_DEPENDENCIES preparation to an > > additional .stamp-configure-prepare step just before .stamp_configured. > > That works but is not particularly beautiful. > > That is not too bad actually. Semantically speaking, preparing the > per-package folders is not really part of the configuration step. It > could be logical to do it in a "prepare" step. > > The only thing that I dislike a bit with this is that it would no > longer be consistent with what we do for download and extract > dependencies: we prepare the per-package folders with download and > extract dependencies respectively in the download and extract steps. So > it would be quite logical to also do the same for the "configuration > dependencies" (which we name just "dependencies" in Buildroot). > > So, this leaves us with 3 options: > > - Keep the dependency preparation within the download, extract and > configure step, as proposed in this v4. This will require changing > the kconfig logic to prepare the configuration file inside the > "configure" step and not as a additional step injected before the > "configure" step. > > - Keep the dependency preparation within the download and extract > steps, and make an exception for the configure step, with a separate > "prepare" step that comes before. Not nice in terms of consistency, > as explained above. > > - Introduce "prepare download", "prepare extract" and "prepare > configure" steps that would do the dependency preparation. > > > I guess a more proper solution would be to somehow move the > > kconfig_fixup code into the configure-step. Maybe use the > > pre-configure-hook, any suggestions? > > I don't recall why they need to be done before the configuration step, > but I'm pretty sure there is a reason for that. > > Yann, Thomas, do you remember ? > My mind triggers a big red warning light right now, so let's be careful :-) The goal is that one can run 'make foo-menuconfig' from a clean tree, without first processing (downloading, building, ...) the dependencies of foo first. If you put this stuff in the configure step of foo, you are bound by its dependencies. There may be other reasons too, not sure. Best regards, Thomas <div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 16, 2018, 14:47 Thomas Petazzoni <<a href="mailto:thomas.petazzoni@bootlin.com">thomas.petazzoni@bootlin.com</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br> <br> Yann was already in Cc, I've added Thomas De Schampheleire as well.<br> Yann, Thomas, there is some kconfig related discussion below, we<br> need you :-)<br> <br> On Thu, 15 Nov 2018 17:41:35 +0100, Andreas Naumann wrote:<br> <br> > This is much clearer to me than v3, however there is a problem with <br> > kconfig packages. Unfortunately, the kconfig infrastructure is running <br> > the 'make xxx_defconfig' step before the configure-step is "officially" <br> > started. This is due to the kconfig_fixup target being a dependency of <br> > -configure. This in turn requires availability of the toolchain and <br> > other prerequisites that are part of _FINAL_DEPENDENCIES earlier as <br> > prepared in your above patch.<br> > <br> > I made a patch which moves the _FINAL_DEPENDENCIES preparation to an <br> > additional .stamp-configure-prepare step just before .stamp_configured. <br> > That works but is not particularly beautiful.<br> <br> That is not too bad actually. Semantically speaking, preparing the<br> per-package folders is not really part of the configuration step. It<br> could be logical to do it in a "prepare" step.<br> <br> The only thing that I dislike a bit with this is that it would no<br> longer be consistent with what we do for download and extract<br> dependencies: we prepare the per-package folders with download and<br> extract dependencies respectively in the download and extract steps. So<br> it would be quite logical to also do the same for the "configuration<br> dependencies" (which we name just "dependencies" in Buildroot).<br> <br> So, this leaves us with 3 options:<br> <br> - Keep the dependency preparation within the download, extract and<br> configure step, as proposed in this v4. This will require changing<br> the kconfig logic to prepare the configuration file inside the<br> "configure" step and not as a additional step injected before the<br> "configure" step.<br> <br> - Keep the dependency preparation within the download and extract<br> steps, and make an exception for the configure step, with a separate<br> "prepare" step that comes before. Not nice in terms of consistency,<br> as explained above.<br> <br> - Introduce "prepare download", "prepare extract" and "prepare<br> configure" steps that would do the dependency preparation.<br> <br> > I guess a more proper solution would be to somehow move the <br> > kconfig_fixup code into the configure-step. Maybe use the <br> > pre-configure-hook, any suggestions?<br> <br> I don't recall why they need to be done before the configuration step,<br> but I'm pretty sure there is a reason for that.<br> <br> Yann, Thomas, do you remember ?<br></blockquote></div></div><div dir="auto"><br></div><div dir="auto">My mind triggers a big red warning light right now, so let's be careful :-)</div><div dir="auto"><br></div><div dir="auto">The goal is that one can run 'make foo-menuconfig' from a clean tree, without first processing (downloading, building, ...) the dependencies of foo first.</div><div dir="auto">If you put this stuff in the configure step of foo, you are bound by its dependencies.</div><div dir="auto"><br></div><div dir="auto">There may be other reasons too, not sure.</div><div dir="auto"><br></div><div dir="auto">Best regards,</div><div dir="auto">Thomas</div></div>
Thomas², All, On 2018-11-16 16:22 +0100, Thomas De Schampheleire spake thusly: > On Fri, Nov 16, 2018, 14:47 Thomas Petazzoni < [1]thomas.petazzoni@bootlin.com wrote: > On Thu, 15 Nov 2018 17:41:35 +0100, Andreas Naumann wrote: [--SNIP--] > > I made a patch which moves the _FINAL_DEPENDENCIES preparation to an > > additional .stamp-configure-prepare step just before .stamp_configured. > > That works but is not particularly beautiful. > > That is not too bad actually. Semantically speaking, preparing the > per-package folders is not really part of the configuration step. It > could be logical to do it in a "prepare" step. And IIRC, you alreqady proposed such a step, specifically to be able to do the autopreconf for OOT building (which I am still working on, btw). Maybe this step can be re-used for various pacakges infras, like the kconfig one, to add preparation steps. > The only thing that I dislike a bit with this is that it would no > longer be consistent with what we do for download and extract > dependencies: we prepare the per-package folders with download and > extract dependencies respectively in the download and extract steps. So > it would be quite logical to also do the same for the "configuration > dependencies" (which we name just "dependencies" in Buildroot). > > So, this leaves us with 3 options: > > - Keep the dependency preparation within the download, extract and > configure step, as proposed in this v4. This will require changing > the kconfig logic to prepare the configuration file inside the > "configure" step and not as a additional step injected before the > "configure" step. > > - Keep the dependency preparation within the download and extract > steps, and make an exception for the configure step, with a separate > "prepare" step that comes before. Not nice in terms of consistency, > as explained above. > > - Introduce "prepare download", "prepare extract" and "prepare > configure" steps that would do the dependency preparation. > > > I guess a more proper solution would be to somehow move the > > kconfig_fixup code into the configure-step. Maybe use the > > pre-configure-hook, any suggestions? > > I don't recall why they need to be done before the configuration step, > but I'm pretty sure there is a reason for that. > > Yann, Thomas, do you remember ? > > My mind triggers a big red warning light right now, so let's be careful :-) Yes, we got bitten pretty hard when preparing the kconfig infra. But what can we remember from an almost-5-year old work initiated after FOSDEM? ;-) > The goal is that one can run 'make foo-menuconfig' from a clean tree, > without first processing (downloading, building, ...) the dependencies > of foo first. > If you put this stuff in the configure step of foo, you are bound byi > its dependencies. There may be other reasons too, not sure. I am all in favour of simplifying it if it can be made simpler by adding an official extra 'prepare' step, that exists for all types of package infras. This we'd have 4 levels of dependencies: 1- download dependencies 2- extract dependencies 3- prepare dependencies 4- configure dependencies Currently, 1, 2, and 4 are implemented in a generic way and thus available to all types of packages, while 3 is implemented only for kconfig-package, in an ad-hoc way, and used only by the linux kernel to ensure the toolchain is available before its kconfig is called. Regards, Yann E. MORIN.
On 16/11/2018 20:57, Yann E. MORIN wrote: > Thomas², All, > > On 2018-11-16 16:22 +0100, Thomas De Schampheleire spake thusly: >> On Fri, Nov 16, 2018, 14:47 Thomas Petazzoni < [1]thomas.petazzoni@bootlin.com wrote: >> On Thu, 15 Nov 2018 17:41:35 +0100, Andreas Naumann wrote: > [--SNIP--] >> > I made a patch which moves the _FINAL_DEPENDENCIES preparation to an >> > additional .stamp-configure-prepare step just before .stamp_configured. >> > That works but is not particularly beautiful. >> >> That is not too bad actually. Semantically speaking, preparing the >> per-package folders is not really part of the configuration step. It >> could be logical to do it in a "prepare" step. > > And IIRC, you alreqady proposed such a step, specifically to be able to > do the autopreconf for OOT building (which I am still working on, btw). > > Maybe this step can be re-used for various pacakges infras, like the > kconfig one, to add preparation steps. > >> The only thing that I dislike a bit with this is that it would no >> longer be consistent with what we do for download and extract >> dependencies: we prepare the per-package folders with download and >> extract dependencies respectively in the download and extract steps. So >> it would be quite logical to also do the same for the "configuration >> dependencies" (which we name just "dependencies" in Buildroot). >> >> So, this leaves us with 3 options: >> >> - Keep the dependency preparation within the download, extract and >> configure step, as proposed in this v4. This will require changing >> the kconfig logic to prepare the configuration file inside the >> "configure" step and not as a additional step injected before the >> "configure" step. >> >> - Keep the dependency preparation within the download and extract >> steps, and make an exception for the configure step, with a separate >> "prepare" step that comes before. Not nice in terms of consistency, >> as explained above. >> >> - Introduce "prepare download", "prepare extract" and "prepare >> configure" steps that would do the dependency preparation. >> >> > I guess a more proper solution would be to somehow move the >> > kconfig_fixup code into the configure-step. Maybe use the >> > pre-configure-hook, any suggestions? >> >> I don't recall why they need to be done before the configuration step, >> but I'm pretty sure there is a reason for that. >> >> Yann, Thomas, do you remember ? >> >> My mind triggers a big red warning light right now, so let's be careful :-) > > Yes, we got bitten pretty hard when preparing the kconfig infra. But > what can we remember from an almost-5-year old work initiated after > FOSDEM? ;-) > >> The goal is that one can run 'make foo-menuconfig' from a clean tree, >> without first processing (downloading, building, ...) the dependencies >> of foo first. >> If you put this stuff in the configure step of foo, you are bound byi >> its dependencies. There may be other reasons too, not sure. > > I am all in favour of simplifying it if it can be made simpler by adding > an official extra 'prepare' step, that exists for all types of package > infras. This we'd have 4 levels of dependencies: > > 1- download dependencies > 2- extract dependencies > 3- prepare dependencies > 4- configure dependencies I think your mixing two things here. The "prepare" that ThomasP is talking about is really the PPD creation. The additional step for autoreconf or kconfig is something else - it's a step with commands that are defined by the package (infra). Regards, Arnout > > Currently, 1, 2, and 4 are implemented in a generic way and thus > available to all types of packages, while 3 is implemented only for > kconfig-package, in an ad-hoc way, and used only by the linux kernel to > ensure the toolchain is available before its kconfig is called. > > Regards, > Yann E. MORIN. >
Hello, On Sun, 18 Nov 2018 22:55:26 +0100, Arnout Vandecappelle wrote: > > Yes, we got bitten pretty hard when preparing the kconfig infra. But > > what can we remember from an almost-5-year old work initiated after > > FOSDEM? ;-) > > > >> The goal is that one can run 'make foo-menuconfig' from a clean tree, > >> without first processing (downloading, building, ...) the dependencies > >> of foo first. > >> If you put this stuff in the configure step of foo, you are bound byi > >> its dependencies. There may be other reasons too, not sure. > > > > I am all in favour of simplifying it if it can be made simpler by adding > > an official extra 'prepare' step, that exists for all types of package > > infras. This we'd have 4 levels of dependencies: > > > > 1- download dependencies > > 2- extract dependencies > > 3- prepare dependencies > > 4- configure dependencies > > I think your mixing two things here. The "prepare" that ThomasP is talking > about is really the PPD creation. The additional step for autoreconf or kconfig > is something else - it's a step with commands that are defined by the package > (infra). I don't think Yann is mixing things here. Indeed we are using the word "prepare" for two different things: preparing the PPD folder, and "doing some stuff before the configure step". For the PPD folder, some preparation currently occurs at the download step (for download dependencies), some preparation occurs at the extract step (for the extract dependencies) and some preparation occurs at the configuration step (for the normal dependencies). The issue with the kconfig infrastructure is that it creates a new type of dependency called "kconfig dependency", which should be ready to do all the kconfig-related activities, but that we want separate from the normal "configure dependencies" because we want to be able to do "make linux-menuconfig" or "make busybox-menuconfig" without having to wait for their dependencies to build. So what Yann is proposing is that instead of having this "kconfig dependency" stuff be shoe-horned into the dependency chain by pkg-kconfig.mk, we should promote it to a new step in the build steps. By the lack of better naming, Yann named that step "prepare", but it has nothing to do with "preparing PPD". Indeed, what we would do is: 1. Download step Prepare PPD with download dependencies Do the normal download stuff 2. Extract step Prepare PPD with extract dependencies Do the normal extract stuff 3. Patch step Prepare PPD with patch dependencies Do the normal patch stuff 4. Prepare step Prepare PPD with the prepare dependencies Do the prepare stuff (i.e only the kconfig stuff for now) 5. Configure step Prepare PPD with the normal dependencies Do the normal configure stuff Does that clarify Yann's proposal ? I personally find it OK, even though it's a bit annoying to introduce yet another step just for the sake of pkg-kconfig. Also: I would not move autoreconf into this prepare step, but keep it as it is today, within the configure step. I don't (today at least) see the point/usefulness of moving the autoreconf into this prepare step. Best regards, Thomas
Hi, Am 19.11.18 um 11:48 schrieb Thomas Petazzoni: > Hello, > > On Sun, 18 Nov 2018 22:55:26 +0100, Arnout Vandecappelle wrote: > >>> Yes, we got bitten pretty hard when preparing the kconfig infra. But >>> what can we remember from an almost-5-year old work initiated after >>> FOSDEM? ;-) >>> >>>> The goal is that one can run 'make foo-menuconfig' from a clean tree, >>>> without first processing (downloading, building, ...) the dependencies >>>> of foo first. >>>> If you put this stuff in the configure step of foo, you are bound byi >>>> its dependencies. There may be other reasons too, not sure. >>> >>> I am all in favour of simplifying it if it can be made simpler by adding >>> an official extra 'prepare' step, that exists for all types of package >>> infras. This we'd have 4 levels of dependencies: >>> >>> 1- download dependencies >>> 2- extract dependencies >>> 3- prepare dependencies >>> 4- configure dependencies >> >> I think your mixing two things here. The "prepare" that ThomasP is talking >> about is really the PPD creation. The additional step for autoreconf or kconfig >> is something else - it's a step with commands that are defined by the package >> (infra). > > I don't think Yann is mixing things here. Indeed we are using the word > "prepare" for two different things: preparing the PPD folder, and > "doing some stuff before the configure step". > > For the PPD folder, some preparation currently occurs at the download > step (for download dependencies), some preparation occurs at the > extract step (for the extract dependencies) and some preparation occurs > at the configuration step (for the normal dependencies). > > The issue with the kconfig infrastructure is that it creates a new type > of dependency called "kconfig dependency", which should be ready to do > all the kconfig-related activities, but that we want separate from the > normal "configure dependencies" because we want to be able to do "make > linux-menuconfig" or "make busybox-menuconfig" without having to wait > for their dependencies to build. > > So what Yann is proposing is that instead of having this "kconfig > dependency" stuff be shoe-horned into the dependency chain by > pkg-kconfig.mk, we should promote it to a new step in the build steps. > By the lack of better naming, Yann named that step "prepare", but it > has nothing to do with "preparing PPD". > > Indeed, what we would do is: > > 1. Download step > Prepare PPD with download dependencies > Do the normal download stuff > > 2. Extract step > Prepare PPD with extract dependencies > Do the normal extract stuff > > 3. Patch step > Prepare PPD with patch dependencies > Do the normal patch stuff > > 4. Prepare step > Prepare PPD with the prepare dependencies > Do the prepare stuff (i.e only the kconfig stuff for now) > > 5. Configure step > Prepare PPD with the normal dependencies > Do the normal configure stuff > > Does that clarify Yann's proposal ? > > I personally find it OK, even though it's a bit annoying to introduce > yet another step just for the sake of pkg-kconfig. > > Also: I would not move autoreconf into this prepare step, but keep it > as it is today, within the configure step. I don't (today at least) see > the point/usefulness of moving the autoreconf into this prepare step. This is basically what i have done, but without explicitely introducing separate kconfig-dependencies, I hope this is readable: From cf583d9ed09c667f24f526edf890d81b374e1d83 Mon Sep 17 00:00:00 2001 From: Andreas Naumann <anaumann@ultratronik.de> Date: Wed, 14 Nov 2018 17:27:31 +0100 Subject: [PATCH v2 1/1] core: Move per-package host preparation into separate step Unfortunately, the kconfig infrastructure is running the 'make xxx_defconfig' step before the configure-step is "officially" started. This is due to the kconfig_fixup target. This however requires availability of the toolchain and other prerequisites, so this patch introduces an intermediate target to prepare the per-package host dir. Signed-off-by: Andreas Naumann <anaumann@ultratronik.de> --- package/pkg-generic.mk | 19 +++++++++++++++---- package/pkg-kconfig.mk | 2 +- 2 files changed, 16 insertions(+), 5 deletions(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 8ec5e8db73..3014c13846 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -232,11 +232,18 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\ $(if $(wildcard $(dir)),,\ $(error BR2_GLOBAL_PATCH_DIR contains nonexistent directory $(dir)))) +# Prepare per-package host dir for configure +$(BUILD_DIR)/%/.stamp_configure_prepared: + @$(call step_start,configure_prepared) + @$(call MESSAGE,"Prepare per-package host") + $(call prepare-per-package-folder,$($(PKG)_FINAL_DEPENDENCIES)) + @$(call step_end,configure_prepared) + $(Q)touch $@ + # Configure $(BUILD_DIR)/%/.stamp_configured: @$(call step_start,configure) @$(call MESSAGE,"Configuring") - $(call prepare-per-package-folder,$($(PKG)_FINAL_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) @@ -663,6 +670,7 @@ $(2)_TARGET_INSTALL_STAGING = $$($(2)_DIR)/.stamp_staging_installed $(2)_TARGET_INSTALL_IMAGES = $$($(2)_DIR)/.stamp_images_installed $(2)_TARGET_INSTALL_HOST = $$($(2)_DIR)/.stamp_host_installed $(2)_TARGET_BUILD = $$($(2)_DIR)/.stamp_built +$(2)_TARGET_CONFIGURE_PREP = $$($(2)_DIR)/.stamp_configure_prepared $(2)_TARGET_CONFIGURE = $$($(2)_DIR)/.stamp_configured $(2)_TARGET_RSYNC = $$($(2)_DIR)/.stamp_rsynced $(2)_TARGET_PATCH = $$($(2)_DIR)/.stamp_patched @@ -756,7 +764,8 @@ $$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE) # dependency by using |. $(1)-configure: $$($(2)_TARGET_CONFIGURE) -$$($(2)_TARGET_CONFIGURE): | $$($(2)_FINAL_DEPENDENCIES) +$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_CONFIGURE_PREP) +$$($(2)_TARGET_CONFIGURE_PREP): | $$($(2)_FINAL_DEPENDENCIES) $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | prepare $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies @@ -768,7 +777,7 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),) # extract # patch # configure -$$($(2)_TARGET_CONFIGURE): $$($(2)_TARGET_PATCH) +$$($(2)_TARGET_CONFIGURE_PREP): $$($(2)_TARGET_PATCH) $(1)-patch: $$($(2)_TARGET_PATCH) $$($(2)_TARGET_PATCH): $$($(2)_TARGET_EXTRACT) @@ -807,7 +816,7 @@ else # Use an order-only dependency so the "<pkg>-clean-for-rebuild" rule # can remove the stamp file without triggering the configure step. -$$($(2)_TARGET_CONFIGURE): | $$($(2)_TARGET_RSYNC) +$$($(2)_TARGET_CONFIGURE_PREP): | $$($(2)_TARGET_RSYNC) $(1)-depends: $$($(2)_FINAL_DEPENDENCIES) @@ -878,6 +887,7 @@ $(1)-rebuild: $(1)-clean-for-rebuild $(1) $(1)-clean-for-reconfigure: $(1)-clean-for-rebuild rm -f $$($(2)_TARGET_CONFIGURE) + rm -f $$($(2)_TARGET_CONFIGURE_PREP) $(1)-reconfigure: $(1)-clean-for-reconfigure $(1) @@ -889,6 +899,7 @@ $$($(2)_TARGET_INSTALL_IMAGES): PKG=$(2) $$($(2)_TARGET_INSTALL_HOST): PKG=$(2) $$($(2)_TARGET_BUILD): PKG=$(2) $$($(2)_TARGET_CONFIGURE): PKG=$(2) +$$($(2)_TARGET_CONFIGURE_PREP): PKG=$(2) $$($(2)_TARGET_RSYNC): SRCDIR=$$($(2)_OVERRIDE_SRCDIR) $$($(2)_TARGET_RSYNC): PKG=$(2) $$($(2)_TARGET_PATCH): PKG=$(2) diff --git a/package/pkg-kconfig.mk b/package/pkg-kconfig.mk index d6c95b897e..5567df01d6 100644 --- a/package/pkg-kconfig.mk +++ b/package/pkg-kconfig.mk @@ -128,7 +128,7 @@ $$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $(1)-patch # Some packages may need additional tools to be present by the time their # kconfig structure is parsed (e.g. the linux kernel may need to call to # the compiler to test its features). -$$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): | $$($(2)_KCONFIG_DEPENDENCIES) +$$($(2)_DIR)/$$($(2)_KCONFIG_DOTCONFIG): $$($(2)_TARGET_CONFIGURE_PREP) | $$($(2)_KCONFIG_DEPENDENCIES) # In order to get a usable, consistent configuration, some fixup may be needed. # The exact rules are specified by the package .mk file.
Thomas, All, On 2018-11-19 11:48 +0100, Thomas Petazzoni spake thusly: > On Sun, 18 Nov 2018 22:55:26 +0100, Arnout Vandecappelle wrote: [--SNIP--] > > > I am all in favour of simplifying it if it can be made simpler by adding > > > an official extra 'prepare' step, that exists for all types of package > > > infras. This we'd have 4 levels of dependencies: > > > > > > 1- download dependencies > > > 2- extract dependencies > > > 3- prepare dependencies > > > 4- configure dependencies > > > > I think your mixing two things here. The "prepare" that ThomasP is talking > > about is really the PPD creation. The additional step for autoreconf or kconfig > > is something else - it's a step with commands that are defined by the package > > (infra). > > I don't think Yann is mixing things here. Indeed we are using the word > "prepare" for two different things: preparing the PPD folder, and > "doing some stuff before the configure step". Yeah, I was actually kinda mixing things here, in fact. ;-) > For the PPD folder, some preparation currently occurs at the download > step (for download dependencies), some preparation occurs at the > extract step (for the extract dependencies) and some preparation occurs > at the configuration step (for the normal dependencies). Thanks, I was missing that piece. PPD preparation is split into as many step as we have. > The issue with the kconfig infrastructure is that it creates a new type > of dependency called "kconfig dependency", which should be ready to do > all the kconfig-related activities, but that we want separate from the > normal "configure dependencies" because we want to be able to do "make > linux-menuconfig" or "make busybox-menuconfig" without having to wait > for their dependencies to build. Exactly. > So what Yann is proposing is that instead of having this "kconfig > dependency" stuff be shoe-horned into the dependency chain by > pkg-kconfig.mk, we should promote it to a new step in the build steps. Yes. > By the lack of better naming, Yann named that step "prepare", but it > has nothing to do with "preparing PPD". So, that's where I got it mixed up, indeed. > Indeed, what we would do is: > > 1. Download step > Prepare PPD with download dependencies > Do the normal download stuff > > 2. Extract step > Prepare PPD with extract dependencies > Do the normal extract stuff > > 3. Patch step > Prepare PPD with patch dependencies > Do the normal patch stuff > > 4. Prepare step > Prepare PPD with the prepare dependencies > Do the prepare stuff (i.e only the kconfig stuff for now) > > 5. Configure step > Prepare PPD with the normal dependencies > Do the normal configure stuff > > Does that clarify Yann's proposal ? Yes, that clarifies it perfectly. Thanks! :-) > I personally find it OK, even though it's a bit annoying to introduce > yet another step just for the sake of pkg-kconfig. > > Also: I would not move autoreconf into this prepare step, but keep it > as it is today, within the configure step. I don't (today at least) see > the point/usefulness of moving the autoreconf into this prepare step. For the PPD stuff? Definitely not. But it will be usefull to run autoreconf in a step that is not the configure step, so we are able to share the source tree between host and target build, as well as stop rsyncing from _OVERRIDE_SRCDIR. But then, I wonder if that is really the same as the 'prepare' step above, because autoreconf will also require the same dependencies as the configure step. Regards, Yann E. MORIN.
On 19/11/2018 20:49, Yann E. MORIN wrote: > Thomas, All, > > On 2018-11-19 11:48 +0100, Thomas Petazzoni spake thusly: [snip] >> Indeed, what we would do is: >> >> 1. Download step >> Prepare PPD with download dependencies >> Do the normal download stuff >> >> 2. Extract step >> Prepare PPD with extract dependencies >> Do the normal extract stuff >> >> 3. Patch step >> Prepare PPD with patch dependencies >> Do the normal patch stuff >> >> 4. Prepare step >> Prepare PPD with the prepare dependencies >> Do the prepare stuff (i.e only the kconfig stuff for now) >> >> 5. Configure step >> Prepare PPD with the normal dependencies >> Do the normal configure stuff >> >> Does that clarify Yann's proposal ? > > Yes, that clarifies it perfectly. Thanks! :-) > >> I personally find it OK, even though it's a bit annoying to introduce >> yet another step just for the sake of pkg-kconfig. Me too. And I'm not even sure if it really solves the pkg-kconfig problem. I have a hard time remembering what all the issues were there. Perhaps, though, we should have a more general strategy of a per-infra definition of what the steps are, instead of having a fixed sequence of steps. I.e., have a pattern that should be followed by an infra-specific step to make sure things are done correctly. So, for the kconfig case we would have an extra kconfig_fixup step (we already have the stamp file, but it's not a real step with hooks and the KCONFIG_DEPENDENCIES were strapped on as an afterthought), and for the autoreconf case we would have an extra autoreconf step (instead of a hook). >> Also: I would not move autoreconf into this prepare step, but keep it >> as it is today, within the configure step. I don't (today at least) see >> the point/usefulness of moving the autoreconf into this prepare step. > > For the PPD stuff? Definitely not. > > But it will be usefull to run autoreconf in a step that is not the > configure step, so we are able to share the source tree between host and > target build, as well as stop rsyncing from _OVERRIDE_SRCDIR. But the problem is, for pkg-kconfig, we actually need the prepare stuff to be done in the build directory, not the source directory... So it *still* doesn't match. > But then, I wonder if that is really the same as the 'prepare' step So no, it's not the same, but... > above, because autoreconf will also require the same dependencies as the > configure step. ... not because of this. Most dependencies are only really needed in the configure step, it's only some of them (host-automake etc. obviously, but also host-autoconf-archive and host-pkg-config for the .m4 files) that are needed for the autoreconf. So if it is split, we would need to move some dependencies to _PREPARE_DEPENDENCIES. Regards, Arnout
Hello, On Tue, 20 Nov 2018 11:22:00 +0100, Arnout Vandecappelle wrote: > >> I personally find it OK, even though it's a bit annoying to introduce > >> yet another step just for the sake of pkg-kconfig. > > Me too. And I'm not even sure if it really solves the pkg-kconfig problem. I > have a hard time remembering what all the issues were there. > > Perhaps, though, we should have a more general strategy of a per-infra > definition of what the steps are, instead of having a fixed sequence of steps. > I.e., have a pattern that should be followed by an infra-specific step to make > sure things are done correctly. So, for the kconfig case we would have an extra > kconfig_fixup step (we already have the stamp file, but it's not a real step > with hooks and the KCONFIG_DEPENDENCIES were strapped on as an afterthought), > and for the autoreconf case we would have an extra autoreconf step (instead of a > hook). And so those "injected" steps should take care of preparing the per-package folder with whatever dependencies they need ? This is probably something that can be done to fix the pkg-kconfig stuff without having to create a new "prepare" step in the generic infrastructure. > But the problem is, for pkg-kconfig, we actually need the prepare stuff to be > done in the build directory, not the source directory... So it *still* doesn't > match. Right. > > above, because autoreconf will also require the same dependencies as the > > configure step. > > ... not because of this. Most dependencies are only really needed in the > configure step, it's only some of them (host-automake etc. obviously, but also > host-autoconf-archive and host-pkg-config for the .m4 files) that are needed for > the autoreconf. So if it is split, we would need to move some dependencies to > _PREPARE_DEPENDENCIES. At this point, I don't want to change anything for autoreconf. It is part of the configure step, it works as it is. I'd like to fix just the pkg-kconfig situation. I'll try to do that within pkg-kconfig.mk. Best regards, Thomas
Hello, On Tue, 20 Nov 2018 11:29:00 +0100, Thomas Petazzoni wrote: > > Perhaps, though, we should have a more general strategy of a per-infra > > definition of what the steps are, instead of having a fixed sequence of steps. > > I.e., have a pattern that should be followed by an infra-specific step to make > > sure things are done correctly. So, for the kconfig case we would have an extra > > kconfig_fixup step (we already have the stamp file, but it's not a real step > > with hooks and the KCONFIG_DEPENDENCIES were strapped on as an afterthought), > > and for the autoreconf case we would have an extra autoreconf step (instead of a > > hook). > > And so those "injected" steps should take care of preparing the > per-package folder with whatever dependencies they need ? > > This is probably something that can be done to fix the pkg-kconfig > stuff without having to create a new "prepare" step in the generic > infrastructure. Just tried that and it works. It will be part of my v5, which I hope to send either later today, worst case tomorrow. With this, I can run "make linux-menuconfig" from a clean tree, and the KCONFIG_DEPENDENCIES are properly copied to the linux per-package folder. Best regards, Thomas
Hello, On Mon, 19 Nov 2018 11:48:06 +0100, Thomas Petazzoni wrote: > 3. Patch step > Prepare PPD with patch dependencies In fact no: this is *not* needed. Patch dependencies are special: $$($(2)_TARGET_PATCH): | $$(patsubst %,%-patch,$$($(2)_FINAL_PATCH_DEPENDENCIES)) Contrary to other dependencies where we depend on the "dependent" package to be fully built and installed, a patch dependency is a dependency between the patch step of package A, and the patch step of package B. Since we only depend on the patch step of the "dependent" package to be completed, and that up the patch step, packages are not allowed to install stuff to $(HOST_DIR), $(TARGET_DIR) or $(STAGING_DIR), there is nothing to do in terms of per-package folder preparation. Best regards, Thomas
diff --git a/Config.in b/Config.in index 03e4eb3928..69e34ec256 100644 --- a/Config.in +++ b/Config.in @@ -700,6 +700,24 @@ config BR2_REPRODUCIBLE This is labeled as an experimental feature, as not all packages behave properly to ensure reproducibility. +config BR2_PER_PACKAGE_FOLDERS + bool "Use per-package folders (experimental)" + help + This option will change the build process of Buildroot + package to use per-package target and host directories. + + This is useful for two related purposes: + + - Cleanly isolate the build of each package, so that a + given package only "sees" the dependencies it has + explicitly expressed, and not other packages that may + have by chance been built before. + + - Enable top-level parallel build. + + This is labeled as an experimental feature, as not all + packages behave properly with per-package folders. + endmenu comment "Security Hardening Options" diff --git a/Makefile b/Makefile index 23032988a5..f3c0e2326e 100644 --- a/Makefile +++ b/Makefile @@ -207,7 +207,8 @@ BINARIES_DIR := $(BASE_DIR)/images # The target directory is common to all packages, # but there is one that is specific to each filesystem. BASE_TARGET_DIR := $(BASE_DIR)/target -TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) +PER_PACKAGE_DIR := $(BASE_DIR)/per-package + # initial definition so that 'make clean' works for most users, even without # .config. HOST_DIR will be overwritten later when .config is included. HOST_DIR := $(BASE_DIR)/host @@ -230,6 +231,7 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),) -include $(BR2_CONFIG) endif +ifeq ($(BR2_PER_PACKAGE_FOLDERS),) # Parallel execution of this Makefile is disabled because it changes # the packages building order, that can be a problem for two reasons: # - If a package has an unspecified optional dependency and that @@ -245,6 +247,13 @@ endif # use the -j<jobs> option when building, e.g: # make -j$((`getconf _NPROCESSORS_ONLN`+1)) .NOTPARALLEL: +endif + +ifeq ($(BR2_PER_PACKAGE_FOLDERS),y) +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(if $(PKG),$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/target,$(BASE_TARGET_DIR))) +else +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) +endif # timezone and locale may affect build output ifeq ($(BR2_REPRODUCIBLE),y) @@ -455,7 +464,11 @@ LZCAT := $(call qstrip,$(BR2_LZCAT)) TAR_OPTIONS = $(call qstrip,$(BR2_TAR_OPTIONS)) -xf # packages compiled for the host go here +ifeq ($(BR2_PER_PACKAGE_FOLDERS),y) +HOST_DIR = $(if $(PKG),$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/host,$(call qstrip,$(BR2_HOST_DIR))) +else HOST_DIR := $(call qstrip,$(BR2_HOST_DIR)) +endif ifneq ($(HOST_DIR),$(BASE_DIR)/host) HOST_DIR_SYMLINK = $(BASE_DIR)/host @@ -701,14 +714,28 @@ $(TARGETS_ROOTFS): target-finalize # Avoid the rootfs name leaking down the dependency chain target-finalize: ROOTFS= -host-finalize: $(HOST_DIR_SYMLINK) +host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) +ifeq ($(BR2_PER_PACKAGE_FOLDERS),y) + @$(call MESSAGE,"Creating global host directory") + $(foreach pkg,$(PACKAGES),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/host/ \ + $(PER_PACKAGE_DIR)/$(pkg)/host/ \ + $(HOST_DIR)$(sep)) +endif .PHONY: staging-finalize staging-finalize: @ln -snf $(STAGING_DIR) $(BASE_DIR)/staging .PHONY: target-finalize -target-finalize: $(PACKAGES) host-finalize +target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize +ifeq ($(BR2_PER_PACKAGE_FOLDERS),y) + @$(call MESSAGE,"Creating global target directory") + $(foreach pkg,$(PACKAGES),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/target/ \ + $(PER_PACKAGE_DIR)/$(pkg)/target/ \ + $(TARGET_DIR)$(sep)) +endif @$(call MESSAGE,"Finalizing target directory") # Check files that are touched by more than one package ./support/scripts/check-uniq-files -t target $(BUILD_DIR)/packages-file-list.txt @@ -972,7 +999,7 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf prepare-kconfig # staging and target directories do NOT list these as # dependencies anywhere else -$(BUILD_DIR) $(BASE_TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST): +$(BUILD_DIR) $(BASE_TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST) $(PER_PACKAGE_DIR): @mkdir -p $@ # outputmakefile generates a Makefile in the output directory, if using a @@ -1011,7 +1038,7 @@ printvars: clean: rm -rf $(BASE_TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) $(HOST_DIR_SYMLINK) \ $(BUILD_DIR) $(BASE_DIR)/staging \ - $(LEGAL_INFO_DIR) $(GRAPHS_DIR) + $(LEGAL_INFO_DIR) $(GRAPHS_DIR) $(PER_PACKAGE_DIR) .PHONY: distclean distclean: clean diff --git a/fs/common.mk b/fs/common.mk index 96658428ba..fc4be3cc05 100644 --- a/fs/common.mk +++ b/fs/common.mk @@ -172,7 +172,7 @@ rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y) TARGETS_ROOTFS += rootfs-$(1) -PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_DEPENDENCIES)) +PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_DEPENDENCIES)) $(ROOTFS_COMMON_DEPENDENCIES) endif # Check for legacy POST_TARGETS rules diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 309fd8cd48..8ec5e8db73 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -98,7 +98,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += check_bin_arch # have a proper DT_RPATH or DT_RUNPATH tag define check_host_rpath $(if $(filter install-host,$(2)),\ - $(if $(filter end,$(1)),support/scripts/check-host-rpath $(3) $(HOST_DIR))) + $(if $(filter end,$(1)),support/scripts/check-host-rpath $(3) $(HOST_DIR) $(PER_PACKAGE_DIR))) endef GLOBAL_INSTRUMENTATION_HOOKS += check_host_rpath @@ -126,6 +126,21 @@ endef GLOBAL_INSTRUMENTATION_HOOKS += step_user endif +# $1: deps +ifeq ($(BR2_PER_PACKAGE_FOLDERS),y) +define prepare-per-package-folder + mkdir -p $(HOST_DIR) $(TARGET_DIR) + $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/host/ \ + $(PER_PACKAGE_DIR)/$(pkg)/host/ \ + $(HOST_DIR)$(sep)) + $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/target/ \ + $(PER_PACKAGE_DIR)/$(pkg)/target/ \ + $(TARGET_DIR)$(sep)) +endef +endif + ################################################################################ # Implicit targets -- produce a stamp file for each step of a package build ################################################################################ @@ -133,6 +148,7 @@ endif # Retrieve the archive $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) + $(call prepare-per-package-folder,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) # Only show the download message if it isn't already downloaded $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ @@ -159,6 +175,7 @@ $(BUILD_DIR)/%/.stamp_actual_downloaded: $(BUILD_DIR)/%/.stamp_extracted: @$(call step_start,extract) @$(call MESSAGE,"Extracting") + $(call prepare-per-package-folder,$($(PKG)_FINAL_EXTRACT_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_EXTRACT_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D) $($(PKG)_EXTRACT_CMDS) @@ -219,6 +236,7 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\ $(BUILD_DIR)/%/.stamp_configured: @$(call step_start,configure) @$(call MESSAGE,"Configuring") + $(call prepare-per-package-folder,$($(PKG)_FINAL_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath index 6c5767da05..787a1763b0 100755 --- a/support/scripts/check-host-rpath +++ b/support/scripts/check-host-rpath @@ -11,6 +11,7 @@ export LC_ALL=C main() { local pkg="${1}" local hostdir="${2}" + local perpackagedir="${3}" local file ret # Remove duplicate and trailing '/' for proper match @@ -20,7 +21,7 @@ main() { while read file; do is_elf "${file}" || continue elf_needs_rpath "${file}" "${hostdir}" || continue - check_elf_has_rpath "${file}" "${hostdir}" && continue + check_elf_has_rpath "${file}" "${hostdir}" "${perpackagedir}" && continue if [ ${ret} -eq 0 ]; then ret=1 printf "***\n" @@ -57,6 +58,7 @@ elf_needs_rpath() { check_elf_has_rpath() { local file="${1}" local hostdir="${2}" + local perpackagedir="${3}" local rpath dir while read rpath; do @@ -65,6 +67,7 @@ check_elf_has_rpath() { dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )" [ "${dir}" = "${hostdir}/lib" ] && return 0 [ "${dir}" = "\$ORIGIN/../lib" ] && return 0 + [[ ${dir} =~ ${perpackagedir}/[^/]*/host/lib ]] && return 0 done done < <( readelf -d "${file}" \ |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \
This commit implemnts the core of the move to per-package SDK and target directories. The main idea is that instead of having a global output/host and output/target in which all packages install files, we switch to per-package host and target folders, that only contain their explicit dependencies. There are two main benefits: - Packages will no longer discover dependencies that they do not explicitly indicate in their <pkg>_DEPENDENCIES variable. - We can support top-level parallel build properly, because a package only "sees" its own host directory and target directory, isolated from the build of other packages that can happen in parallel. It works as follows: - A new output/per-package/ folder is created, which will contain one sub-folder per package, and inside it, a "host" folder and a "target" folder: output/per-package/busybox/target output/per-package/busybox/host output/per-package/host-fakeroot/target output/per-package/host-fakeroot/host This output/per-package/ folder is PER_PACKAGE_DIR. - The global TARGET_DIR and HOST_DIR variable now automatically point to the per-package directory when PKG is defined. So whenever a package references $(HOST_DIR) or $(TARGET_DIR) in its build process, it effectively references the per-package host/target directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it is handled as well. - Of course, packages have dependencies, so those dependencies must be installed in the per-package host and target folders. To do so, we simply rsync (using hard links to save space and time) the host and target folders of the direct dependencies of the package to the current package host and target folders. We only need to take care of direct dependencies (and not recursively all dependencies), because we accumulate into those per-package host and target folders the files installed by the dependencies. Note that this only works because we make the assumption that one package does *not* overwrite files installed by another package. This is done for "extract dependencies" at the beginning of the extract step, and for "normal dependencies" at the beginning of the configure step. This is basically enough to make per-package SDK and target work. The only gotcha is that at the end of the build, output/target and output/host are empty, which means that: - The filesystem image creation code cannot work. - We don't have a SDK to build code outside of Buildroot. In order to fix this, this commit extends the target-finalize step so that it starts by populating output/target and output/host by rsync-ing into them the target and host directories of all packages listed in the $(PACKAGES) variable. It is necessary to do this sequentially in the target-finalize step and not in each package. Doing it in package installation means that it can be done in parallel. In that case, there is a chance that two rsyncs are creating the same hardlink or directory at the same time, which makes one of them fail. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- Changes since v2: - Account for <pkg>_EXTRACT_DEPENDENCIES in the extract step of the package, by rsync'ing their host and target directories to the current package host/target directory. Changes since v1: - Remove the LD_LIBRARY_PATH change since we're now longer relying on LD_LIBRARY_PATH to allow host programs to find their libraries. - Improve commit log according to Arnout suggestions - Remove -u option from rsync calls in the main Makefile, suggested by Arnout - Drop entirely the definitions of <pkg>_TARGET_DIR, <pkg>_STAGING_DIR and <pkg>_HOST_DIR, and instead make the global TARGET_DIR, HOST_DIR variables point to the per-package directories when PKG is defined. Suggested by Arnout. --- Config.in | 18 ++++++++++++++++ Makefile | 37 +++++++++++++++++++++++++++----- fs/common.mk | 2 +- package/pkg-generic.mk | 20 ++++++++++++++++- support/scripts/check-host-rpath | 5 ++++- 5 files changed, 74 insertions(+), 8 deletions(-)