| Message ID | 1499273594-12106-6-git-send-email-wg@grandegger.com |
|---|---|
| State | Superseded |
| Headers | show |
On 05-07-17 18:53, Wolfgang Grandegger wrote: > We introduce the "finalize-host" target for that purpose. > > Signed-off-by: Wolfgang Grandegger <wg@grandegger.com> > --- > Makefile | 7 ++++++- > 1 file changed, 6 insertions(+), 1 deletion(-) > > diff --git a/Makefile b/Makefile > index 1a826fc..2a265e0 100644 > --- a/Makefile > +++ b/Makefile > @@ -553,7 +553,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG) > prepare: $(BUILD_DIR)/buildroot-config/auto.conf > > .PHONY: world > -world: target-post-image > +world: target-post-image host-finalize > + > +.PHONY: host-finalize > +host-finalize: > + @$(call MESSAGE,"Rendering the SDK relocatable") > + $(TOPDIR)/support/scripts/fix-rpath host I'm not sure why, but this is taking a very long time for me - twice as much as target and staging together. Possibly this is because the external toolchain (which tends to be large) is also still processed. Passing TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR in the environment of this call would help. This sanitization takes a non-trivial amount of time. For example, on a config with basically just systemd enabled, rebuilding the rootfs tarballs without the sanitization takes just 5 seconds. target+staging sanitization adds 10 seconds to that. host sanitization adds another 20 seconds. So globally we go from 5 to 34 seconds as the baseline time to rebuild anything. Also, we discussed this before (also with Thomas): I'm not so sure we want to do this as part of the normal build. It would make sense to me to have a new "sdk" target which is NOT a dependency of world, that should be called explicitly to get a relocatable toolchain. That already solves a big part of the time issue I mentioned above. But it also means we can later add other things to that sdk target, like creating a tarball of it. And anyway the host sanitization is only useful if you're actually going to move the sdk somewhere else. Also, host-finalize (or sdk) should depend on toolchain, and probably also on $(filter host-%,$(PACKAGES)). The latter is currently incomplete, but when we get around to selecting BR2_PACKAGE_HOST_* for all host packages, we will make sure that 'make sdk' really generates the complete sdk. And finally, I think that the staging sanitization fits better here than in target-finalize. It has nothing to do with target, and the only reason that we do such sanitization is to make the sdk relocatable. Regards, Arnout > > # When creating HOST_DIR, also symlink usr -> . > $(HOST_DIR): >
Am 20.07.2017 um 01:51 schrieb Arnout Vandecappelle: > > > On 05-07-17 18:53, Wolfgang Grandegger wrote: >> We introduce the "finalize-host" target for that purpose. >> >> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com> >> --- >> Makefile | 7 ++++++- >> 1 file changed, 6 insertions(+), 1 deletion(-) >> >> diff --git a/Makefile b/Makefile >> index 1a826fc..2a265e0 100644 >> --- a/Makefile >> +++ b/Makefile >> @@ -553,7 +553,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG) >> prepare: $(BUILD_DIR)/buildroot-config/auto.conf >> >> .PHONY: world >> -world: target-post-image >> +world: target-post-image host-finalize >> + >> +.PHONY: host-finalize >> +host-finalize: >> + @$(call MESSAGE,"Rendering the SDK relocatable") >> + $(TOPDIR)/support/scripts/fix-rpath host > > I'm not sure why, but this is taking a very long time for me - twice as much as > target and staging together. Possibly this is because the external toolchain > (which tends to be large) is also still processed. Passing > TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR in the environment of this call would help. > > This sanitization takes a non-trivial amount of time. For example, on a config > with basically just systemd enabled, rebuilding the rootfs tarballs without the > sanitization takes just 5 seconds. target+staging sanitization adds 10 seconds > to that. host sanitization adds another 20 seconds. So globally we go from 5 to > 34 seconds as the baseline time to rebuild anything. Yes, that's the downside of doing it at the end of the build on all 3 trees. Most of the time is consumed for scanning for ELF files. The execution time should be therefore proportional to the number of scanned files. Maybe a database/cache could speed up things. Could you please send me your "defconfig" to check the times here? > > Also, we discussed this before (also with Thomas): I'm not so sure we want to > do this as part of the normal build. It would make sense to me to have a new > "sdk" target which is NOT a dependency of world, that should be called > explicitly to get a relocatable toolchain. That already solves a big part of the > time issue I mentioned above. But it also means we can later add other things to > that sdk target, like creating a tarball of it. And anyway the host sanitization > is only useful if you're actually going to move the sdk somewhere else. I already understood that "make sdk" is the preferred solution. I just thought somebody else would implement that ;). > Also, host-finalize (or sdk) should depend on toolchain, and probably also on > $(filter host-%,$(PACKAGES)). The latter is currently incomplete, but when we > get around to selecting BR2_PACKAGE_HOST_* for all host packages, we will make > sure that 'make sdk' really generates the complete sdk. I might have missed a few things here. > And finally, I think that the staging sanitization fits better here than in > target-finalize. It has nothing to do with target, and the only reason that we > do such sanitization is to make the sdk relocatable. D'accord! Wolfgang.
On 20-07-17 10:40, Wolfgang Grandegger wrote: > > > Am 20.07.2017 um 01:51 schrieb Arnout Vandecappelle: >> >> >> On 05-07-17 18:53, Wolfgang Grandegger wrote: >>> We introduce the "finalize-host" target for that purpose. >>> >>> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com> >>> --- >>> Makefile | 7 ++++++- >>> 1 file changed, 6 insertions(+), 1 deletion(-) >>> >>> diff --git a/Makefile b/Makefile >>> index 1a826fc..2a265e0 100644 >>> --- a/Makefile >>> +++ b/Makefile >>> @@ -553,7 +553,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG) >>> prepare: $(BUILD_DIR)/buildroot-config/auto.conf >>> .PHONY: world >>> -world: target-post-image >>> +world: target-post-image host-finalize >>> + >>> +.PHONY: host-finalize >>> +host-finalize: >>> + @$(call MESSAGE,"Rendering the SDK relocatable") >>> + $(TOPDIR)/support/scripts/fix-rpath host >> >> I'm not sure why, but this is taking a very long time for me - twice as much as >> target and staging together. Possibly this is because the external toolchain >> (which tends to be large) is also still processed. Passing >> TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR in the environment of this call would >> help. >> >> This sanitization takes a non-trivial amount of time. For example, on a config >> with basically just systemd enabled, rebuilding the rootfs tarballs without the >> sanitization takes just 5 seconds. target+staging sanitization adds 10 seconds >> to that. host sanitization adds another 20 seconds. So globally we go from 5 to >> 34 seconds as the baseline time to rebuild anything. > > Yes, that's the downside of doing it at the end of the build on all 3 trees. > Most of the time is consumed for scanning for ELF files. The execution time > should be therefore proportional to the number of scanned files. Maybe a > database/cache could speed up things. That was the idea behind doing things after each package build :-) For a cache, you still need to md5 all files to detect if they have change since the last time. Checking for an ELF header should be faster... > Could you please send me your "defconfig" to check the times here? It's just an external toolchain + kernel + BR2_INIT_SYSTEMD=y >> Also, we discussed this before (also with Thomas): I'm not so sure we want to >> do this as part of the normal build. It would make sense to me to have a new >> "sdk" target which is NOT a dependency of world, that should be called >> explicitly to get a relocatable toolchain. That already solves a big part of the >> time issue I mentioned above. But it also means we can later add other things to >> that sdk target, like creating a tarball of it. And anyway the host sanitization >> is only useful if you're actually going to move the sdk somewhere else. > > I already understood that "make sdk" is the preferred solution. I just thought > somebody else would implement that ;). Well, just rename host-finalize to sdk, and remove it from the dependencies of "world". Oh, and add it to the 'make help' output. Documentation should be added as well, but that should be done anyway (to explain that the relocate-sdk script should be called). >> Also, host-finalize (or sdk) should depend on toolchain, and probably also on >> $(filter host-%,$(PACKAGES)). The latter is currently incomplete, but when we >> get around to selecting BR2_PACKAGE_HOST_* for all host packages, we will make >> sure that 'make sdk' really generates the complete sdk. > > I might have missed a few things here. Dependencies are really needed, otherwise a top-level parallel build would run host-finalize in the beginning of the build instead of at the end. Oh, and forget about filtering host-% from PACKAGES. If staging sanitization is also done here, then it should depend on all packages. Regards, Arnout > >> And finally, I think that the staging sanitization fits better here than in >> target-finalize. It has nothing to do with target, and the only reason that we >> do such sanitization is to make the sdk relocatable. > > D'accord! > > Wolfgang.
Am 20.07.2017 um 10:49 schrieb Arnout Vandecappelle: > > > On 20-07-17 10:40, Wolfgang Grandegger wrote: >> >> >> Am 20.07.2017 um 01:51 schrieb Arnout Vandecappelle: >>> >>> >>> On 05-07-17 18:53, Wolfgang Grandegger wrote: >>>> We introduce the "finalize-host" target for that purpose. >>>> >>>> Signed-off-by: Wolfgang Grandegger <wg@grandegger.com> >>>> --- >>>> Makefile | 7 ++++++- >>>> 1 file changed, 6 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/Makefile b/Makefile >>>> index 1a826fc..2a265e0 100644 >>>> --- a/Makefile >>>> +++ b/Makefile >>>> @@ -553,7 +553,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG) >>>> prepare: $(BUILD_DIR)/buildroot-config/auto.conf >>>> .PHONY: world >>>> -world: target-post-image >>>> +world: target-post-image host-finalize >>>> + >>>> +.PHONY: host-finalize >>>> +host-finalize: >>>> + @$(call MESSAGE,"Rendering the SDK relocatable") >>>> + $(TOPDIR)/support/scripts/fix-rpath host >>> >>> I'm not sure why, but this is taking a very long time for me - twice as much as >>> target and staging together. Possibly this is because the external toolchain >>> (which tends to be large) is also still processed. Passing >>> TOOLCHAIN_EXTERNAL_DOWNLOAD_INSTALL_DIR in the environment of this call would >>> help. >>> >>> This sanitization takes a non-trivial amount of time. For example, on a config >>> with basically just systemd enabled, rebuilding the rootfs tarballs without the >>> sanitization takes just 5 seconds. target+staging sanitization adds 10 seconds >>> to that. host sanitization adds another 20 seconds. So globally we go from 5 to >>> 34 seconds as the baseline time to rebuild anything. >> >> Yes, that's the downside of doing it at the end of the build on all 3 trees. >> Most of the time is consumed for scanning for ELF files. The execution time >> should be therefore proportional to the number of scanned files. Maybe a >> database/cache could speed up things. > > That was the idea behind doing things after each package build :-) > > For a cache, you still need to md5 all files to detect if they have change > since the last time. Checking for an ELF header should be faster... > >> Could you please send me your "defconfig" to check the times here? > > It's just an external toolchain + kernel + BR2_INIT_SYSTEMD=y > > >>> Also, we discussed this before (also with Thomas): I'm not so sure we want to >>> do this as part of the normal build. It would make sense to me to have a new >>> "sdk" target which is NOT a dependency of world, that should be called >>> explicitly to get a relocatable toolchain. That already solves a big part of the >>> time issue I mentioned above. But it also means we can later add other things to >>> that sdk target, like creating a tarball of it. And anyway the host sanitization >>> is only useful if you're actually going to move the sdk somewhere else. >> >> I already understood that "make sdk" is the preferred solution. I just thought >> somebody else would implement that ;). > > Well, just rename host-finalize to sdk, and remove it from the dependencies of > "world". Oh, and add it to the 'make help' output. > > Documentation should be added as well, but that should be done anyway (to > explain that the relocate-sdk script should be called). OK, just to double-check. "make sdk" will sanitize the staging and host tree and add relocate-sdk.sh. But the target tree will always be sanitized. Wolfgang.
On 20-07-17 11:15, Wolfgang Grandegger wrote: > > > Am 20.07.2017 um 10:49 schrieb Arnout Vandecappelle: [snip] >> Well, just rename host-finalize to sdk, and remove it from the dependencies of >> "world". Oh, and add it to the 'make help' output. >> >> Documentation should be added as well, but that should be done anyway (to >> explain that the relocate-sdk script should be called). > > OK, just to double-check. "make sdk" will sanitize the staging and host tree and > add relocate-sdk.sh. But the target tree will always be sanitized. Exactly! Regards, Arnout
Hello, On Thu, 20 Jul 2017 01:51:53 +0200, Arnout Vandecappelle wrote: > This sanitization takes a non-trivial amount of time. For example, on a config > with basically just systemd enabled, rebuilding the rootfs tarballs without the > sanitization takes just 5 seconds. target+staging sanitization adds 10 seconds > to that. host sanitization adds another 20 seconds. So globally we go from 5 to > 34 seconds as the baseline time to rebuild anything. This is indeed not good. > Also, we discussed this before (also with Thomas): I'm not so sure we want to > do this as part of the normal build. It would make sense to me to have a new > "sdk" target which is NOT a dependency of world, that should be called > explicitly to get a relocatable toolchain. That already solves a big part of the > time issue I mentioned above. But it also means we can later add other things to > that sdk target, like creating a tarball of it. And anyway the host sanitization > is only useful if you're actually going to move the sdk somewhere else. Yes for "make sdk". > Also, host-finalize (or sdk) should depend on toolchain, and probably also on > $(filter host-%,$(PACKAGES)). The latter is currently incomplete, but when we > get around to selecting BR2_PACKAGE_HOST_* for all host packages, we will make > sure that 'make sdk' really generates the complete sdk. For the SDK, you also need to build all the target packages. The SDK contains the sysroot (i.e STAGING_DIR), where the target packages install libraries and headers. Basically, sdk needs to depend on "world". > And finally, I think that the staging sanitization fits better here than in > target-finalize. It has nothing to do with target, and the only reason that we > do such sanitization is to make the sdk relocatable. The sanitization in staging and target has nothing to do with making the SDK relocatable. Why do you think it's related ? Best regards, Thomas
Hello, On Thu, 20 Jul 2017 12:45:38 +0200, Arnout Vandecappelle wrote: > > Am 20.07.2017 um 10:49 schrieb Arnout Vandecappelle: > > [snip] > >> Well, just rename host-finalize to sdk, and remove it from the dependencies of > >> "world". Oh, and add it to the 'make help' output. > >> > >> Documentation should be added as well, but that should be done anyway (to > >> explain that the relocate-sdk script should be called). > > > > OK, just to double-check. "make sdk" will sanitize the staging and host tree and > > add relocate-sdk.sh. But the target tree will always be sanitized. > > Exactly! No, I don't agree here. Why would the staging sanitization be related to doing the SDK ? Best regards, Thomas
On 20-07-17 15:44, Thomas Petazzoni wrote: > Hello, > > On Thu, 20 Jul 2017 01:51:53 +0200, Arnout Vandecappelle wrote: [snip] >> Also, host-finalize (or sdk) should depend on toolchain, and probably also on >> $(filter host-%,$(PACKAGES)). The latter is currently incomplete, but when we >> get around to selecting BR2_PACKAGE_HOST_* for all host packages, we will make >> sure that 'make sdk' really generates the complete sdk. > > For the SDK, you also need to build all the target packages. The SDK > contains the sysroot (i.e STAGING_DIR), where the target packages > install libraries and headers. This is what I wrote in an earlier reply to Wolfgang. > Basically, sdk needs to depend on "world". Not really, it doesn't need to depend on target-finalize or the rootfs targets. >> And finally, I think that the staging sanitization fits better here than in >> target-finalize. It has nothing to do with target, and the only reason that we >> do such sanitization is to make the sdk relocatable. > > The sanitization in staging and target has nothing to do with making > the SDK relocatable. Why do you think it's related ? target indeed has nothing to do with making the SDK relocatable. It is only needed to deal with stupid packages that don't correctly use --prefix/DESTDIR and that end up putting the full absolute build-time directory in rpath. staging is different. Sanitizing staging is not really needed, in the sense that any rpath in there is simply not going to be used. We want to sanitize staging for the following reasons: - To avoid leaking references to the original output directory. This way, we can validate that the SDK is relocatable by running a simple "grep -r ${BASE_DIR} ${HOST_DIR}". Obviously RPATH sanitization is not sufficient (e.g. also the references to source files have to be stripped), but it's a step in the right direction. This reason is obviously only relevant for the SDK. - To make sure that when an executable is copied to target that it actually executes correctly. Since within Buildroot we never copy stuff from staging to target, this is clearly only relevant for the SDK. Wolfgang, could you make sure that this explanation ends up in the commit log of patch 3/9? Regards, Arnout > > Best regards, > > Thomas >
Hello, On Thu, 20 Jul 2017 15:58:49 +0200, Arnout Vandecappelle wrote: > >> Also, host-finalize (or sdk) should depend on toolchain, and probably also on > >> $(filter host-%,$(PACKAGES)). The latter is currently incomplete, but when we > >> get around to selecting BR2_PACKAGE_HOST_* for all host packages, we will make > >> sure that 'make sdk' really generates the complete sdk. > > > > For the SDK, you also need to build all the target packages. The SDK > > contains the sysroot (i.e STAGING_DIR), where the target packages > > install libraries and headers. > > This is what I wrote in an earlier reply to Wolfgang. Yes, I was replying as I was reading the thread, and saw your other reply afterwards. > > Basically, sdk needs to depend on "world". > > Not really, it doesn't need to depend on target-finalize or the rootfs targets. Correct, but I am not sure it is really worth being smart here. > >> And finally, I think that the staging sanitization fits better here than in > >> target-finalize. It has nothing to do with target, and the only reason that we > >> do such sanitization is to make the sdk relocatable. > > > > The sanitization in staging and target has nothing to do with making > > the SDK relocatable. Why do you think it's related ? > > target indeed has nothing to do with making the SDK relocatable. It is only > needed to deal with stupid packages that don't correctly use --prefix/DESTDIR > and that end up putting the full absolute build-time directory in rpath. Agreed. > staging is different. Sanitizing staging is not really needed, in the sense > that any rpath in there is simply not going to be used. We want to sanitize > staging for the following reasons: > > - To avoid leaking references to the original output directory. This way, we can > validate that the SDK is relocatable by running a simple "grep -r ${BASE_DIR} > ${HOST_DIR}". Obviously RPATH sanitization is not sufficient (e.g. also the > references to source files have to be stripped), but it's a step in the right > direction. This reason is obviously only relevant for the SDK. > > - To make sure that when an executable is copied to target that it actually > executes correctly. Since within Buildroot we never copy stuff from staging to > target, this is clearly only relevant for the SDK. OK, makes sense. I indeed believe having those details in the commit log or in the code would be nice. So do we agree that: - target sanitization will be done in target-finalize - staging sanitization will be done in "make sdk" - host sanitization will be done in "make sdk" Correct ? Thanks, Thomas
On 20-07-17 16:06, Thomas Petazzoni wrote: > Hello, > > On Thu, 20 Jul 2017 15:58:49 +0200, Arnout Vandecappelle wrote: [snip] >>> Basically, sdk needs to depend on "world". >> >> Not really, it doesn't need to depend on target-finalize or the rootfs targets. > > Correct, but I am not sure it is really worth being smart here. Indeed it isn't. And having sdk: world will discourage anyone trying to add world: sdk :-) [snip] > So do we agree that: > > - target sanitization will be done in target-finalize > > - staging sanitization will be done in "make sdk" > > - host sanitization will be done in "make sdk" > > Correct ? Yes! And also other sanitization (like stripping references to the build directories) will be done in "make sdk". Regards, Arnout
diff --git a/Makefile b/Makefile index 1a826fc..2a265e0 100644 --- a/Makefile +++ b/Makefile @@ -553,7 +553,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BR2_CONFIG) prepare: $(BUILD_DIR)/buildroot-config/auto.conf .PHONY: world -world: target-post-image +world: target-post-image host-finalize + +.PHONY: host-finalize +host-finalize: + @$(call MESSAGE,"Rendering the SDK relocatable") + $(TOPDIR)/support/scripts/fix-rpath host # When creating HOST_DIR, also symlink usr -> . $(HOST_DIR):
We introduce the "finalize-host" target for that purpose. Signed-off-by: Wolfgang Grandegger <wg@grandegger.com> --- Makefile | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)