| Message ID | 20180403143151.13372-1-chemobejk@gmail.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [1/1] core: enhance printvars for variables with newlines | expand |
Hello, On Tue, 3 Apr 2018 17:31:51 +0300, Stefan Becker wrote: > If the variable content has newlines in it then the currently dumped > content can't be fed again to GNU make. Add the option DEFINE_VARS which > causes the variables to be dumped using > > define VAR > ... line 1 ... > ... line 2 ... > ... > endef > > Updated the manual accordingly. Just curious, what is the use case for feeding the printvars output back into GNU Make ? Thanks! Thomas
Hi, On Tue, Apr 3, 2018 at 5:50 PM, Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Just curious, what is the use case for feeding the printvars output > back into GNU Make ? Short answer: because we have factored out buildroot image creation step to a separate build. Previously it was enough to provide the SDK, output/target/ and a selected list of source *.mk file s from the "upstream" buildroot build to be able to do that. With todays @master update the root fs creation requires internal variables, like PACKAGES_USERS, to be set correctly to create a valid, bootable image. Which took me a while to root cause... Hence we need to use "printvars" to extract the content of those internal variables from the "upstream" buildroot build into a .mk file that is passed down to the "downstream" image build, which include's it in its top-level Makefile (a stripped-down version of buildroot top-level Makefile). The variables we are interested in unfortunately contain newlines. Regards, Stefan
Stefan, All, On 2018-04-03 18:15 +0300, Stefan Becker spake thusly: > Hi, > > On Tue, Apr 3, 2018 at 5:50 PM, Thomas Petazzoni > <thomas.petazzoni@bootlin.com> wrote: > > > > Just curious, what is the use case for feeding the printvars output > > back into GNU Make ? > > Short answer: because we have factored out buildroot image creation > step to a separate build. > > Previously it was enough to provide the SDK, output/target/ and a > selected list of source *.mk file s from the "upstream" buildroot > build to be able to do that. With todays @master update the root fs > creation requires internal variables, like PACKAGES_USERS, to be set > correctly to create a valid, bootable image. Which took me a while to > root cause... The real solution IMHO is that you base your iamge creatikon out of the rootfs.tar image generated by Buildroot. If you are using a br2-external tree, you can also define your own filesystem iamge type that depends on rootfs.tar. Then your image generator (or a wrapper around that) would extract (under fakeroot) rootfs.tar, and generate your own image format out of that. (Note: this is not a reason not to apply your patch, of course, just an alternate solution for your use-case). Regards, Yann E. MORIN. > Hence we need to use "printvars" to extract the content of those > internal variables from the "upstream" buildroot build into a .mk file > that is passed down to the "downstream" image build, which include's > it in its top-level Makefile (a stripped-down version of buildroot > top-level Makefile). The variables we are interested in unfortunately > contain newlines. > > Regards, Stefan
Hi, On Tue, Apr 3, 2018 at 7:21 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > The real solution IMHO is that you base your iamge creatikon out of the > rootfs.tar image generated by Buildroot. > I'm not sure that is feasible, because then you would loose access to things that need to be done inside fakeroot (f.ex.). Plus @master the new rootfs-common.tar has been added, but the build deletes it again and thus it can't be exported as build artifact. If you are using a br2-external tree, you can also define your own > filesystem iamge type that depends on rootfs.tar. Then your image > generator (or a wrapper around that) would extract (under fakeroot) > rootfs.tar, and generate your own image format out of that. > ...which would mean we are back in the slow monolithic meta build system with our components. Sorry, but that won't do :-) BTW: we do have a br2-external tree but it is basically "pure". It is only needed so that we do not pollute the buildroot tree with things that are needed to run the buildroot build system and create the build artifacts afterwards. Regards, Stefan <div dir="ltr">Hi,<br><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 7:21 PM, Yann E. MORIN <span dir="ltr"><<a href="mailto:yann.morin.1998@free.fr" target="_blank">yann.morin.1998@free.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The real solution IMHO is that you base your iamge creatikon out of the<br> rootfs.tar image generated by Buildroot.<br></blockquote><div><br></div><div>I'm not sure that is feasible, because then you would loose access to things that need to be done inside fakeroot (f.ex.).<br><br></div><div>Plus @master the new rootfs-common.tar has been added, but the build deletes it again and thus it can't be exported as build artifact.<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> If you are using a br2-external tree, you can also define your own<br> filesystem iamge type that depends on rootfs.tar. Then your image<br> generator (or a wrapper around that) would extract (under fakeroot)<br> rootfs.tar, and generate your own image format out of that.<br></blockquote><div><br></div><div>...which would mean we are back in the slow monolithic meta build system with our components. Sorry, but that won't do :-)<br><br>BTW: we do have a br2-external tree but it is basically "pure". It is only needed so that we do not pollute the buildroot tree with things that are needed to run the buildroot build system and create the build artifacts afterwards.<br><br></div>Regards, Stefan<br><br></div></div></div></div>
Stefan, All, On 2018-04-03 19:37 +0300, Stefan Becker spake thusly: > Hi, > On Tue, Apr 3, 2018 at 7:21 PM, Yann E. MORIN < [1]yann.morin.1998@free.fr> wrote: > > The real solution IMHO is that you base your iamge creatikon out of the > rootfs.tar image generated by Buildroot. > > I'm not sure that is feasible, because then you would loose access to things that need to be done inside fakeroot (f.ex.). Of course, as I wrote below, your script that does the work woudl need to run under fakeroot, obviously. This can be easily achieved with a snippet like that: #!/bin/sh if [ $(id -u) -ne 0 ]; then fakeroot "${0]" "${@}" exit ${?} fi # here goes your real filesystem code... > Plus @master the new rootfs-common.tar has been added, but the build deletes it again and thus it can't be exported as build > artifact. Beause this is only an internal temporary file that you should not be concerned with. Besides, it is not even totaly complete either... I was talking about the *real* final rootfs.tar that you get with BR2_TARGET_ROOTFS_TAR. > If you are using a br2-external tree, you can also define your own > filesystem iamge type that depends on rootfs.tar. Then your image > generator (or a wrapper around that) would extract (under fakeroot) > rootfs.tar, and generate your own image format out of that. > > ...which would mean we are back in the slow monolithic meta build > system with our components. Sorry, but that won't do :-) Of course, if you are trying to fast-track Buildroot with local hacks, you are back on your own... > BTW: we do have a br2-external tree but it is basically "pure". It is only needed so that we do not pollute the buildroot tree with > things that are needed to run the buildroot build system and create the build artifacts afterwards. Sorry, I don't understand what you mean by "pure". There is no "purity" to talk of about a br2-external tree: by definition, you put in there whatever you want. You probably meant that you are not touching the buildroot tree. Yes, I understand. But nothing prevents you from putting your own filesystem (or rather, image) generation tool as a new filesystem type. See for example what I explained at ELCE last year: https://elinux.org/images/8/8e/2017-10-24_-_ELCE-Buildroot.pdf (slide 16) https://youtu.be/SN2hYO2rYtk?t=728 Regards, Yann E. MORIN.
Ping? Is there anything left which prevents this from getting merged to master? Regards, Stefan On Tue, Apr 3, 2018 at 5:31 PM, Stefan Becker <chemobejk@gmail.com> wrote: > If the variable content has newlines in it then the currently dumped > content can't be fed again to GNU make. Add the option DEFINE_VARS which > causes the variables to be dumped using > > define VAR > ... line 1 ... > ... line 2 ... > ... > endef > > Updated the manual accordingly. > > Signed-off-by: Stefan Becker <chemobejk@gmail.com> > --- > Makefile | 10 +++++++--- > docs/manual/make-tips.txt | 20 ++++++++++++++++++++ > 2 files changed, 27 insertions(+), 3 deletions(-) > > diff --git a/Makefile b/Makefile > index 0724f28f45..3e27195de0 100644 > --- a/Makefile > +++ b/Makefile > @@ -974,9 +974,13 @@ printvars: > $(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \ > $(if $(filter-out environment% default automatic, \ > $(origin $V)), \ > - $(if $(QUOTED_VARS),\ > - $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > - $(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))) > + $(if $(DEFINE_VARS), \ > + $(info define $V) \ > + $(info $(if $(RAW_VARS),$(value $V),$($V))) \ > + $(info endef), \ > + $(if $(QUOTED_VARS),\ > + $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > + $(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))) > # ' Syntax colouring... > > .PHONY: clean > diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt > index ea1d825bef..ba87e5d873 100644 > --- a/docs/manual/make-tips.txt > +++ b/docs/manual/make-tips.txt > @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables: > > - +VARS+ will limit the listing to variables which names match the > specified make-pattern > +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve > + newlines in the value > - +QUOTED_VARS+, if set to +YES+, will single-quote the value > - +RAW_VARS+, if set to +YES+, will print the unexpanded value > > @@ -106,6 +108,24 @@ For example: > BUSYBOX_RDEPENDENCIES=ncurses util-linux > ---- > > +---- > + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES > + define BUSYBOX_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_ALL_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_PATCH_DEPENDENCIES > + endef > + define BUSYBOX_RDEPENDENCIES > + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache > + endef > +---- > + > ---- > $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES > BUSYBOX_DEPENDENCIES='skeleton toolchain' > -- > 2.14.3 >
diff --git a/Makefile b/Makefile index 0724f28f45..3e27195de0 100644 --- a/Makefile +++ b/Makefile @@ -974,9 +974,13 @@ printvars: $(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \ $(if $(filter-out environment% default automatic, \ $(origin $V)), \ - $(if $(QUOTED_VARS),\ - $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ - $(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))) + $(if $(DEFINE_VARS), \ + $(info define $V) \ + $(info $(if $(RAW_VARS),$(value $V),$($V))) \ + $(info endef), \ + $(if $(QUOTED_VARS),\ + $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ + $(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))) # ' Syntax colouring... .PHONY: clean diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt index ea1d825bef..ba87e5d873 100644 --- a/docs/manual/make-tips.txt +++ b/docs/manual/make-tips.txt @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables: - +VARS+ will limit the listing to variables which names match the specified make-pattern +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve + newlines in the value - +QUOTED_VARS+, if set to +YES+, will single-quote the value - +RAW_VARS+, if set to +YES+, will print the unexpanded value @@ -106,6 +108,24 @@ For example: BUSYBOX_RDEPENDENCIES=ncurses util-linux ---- +---- + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES + define BUSYBOX_DEPENDENCIES + skeleton toolchain + endef + define BUSYBOX_FINAL_ALL_DEPENDENCIES + skeleton toolchain + endef + define BUSYBOX_FINAL_DEPENDENCIES + skeleton toolchain + endef + define BUSYBOX_FINAL_PATCH_DEPENDENCIES + endef + define BUSYBOX_RDEPENDENCIES + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache + endef +---- + ---- $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES BUSYBOX_DEPENDENCIES='skeleton toolchain'
If the variable content has newlines in it then the currently dumped content can't be fed again to GNU make. Add the option DEFINE_VARS which causes the variables to be dumped using define VAR ... line 1 ... ... line 2 ... ... endef Updated the manual accordingly. Signed-off-by: Stefan Becker <chemobejk@gmail.com> --- Makefile | 10 +++++++--- docs/manual/make-tips.txt | 20 ++++++++++++++++++++ 2 files changed, 27 insertions(+), 3 deletions(-)