| Message ID | 20180416115855.26762-1-chemobejk@gmail.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [RESEND] core: enhance printvars for variables with newlines | expand |
Stefan, All, On 2018-04-16 14:58 +0300, Stefan Becker spake thusly: > If the variable content has newlines in it then the currently dumped > content can't be fed again to GNU make. Add the option DEFINE_VARS which > causes the variables to be dumped using > > define VAR > ... line 1 ... > ... line 2 ... > ... > endef > > Updated the manual accordingly. > > Signed-off-by: Stefan Becker <chemobejk@gmail.com> What did change from the previous iteration, which is visible, and still pending, there: https://patchwork.ozlabs.org/project/buildroot/list/ As you can see, there are a *lot* of pending patches, so resending yours will not help much, unless there was a change, in which case you should say so as a post-commit log (i.e. below a '---' line). I'm marking the old iteration as supserseded, now... Regards, Yann E. MORIN. > --- > Makefile | 10 +++++++--- > docs/manual/make-tips.txt | 20 ++++++++++++++++++++ > 2 files changed, 27 insertions(+), 3 deletions(-) > > diff --git a/Makefile b/Makefile > index 0724f28f45..3e27195de0 100644 > --- a/Makefile > +++ b/Makefile > @@ -974,9 +974,13 @@ printvars: > $(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \ > $(if $(filter-out environment% default automatic, \ > $(origin $V)), \ > - $(if $(QUOTED_VARS),\ > - $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > - $(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))) > + $(if $(DEFINE_VARS), \ > + $(info define $V) \ > + $(info $(if $(RAW_VARS),$(value $V),$($V))) \ > + $(info endef), \ > + $(if $(QUOTED_VARS),\ > + $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > + $(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))) > # ' Syntax colouring... > > .PHONY: clean > diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt > index ea1d825bef..ba87e5d873 100644 > --- a/docs/manual/make-tips.txt > +++ b/docs/manual/make-tips.txt > @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables: > > - +VARS+ will limit the listing to variables which names match the > specified make-pattern > +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve > + newlines in the value > - +QUOTED_VARS+, if set to +YES+, will single-quote the value > - +RAW_VARS+, if set to +YES+, will print the unexpanded value > > @@ -106,6 +108,24 @@ For example: > BUSYBOX_RDEPENDENCIES=ncurses util-linux > ---- > > +---- > + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES > + define BUSYBOX_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_ALL_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_PATCH_DEPENDENCIES > + endef > + define BUSYBOX_RDEPENDENCIES > + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache > + endef > +---- > + > ---- > $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES > BUSYBOX_DEPENDENCIES='skeleton toolchain' > -- > 2.14.3 >
On 16-04-18 13:58, Stefan Becker wrote: > If the variable content has newlines in it then the currently dumped > content can't be fed again to GNU make. Add the option DEFINE_VARS which > causes the variables to be dumped using > > define VAR > ... line 1 ... > ... line 2 ... > ... > endef I'm still not convinced that this is such a great idea. 1. As explained in reply to your first iteration, there shouldn't be a need for including Buildroot variables in a surrounding Makefile. 2. At least equally relevant would be output that is appropriate for Python, JSON, ... 3. The implementation is not complete. In case of RAW_VARS, it's pretty hard to use in practice because also all the referenced variables would need to be retrieved. In the expanded case, any remaining $ will be interpreted by your surrounding make instead of being passed down to the shell like they should. Try e.g. CANFESTIVAL_INSTALL_TARGET_CMDS. Note that QUOTED_VARS is not perfect either, mainly because there are some make variables that are not legal shell variables (e.g. all the 4th stuff). So, if anything changes in here, I'd rather have some output that is in some intermediate format that can easily be converted into or parsed by shell, python, etc. Regards, Arnout > > Updated the manual accordingly. > > Signed-off-by: Stefan Becker <chemobejk@gmail.com> > --- > Makefile | 10 +++++++--- > docs/manual/make-tips.txt | 20 ++++++++++++++++++++ > 2 files changed, 27 insertions(+), 3 deletions(-) > > diff --git a/Makefile b/Makefile > index 0724f28f45..3e27195de0 100644 > --- a/Makefile > +++ b/Makefile > @@ -974,9 +974,13 @@ printvars: > $(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \ > $(if $(filter-out environment% default automatic, \ > $(origin $V)), \ > - $(if $(QUOTED_VARS),\ > - $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > - $(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))) > + $(if $(DEFINE_VARS), \ > + $(info define $V) \ > + $(info $(if $(RAW_VARS),$(value $V),$($V))) \ > + $(info endef), \ > + $(if $(QUOTED_VARS),\ > + $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > + $(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))) > # ' Syntax colouring... > > .PHONY: clean > diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt > index ea1d825bef..ba87e5d873 100644 > --- a/docs/manual/make-tips.txt > +++ b/docs/manual/make-tips.txt > @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables: > > - +VARS+ will limit the listing to variables which names match the > specified make-pattern > +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve > + newlines in the value > - +QUOTED_VARS+, if set to +YES+, will single-quote the value > - +RAW_VARS+, if set to +YES+, will print the unexpanded value > > @@ -106,6 +108,24 @@ For example: > BUSYBOX_RDEPENDENCIES=ncurses util-linux > ---- > > +---- > + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES > + define BUSYBOX_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_ALL_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_DEPENDENCIES > + skeleton toolchain > + endef > + define BUSYBOX_FINAL_PATCH_DEPENDENCIES > + endef > + define BUSYBOX_RDEPENDENCIES > + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache > + endef > +---- > + > ---- > $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES > BUSYBOX_DEPENDENCIES='skeleton toolchain' >
Hi, On Thu, Apr 19, 2018 at 1:27 AM, Arnout Vandecappelle <arnout@mind.be> wrote: > > I'm still not convinced that this is such a great idea. > > 1. As explained in reply to your first iteration, there shouldn't be a need for > including Buildroot variables in a surrounding Makefile. There is nothing you can say that will change the fact that a monolithic meta build system is too slow for the CI of our own components. So let's just agree to disagree on this point. > 2. At least equally relevant would be output that is appropriate for Python, > JSON, ... I see your point. How about this instead? $ external/scripts/buildstep.sh printvars MULTILINE_VARS=1 VARS="CANFESTIVAL_INSTALL_TARGET_CMDS" MULTILINE_VAR_START CANFESTIVAL_INSTALL_TARGET_CMDS for d in src drivers ; do PATH="/workarea/stefanb/repos/buildroot-new/output/host/bin:/workarea/stefanb/repos/buildroot-new/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/stefanb/.local/bin:/home/stefanb/bin" /usr/bin/make -j1 -C ./$d install PYTHON=/workarea/stefanb/repos/buildroot-new/output/host/bin/python2 DESTDIR=/workarea/stefanb/repos/buildroot-new/output/target || exit 1 ; done MULTILINE_VAR_END CANFESTIVAL_INSTALL_TARGET_CMDS i.e. the consumer of the MULTILINE_VARS=1 output is then responsible to post process it into the format he requires, e.g. for my intended purpose I would use ... | sed \ -e 's/^MULTILINE_VAR_START \(.\+\)$/define \1/' \ -e 's/^MULTILINE_VAR_END .\+$/endef/' > 3. The implementation is not complete. In case of RAW_VARS, it's pretty hard to > use in practice because also all the referenced variables would need to be > retrieved. In the expanded case, any remaining $ will be interpreted by your > surrounding make instead of being passed down to the shell like they should. Try > e.g. CANFESTIVAL_INSTALL_TARGET_CMDS. > > Note that QUOTED_VARS is not perfect either, mainly because there are some make > variables that are not legal shell variables (e.g. all the 4th stuff). IMHO RAW_VARS, as-is today, is only meant to produce human readable format for debug purposes. I.e. RAW_VARS=1 with any of the formats none/QUOTE_VARS/MULTILINE_VARS will not be usable for post-processing, unless the post-processing knows how to parse for variables and request them. Fixing RAW_VARS in such a way that any variable detected would be added to VARS list and then recurse to get all referenced variables, is outside the scope of my change. Or alternatively: drop RAW_VARS support from QUOTED_VARS/MULTILINE_VARS branches. Regards, Stefan
Arnout, All, On 2018-04-19 00:27 +0200, Arnout Vandecappelle spake thusly: > On 16-04-18 13:58, Stefan Becker wrote: > > If the variable content has newlines in it then the currently dumped > > content can't be fed again to GNU make. Add the option DEFINE_VARS which > > causes the variables to be dumped using > > > > define VAR > > ... line 1 ... > > ... line 2 ... > > ... > > endef > > I'm still not convinced that this is such a great idea. Neither am I... And if I look around, I think the problem explained in the commit log is already covered: $ make defconfig $ make -s printvars VARS=HOST_CARGO_INSTALL_CMDS QUOTED_VARS=YES HOST_CARGO_INSTALL_CMDS=' /usr/bin/install -D -m 0755 ./target/release/cargo /home/ymorin/dev/buildroot/O/host/bin/cargo /usr/bin/install -D package/cargo/config.in /home/ymorin/dev/buildroot/O/host/share/cargo/config /bin/sed -i -e '\''s/@RUSTC_TARGET_NAME@/i686-unknown-linux-gnu/'\'' /home/ymorin/dev/buildroot/O/host/share/cargo/config /bin/sed -i -e '\''s/@CROSS_PREFIX@/i686-buildroot-linux-uclibc-/'\'' /home/ymorin/dev/buildroot/O/host/share/cargo/config' I.e. the newlines are embedded in the quoted value, so at the very worst, one can re-inject this value in a new variable (with a bit of shell trickery, yes, but rather easily). Note that, when it encounters a \-continued line, make joins it with the next line, squeezing leading and trailing spaces/atbs into a single space. But otherwise, newlines are kept as-is. This can be seen above, but also in this one, where it is even more obvious that newlines are kept: $ eval make -s printvars VARS=SYSTEMD_USERS QUOTED_VARS=YES SYSTEMD_USERS=' - - input -1 * - - - Input device group - - systemd-journal -1 * - - - Journal - - render -1 * - - - DRI rendering nodes - - kvm -1 * - - - kvm nodes systemd-bus-proxy -1 systemd-bus-proxy -1 * - - - Proxy D-Bus messages to/from a bus systemd-journal-gateway -1 systemd-journal-gateway -1 * /var/log/journal - - Journal Gateway systemd-journal-remote -1 systemd-journal-remote -1 * /var/log/journal/remote - - Journal Remote systemd-journal-upload -1 systemd-journal-upload -1 * - - - Journal Upload ' Compare with the non-expanded, raw value: $ eval make -s printvars VARS=SYSTEMD_USERS QUOTED_VARS=YES RAW_VARS=YES SYSTEMD_USERS=' - - input -1 * - - - Input device group - - systemd-journal -1 * - - - Journal - - render -1 * - - - DRI rendering nodes - - kvm -1 * - - - kvm nodes systemd-bus-proxy -1 systemd-bus-proxy -1 * - - - Proxy D-Bus messages to/from a bus systemd-journal-gateway -1 systemd-journal-gateway -1 * /var/log/journal - - Journal Gateway systemd-journal-remote -1 systemd-journal-remote -1 * /var/log/journal/remote - - Journal Remote systemd-journal-upload -1 systemd-journal-upload -1 * - - - Journal Upload $(SYSTEMD_COREDUMP_USER) $(SYSTEMD_NETWORKD_USER) $(SYSTEMD_RESOLVED_USER) $(SYSTEMD_TIMESYNCD_USER)' So, newlines are still present, and we can especially see this is the case around the variables that are expanded to empty (the coredump, networkd, resolved, and timesyncd users). So, the whole excuse for the change, as explained in the commit log, does not stand: newlines are preserved. > 1. As explained in reply to your first iteration, there shouldn't be a need for > including Buildroot variables in a surrounding Makefile. That's because, as I remember and understand it, Stefan finds that Buildroot is not fast enough for his use-case, and so extracts such commands in an attempt to feed them back into his own build mechanism around Buildroot. Or something like that; I did not understand the details fully... > 2. At least equally relevant would be output that is appropriate for Python, > JSON, ... I'm not sure I agree with you here: json is about representtaion, python is a completely different language, while what Stefan is initially asking for is a way to retrieve part of the makefile rules for feeding them back into make. But there is a solution for that: run make in debug mode, and scrape its output. > 3. The implementation is not complete. In case of RAW_VARS, it's pretty hard to > use in practice because also all the referenced variables would need to be > retrieved. In the expanded case, any remaining $ will be interpreted by your > surrounding make instead of being passed down to the shell like they should. Try > e.g. CANFESTIVAL_INSTALL_TARGET_CMDS. > > Note that QUOTED_VARS is not perfect either, mainly because there are some make > variables that are not legal shell variables (e.g. all the 4th stuff). Right. > So, if anything changes in here, I'd rather have some output that is in some > intermediate format that can easily be converted into or parsed by shell, > python, etc. And I'd rather not: quoted values already contain the expanded value that is passed to the shell (and because raw shell code), and we can't do better than that. The raw values contain the code as it was written by the packager, and we can't do better than that either. Anyway, as explained by Arnout, this new output format can't produce working make code, so I've marked this patch as rejected in patchwork. If you can come with a better solution that works in all cases and is generic enough, then we can look back at it. Regards, Yann E. MORIN. > Regards, > Arnout > > > > > Updated the manual accordingly. > > > > Signed-off-by: Stefan Becker <chemobejk@gmail.com> > > --- > > Makefile | 10 +++++++--- > > docs/manual/make-tips.txt | 20 ++++++++++++++++++++ > > 2 files changed, 27 insertions(+), 3 deletions(-) > > > > diff --git a/Makefile b/Makefile > > index 0724f28f45..3e27195de0 100644 > > --- a/Makefile > > +++ b/Makefile > > @@ -974,9 +974,13 @@ printvars: > > $(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \ > > $(if $(filter-out environment% default automatic, \ > > $(origin $V)), \ > > - $(if $(QUOTED_VARS),\ > > - $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > > - $(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))) > > + $(if $(DEFINE_VARS), \ > > + $(info define $V) \ > > + $(info $(if $(RAW_VARS),$(value $V),$($V))) \ > > + $(info endef), \ > > + $(if $(QUOTED_VARS),\ > > + $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ > > + $(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))) > > # ' Syntax colouring... > > > > .PHONY: clean > > diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt > > index ea1d825bef..ba87e5d873 100644 > > --- a/docs/manual/make-tips.txt > > +++ b/docs/manual/make-tips.txt > > @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables: > > > > - +VARS+ will limit the listing to variables which names match the > > specified make-pattern > > +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve > > + newlines in the value > > - +QUOTED_VARS+, if set to +YES+, will single-quote the value > > - +RAW_VARS+, if set to +YES+, will print the unexpanded value > > > > @@ -106,6 +108,24 @@ For example: > > BUSYBOX_RDEPENDENCIES=ncurses util-linux > > ---- > > > > +---- > > + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES > > + define BUSYBOX_DEPENDENCIES > > + skeleton toolchain > > + endef > > + define BUSYBOX_FINAL_ALL_DEPENDENCIES > > + skeleton toolchain > > + endef > > + define BUSYBOX_FINAL_DEPENDENCIES > > + skeleton toolchain > > + endef > > + define BUSYBOX_FINAL_PATCH_DEPENDENCIES > > + endef > > + define BUSYBOX_RDEPENDENCIES > > + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache > > + endef > > +---- > > + > > ---- > > $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES > > BUSYBOX_DEPENDENCIES='skeleton toolchain' > > > > -- > Arnout Vandecappelle arnout at mind be > Senior Embedded Software Architect +32-16-286500 > Essensium/Mind http://www.mind.be > G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven > LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle > GPG fingerprint: 7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF
Hi, On Thu, Apr 19, 2018 at 10:47 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote: > Compare with the non-expanded, raw value: > > $ eval make -s printvars VARS=SYSTEMD_USERS QUOTED_VARS=YES RAW_VARS=YES > SYSTEMD_USERS=' - - input -1 * - - - Input device group > - - systemd-journal -1 * - - - Journal > - - render -1 * - - - DRI rendering nodes > - - kvm -1 * - - - kvm nodes > systemd-bus-proxy -1 systemd-bus-proxy -1 * - - - Proxy D-Bus messages to/from a bus > systemd-journal-gateway -1 systemd-journal-gateway -1 * /var/log/journal - - Journal Gateway > systemd-journal-remote -1 systemd-journal-remote -1 * /var/log/journal/remote - - Journal Remote > systemd-journal-upload -1 systemd-journal-upload -1 * - - - Journal Upload > $(SYSTEMD_COREDUMP_USER) > $(SYSTEMD_NETWORKD_USER) > $(SYSTEMD_RESOLVED_USER) > $(SYSTEMD_TIMESYNCD_USER)' > > So, newlines are still present, and we can especially see this is the > case around the variables that are expanded to empty (the coredump, > networkd, resolved, and timesyncd users). > > So, the whole excuse for the change, as explained in the commit log, > does not stand: newlines are preserved. Can you please explain to me how to detect reliably where the variable content ends with QUOTED_VARS=YES? Example of variables with and without newlines in the same output: $ external/scripts/buildstep.sh printvars QUOTED_VARS=1 VARS="CANFESTIVAL_INSTALL_TARGET_CMDS SYSTEMD_USERS" CANFESTIVAL_INSTALL_TARGET_CMDS=' for d in src drivers ; do PATH="/workarea/stefanb/repos/buildroot-new/output/host/bin:/workarea/stefanb/repos/buildroot-new/output/host/sbin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/stefanb/.local/bin:/home/stefanb/bin" /usr/bin/make -j1 -C ./$d install PYTHON=/workarea/stefanb/repos/buildroot-new/output/host/bin/python2 DESTDIR=/workarea/stefanb/repos/buildroot-new/output/target || exit 1 ; done' SYSTEMD_USERS=' - - input -1 * - - - Input device group - - systemd-journal -1 * - - - Journal ... systemd-resolve -1 systemd-resolve -1 * - - - Network Name Resolution Manager systemd-timesync -1 systemd-timesync -1 * - - - Network Time Synchronization' $ external/scripts/buildstep.sh printvars QUOTED_VARS=1 VARS="CANFESTIVAL_INSTALL_TARGET_CMDS SYSTEMD_USERS" | wc -l 13 MULTILINE_VARS=YES adds an additional seperator that makes it possible to reliably detect the end of the variable content. One improvement idea would be to use the content of MULTILINE_VARS as the separator, e.g. .... MULTILINE_VARS=some-random-seperator-XYZ ... START-some-random-seperator-XYZ VAR line 1 line 2 ... END-some-random-seperator-XYZ VAR That way you could cases where the variable content matches a fixed separator. Regards, Stefan
Hello Stefan, On Thu, 19 Apr 2018 10:14:40 +0300, Stefan Becker wrote: > On Thu, Apr 19, 2018 at 1:27 AM, Arnout Vandecappelle <arnout@mind.be> wrote: > > > > I'm still not convinced that this is such a great idea. > > > > 1. As explained in reply to your first iteration, there shouldn't be a need for > > including Buildroot variables in a surrounding Makefile. > > There is nothing you can say that will change the fact that a > monolithic meta build system is too slow for the CI of our own > components. So let's just agree to disagree on this point. I'm jumping back to this very old thread. Could you explain (perhaps re-explain) what you're trying to do ? If you need to just re-run a specific step of a specific package, why don't you use "make foo-build", "make foo-rebuild", etc. ? It seems very hackish to extract the commands that Buildroot executes to execute them outside the context of Buildroot. This doesn't really help to make a case for accepting a patch like you're proposing. Could you first convince us that there is a valid and reasonable use-case, and then we can discuss how to support this use-case with Buildroot. Best regards, Thomas
Hi, On Sun, Dec 16, 2018 at 3:54 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > I'm jumping back to this very old thread. Could you explain (perhaps > re-explain) what you're trying to do ? Modular builds to enable usable CI builds, i.e. any change a developer pushes to one of the internal components will be available as a flash-able image in less than 5 minutes. CI builds are not possible with a monolithic mega meta build system approach like buildroot. > If you need to just re-run a specific step of a specific package, why > don't you use "make foo-build", "make foo-rebuild", etc. ? buildroot meta build system assumes that the *whole* source code & build tree is available. In a modular build approach that is not the case for the "combining" build, e.g. the flash image build, as it only receives the build artefacts from the module builds. > It seems very hackish to extract the commands that Buildroot executes > to execute them outside the context of Buildroot. This doesn't really > help to make a case for accepting a patch like you're proposing. Could > you first convince us that there is a valid and reasonable use-case, > and then we can discuss how to support this use-case with Buildroot. Understandable from the PoV of monolithic meta build system advocates like buildroot, yocto, Android, ... But I live in the real world and simply can''t ignore Amdahl's law. Furthermore I can't, or management doesn't allow me, to spend huge amounts of money on build machines that can run monolithic mega meta build systems within time spans acceptable for CI (which in the end is futile anyway due to Amdahl's law as SW only ever grows in size. Trust me, I've been there...). Hence I've chosen a modular build approach to address the issue. As it became clear that this patch would not be accepted any time soon, I reverted it and implemented a different approach based on quoted var printout and sed post-processing. While not 100% fool-proof, it does work for the few build variables that are required for the modular build approach. Anyway, I no longer work for the company that is using buildroot, so I don't really care if you accept this patch or not. Regards, Stefan
Hello Stefan, Thanks for the feedback. On Sun, 16 Dec 2018 16:57:02 +0200, Stefan Becker wrote: > > I'm jumping back to this very old thread. Could you explain (perhaps > > re-explain) what you're trying to do ? > > Modular builds to enable usable CI builds, i.e. any change a developer > pushes to one of the internal components will be available as a > flash-able image in less than 5 minutes. CI builds are not possible > with a monolithic mega meta build system approach like buildroot. Yes, this I understand. > > If you need to just re-run a specific step of a specific package, why > > don't you use "make foo-build", "make foo-rebuild", etc. ? > > buildroot meta build system assumes that the *whole* source code & > build tree is available. In a modular build approach that is not the > case for the "combining" build, e.g. the flash image build, as it only > receives the build artefacts from the module builds. I'm not sure to follow you on this last sentence. But yes, Buildroot assumes that the whole source code and build tree is available. > > It seems very hackish to extract the commands that Buildroot executes > > to execute them outside the context of Buildroot. This doesn't really > > help to make a case for accepting a patch like you're proposing. Could > > you first convince us that there is a valid and reasonable use-case, > > and then we can discuss how to support this use-case with Buildroot. > > Understandable from the PoV of monolithic meta build system advocates > like buildroot, yocto, Android, ... > > But I live in the real world and simply can''t ignore Amdahl's law. > Furthermore I can't, or management doesn't allow me, to spend huge > amounts of money on build machines that can run monolithic mega meta > build systems within time spans acceptable for CI (which in the end is > futile anyway due to Amdahl's law as SW only ever grows in size. Trust > me, I've been there...). Hence I've chosen a modular build approach to > address the issue. > > As it became clear that this patch would not be accepted any time > soon, I reverted it and implemented a different approach based on > quoted var printout and sed post-processing. While not 100% > fool-proof, it does work for the few build variables that are required > for the modular build approach. You did not answer my question: why didn't you use features of Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ? How in practice you were using those commands extracted from Buildroot variables to achieve "modular builds" ? Best regards, Thomas
Hi, On Sun, Dec 16, 2018 at 5:14 PM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > You did not answer my question: why didn't you use features of > Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ? That would require the buildroot source tree to be available. Plus adding all internal components, even the ones that have nothing to with buildroot and do not use any of its host build tools, to the buildroot meta build system for buildroot build to recognize the generated build artefacts. > How in practice you were using those commands extracted from Buildroot > variables to achieve "modular builds" ? That depends on the module, e.g. * buildroot: output/target/, host/ plus selected buildroot meta build system source files required for flash image build + module install script * kernel: output from modules_install & install + module install script * ... (you get the picture) But definitely no source code is included in the module build artefact. All internal components have been removed from buildroot meta build system, i.e. buildroot module build only builds components included in its component list, i.e. buildroot module build only needs to be executed after pulling changes from upstream. Regards, Stefan
Hi, On Sun, Dec 16, 2018 at 5:33 PM Stefan Becker <chemobejk@gmail.com> wrote: > > * buildroot: output/target/, host/ plus selected buildroot meta build > system source files required for flash image build + module install > script To be more exact: * build artefact 1: SDK * build artefact 2: output/target + selected buildroot meta build system source files required for flash image build + module install script Build artefact is a dependency for module builds that require target toolchain, e.g. kernel build. Regards, Stefan
On Sun, 2018-12-16 at 16:14 +0100, Thomas Petazzoni wrote: > > > Understandable from the PoV of monolithic meta build system advocates > > like buildroot, yocto, Android, ... > > > > As it became clear that this patch would not be accepted any time > > soon, I reverted it and implemented a different approach based on > > quoted var printout and sed post-processing. While not 100% > > fool-proof, it does work for the few build variables that are required > > for the modular build approach. > > You did not answer my question: why didn't you use features of > Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ? I am also using buildroot with modern style CI. While I find <pkg>- rebuild works great for manual developer workflow, there are issues with CI integration. In the CI system, we have many many git branches active at any time. They come and go frequently and don't start at the same branch point. There are also many build agents (themselves VMs) which perform the actual build. So say we have a new branch we wish to test. The agent it is dispatched to happens to have the output tree from the last successful build on that agent (this is in itself very problematic!). That last build might be from anything in the git tree. The head of a different branch. An ancestor commit but from months ago. A commit from the future w.r.t. the branch we are building now. The chance of <pkg>-rebuild working correctly is quite small. There will undoubtedly be many other packages which have changed between the last build on this agent and the once being attempted. In addition to the above, preserving the build tree between jobs on a agent ends up being a major obstacle in Bamboo. Trying to archive and restore the multi-gigabyte output is very slow. And given two git hashes to a tree with a buildroot + br-extern, what packages exactly must be rebuilt with <pkg>-rebuild to go from one hash to the other? This is also not so easy to find automatically. So, my latest attempt to solve this problem - getting back quicker CI feedback - was done another way. What I do is build an SDK with buildroot. This is part of a nightly build process that also produces a firmware image and runs system tests against real hardware, in addition to making an SDK. Two SDKs in fact, one for the real ARM target and another for an x86_64 target. Each internal package has a CI job which fetches the latest SDK and builds the package, pointing cmake or autoconf or whatever build system the package uses at the SDK. Since only the package is built, it can produce feedback quickly. The toolchain and libraries and so on is all from the SDK and so the same as in the actual buildroot build of the entire firmware image. The x86_64 SDK has the same package versions as the ARM firmware, made via the same buildroot. Because the target is x86_64, the binaries it produces can be run locally on the x86_64 build agents. I do this to allow running the unit tests of a package quickly in CI, with needing to use real hardware or qemu to run an ARM unit test suite. I make a new package infra for our packages to try to handle them all in a consistent way. A complete .mk file for one might look like this: TRANSPORT_PROXY_DEPENDENCIES = zeromq lttng-libust $(eval $(impinj-cmake-package)) Given the SDK, there's really nothing else I need from buildroot, and the above .mk file, to build this package in CI.
> From: "Trent Piepho" <tpiepho@impinj.com> > To: "Thomas Petazzoni" <thomas.petazzoni@bootlin.com>, chemobejk@gmail.com > Cc: "Yann Morin" <yann.morin.1998@free.fr>, "buildroot" <buildroot@buildroot.org> > Sent: Segunda-feira, 17 de dezembro de 2018 18:08:58 > Subject: Re: [Buildroot] [PATCH RESEND] core: enhance printvars for variables with newlines > On Sun, 2018-12-16 at 16:14 +0100, Thomas Petazzoni wrote: >> >> > Understandable from the PoV of monolithic meta build system advocates >> > like buildroot, yocto, Android, ... >> > >> > As it became clear that this patch would not be accepted any time >> > soon, I reverted it and implemented a different approach based on >> > quoted var printout and sed post-processing. While not 100% >> > fool-proof, it does work for the few build variables that are required >> > for the modular build approach. >> >> You did not answer my question: why didn't you use features of >> Buildroot such as "make <pkg>-build" or "make <pkg>-rebuild" ? > > I am also using buildroot with modern style CI. While I find <pkg>- > rebuild works great for manual developer workflow, there are issues > with CI integration. > > In the CI system, we have many many git branches active at any time. > They come and go frequently and don't start at the same branch point. > There are also many build agents (themselves VMs) which perform the > actual build. > > So say we have a new branch we wish to test. The agent it is > dispatched to happens to have the output tree from the last successful > build on that agent (this is in itself very problematic!). That last > build might be from anything in the git tree. The head of a different > branch. An ancestor commit but from months ago. A commit from the > future w.r.t. the branch we are building now. > > The chance of <pkg>-rebuild working correctly is quite small. There > will undoubtedly be many other packages which have changed between the > last build on this agent and the once being attempted. > > In addition to the above, preserving the build tree between jobs on a > agent ends up being a major obstacle in Bamboo. Trying to archive and > restore the multi-gigabyte output is very slow. > > And given two git hashes to a tree with a buildroot + br-extern, what > packages exactly must be rebuilt with <pkg>-rebuild to go from one hash > to the other? This is also not so easy to find automatically. You may be interested on these threads: http://lists.busybox.net/pipermail/buildroot/2015-October/143107.html http://lists.busybox.net/pipermail/buildroot/2017-October/204165.html > So, my latest attempt to solve this problem - getting back quicker CI > feedback - was done another way. > > What I do is build an SDK with buildroot. This is part of a nightly > build process that also produces a firmware image and runs system tests > against real hardware, in addition to making an SDK. Two SDKs in fact, > one for the real ARM target and another for an x86_64 target. > > Each internal package has a CI job which fetches the latest SDK and > builds the package, pointing cmake or autoconf or whatever build system > the package uses at the SDK. Since only the package is built, it can > produce feedback quickly. The toolchain and libraries and so on is > all from the SDK and so the same as in the actual buildroot build of > the entire firmware image. > > The x86_64 SDK has the same package versions as the ARM firmware, made > via the same buildroot. Because the target is x86_64, the binaries it > produces can be run locally on the x86_64 build agents. I do this to > allow running the unit tests of a package quickly in CI, with needing > to use real hardware or qemu to run an ARM unit test suite. > > I make a new package infra for our packages to try to handle them all > in a consistent way. A complete .mk file for one might look like this: > > TRANSPORT_PROXY_DEPENDENCIES = zeromq lttng-libust > $(eval $(impinj-cmake-package)) > > Given the SDK, there's really nothing else I need from buildroot, and > the above .mk file, to build this package in CI. If you need an SDK perhaps you should try Yocto, which in its turn also has some drawbacks. We (DATACOM) attempted to use it years ago and gave up because the development process was too complicated and slow. The main drawback of an SDK is that as you integrate new packages and change the base system the SDK becomes outdated. It may work for a few packages but as we started adding modules (currently there are 295) it became impossible to build and publish new SDK versions at the same pace without going crazy updating our development environments every day.
On Mon, 2018-12-17 at 22:24 -0200, Carlos Santos wrote: > > > > What I do is build an SDK with buildroot. This is part of a nightly > > build process that also produces a firmware image and runs system tests > > against real hardware, in addition to making an SDK. Two SDKs in fact, > > one for the real ARM target and another for an x86_64 target. > > > > Each internal package has a CI job which fetches the latest SDK and > > builds the package, pointing cmake or autoconf or whatever build system > > the package uses at the SDK. Since only the package is built, it can > > produce feedback quickly. The toolchain and libraries and so on is > > all from the SDK and so the same as in the actual buildroot build of > > the entire firmware image. > > > > The x86_64 SDK has the same package versions as the ARM firmware, made > > via the same buildroot. Because the target is x86_64, the binaries it > > produces can be run locally on the x86_64 build agents. I do this to > > allow running the unit tests of a package quickly in CI, with needing > > to use real hardware or qemu to run an ARM unit test suite. > > > > I make a new package infra for our packages to try to handle them all > > in a consistent way. A complete .mk file for one might look like this: > > > > TRANSPORT_PROXY_DEPENDENCIES = zeromq lttng-libust > > $(eval $(impinj-cmake-package)) > > > > Given the SDK, there's really nothing else I need from buildroot, and > > the above .mk file, to build this package in CI. > > If you need an SDK perhaps you should try Yocto, which in its turn also > has some drawbacks. We (DATACOM) attempted to use it years ago and gave > up because the development process was too complicated and slow. I'm familiar with yocto, but chose buildroot for similar reasons. Buildroot has an SDK feature and it works pretty well for what I want. It's not an alternative to using buildroot, where you make a binary BSP, check it into a repository, and the forever use and patch the binary BSP and never use buildroot again. I am not a fan of that design. > The main drawback of an SDK is that as you integrate new packages and > change the base system the SDK becomes outdated. It may work for a few > packages but as we started adding modules (currently there are 295) it > became impossible to build and publish new SDK versions at the same pace > without going crazy updating our development environments every day. The key here is that the SDK is created by buildroot from the very same repository that creates the firmware. If your buildroot build is up to date, then your SDK is up to date too. What the SDK really is, is a way to make binary artifact of an entire buildroot build, then use that artifact to rebuild one single package without rebuilding the rest of buildroot.
diff --git a/Makefile b/Makefile index 0724f28f45..3e27195de0 100644 --- a/Makefile +++ b/Makefile @@ -974,9 +974,13 @@ printvars: $(sort $(if $(VARS),$(filter $(VARS),$(.VARIABLES)),$(.VARIABLES))), \ $(if $(filter-out environment% default automatic, \ $(origin $V)), \ - $(if $(QUOTED_VARS),\ - $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ - $(info $V=$(if $(RAW_VARS),$(value $V),$($V)))))) + $(if $(DEFINE_VARS), \ + $(info define $V) \ + $(info $(if $(RAW_VARS),$(value $V),$($V))) \ + $(info endef), \ + $(if $(QUOTED_VARS),\ + $(info $V='$(subst ','\'',$(if $(RAW_VARS),$(value $V),$($V)))'), \ + $(info $V=$(if $(RAW_VARS),$(value $V),$($V))))))) # ' Syntax colouring... .PHONY: clean diff --git a/docs/manual/make-tips.txt b/docs/manual/make-tips.txt index ea1d825bef..ba87e5d873 100644 --- a/docs/manual/make-tips.txt +++ b/docs/manual/make-tips.txt @@ -92,6 +92,8 @@ It is possible to tweak the output using some variables: - +VARS+ will limit the listing to variables which names match the specified make-pattern +- +DEFINE_VARS+, if set to +YES+, will use define...endef to preserve + newlines in the value - +QUOTED_VARS+, if set to +YES+, will single-quote the value - +RAW_VARS+, if set to +YES+, will print the unexpanded value @@ -106,6 +108,24 @@ For example: BUSYBOX_RDEPENDENCIES=ncurses util-linux ---- +---- + $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES DEFINE_VARS=YES + define BUSYBOX_DEPENDENCIES + skeleton toolchain + endef + define BUSYBOX_FINAL_ALL_DEPENDENCIES + skeleton toolchain + endef + define BUSYBOX_FINAL_DEPENDENCIES + skeleton toolchain + endef + define BUSYBOX_FINAL_PATCH_DEPENDENCIES + endef + define BUSYBOX_RDEPENDENCIES + ncurses util-linux'linux-pam skeleton toolchain host-skeleton host-ccache + endef +---- + ---- $ make -s printvars VARS=BUSYBOX_%DEPENDENCIES QUOTED_VARS=YES BUSYBOX_DEPENDENCIES='skeleton toolchain'
If the variable content has newlines in it then the currently dumped content can't be fed again to GNU make. Add the option DEFINE_VARS which causes the variables to be dumped using define VAR ... line 1 ... ... line 2 ... ... endef Updated the manual accordingly. Signed-off-by: Stefan Becker <chemobejk@gmail.com> --- Makefile | 10 +++++++--- docs/manual/make-tips.txt | 20 ++++++++++++++++++++ 2 files changed, 27 insertions(+), 3 deletions(-)