Message ID | 20181228104335.22379-1-thomas.petazzoni@bootlin.com |
---|---|
Headers | show |
Series | Top-level parallel build support | expand |
Hello, On Fri, 28 Dec 2018 11:43:27 +0100, Thomas Petazzoni wrote: > Here is a seventh iteration of the per-package SDK and target > directory implementation, offering top-level parallel build > capability. Regarding top-level parallel build, I prepared this Wiki page that documents the progress of the effort: https://elinux.org/Buildroot:Top_Level_Parallel_Build Especially, I'd like to make it clear: this patch series is only the *core* support for per-package directories. There are a number of fixes needed for Meson, Python and more to make it usable with some packages. I wanted to keep this patch series reasonable in size by only submitting the core support. If you want to do some testing, please use https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7-work which has all the fixes I currently have. Best regards, Thomas
Hi Thomas, finally I found some time to try this v7 on top of 2019.02-rc1. Am 28.12.18 um 18:21 schrieb Thomas Petazzoni: > Hello, > > On Fri, 28 Dec 2018 11:43:27 +0100, Thomas Petazzoni wrote: > >> Here is a seventh iteration of the per-package SDK and target >> directory implementation, offering top-level parallel build >> capability. > > Regarding top-level parallel build, I prepared this Wiki page that > documents the progress of the effort: > > https://elinux.org/Buildroot:Top_Level_Parallel_Build > My update to the known remaining issues is: 1. I no longer see the ccache problem for the kconfig packages (uboot+linux) 2. the package-file-list problem has become a blocker since commit 3c8f0d9 core/pkg-infra: restore completeness of packages files lists with failures like comm: /home/jenkins/.../build/.files-list-staging.new: No such file or directory Would it be an idea to write the statistics per package :-) inside the package build folder and assemble them into one file only once all packages are built? best regards, Andreas PS. Actually I just tried flock, which looks promising. If it works I'll send a patch next week. > Especially, I'd like to make it clear: this patch series is only the > *core* support for per-package directories. There are a number of fixes > needed for Meson, Python and more to make it usable with some packages. > I wanted to keep this patch series reasonable in size by only > submitting the core support. > > If you want to do some testing, please use > https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7-work > which has all the fixes I currently have. > > Best regards, > > Thomas >
Hi All, On Fri, Feb 22, 2019 at 6:19 PM Andreas Naumann <dev@andin.de> wrote: > > Hi Thomas, > > finally I found some time to try this v7 on top of 2019.02-rc1. > > Am 28.12.18 um 18:21 schrieb Thomas Petazzoni: > > Hello, > > > > On Fri, 28 Dec 2018 11:43:27 +0100, Thomas Petazzoni wrote: > > > >> Here is a seventh iteration of the per-package SDK and target > >> directory implementation, offering top-level parallel build > >> capability. > > > > Regarding top-level parallel build, I prepared this Wiki page that > > documents the progress of the effort: > > > > https://elinux.org/Buildroot:Top_Level_Parallel_Build > > > > My update to the known remaining issues is: > 1. I no longer see the ccache problem for the kconfig packages > (uboot+linux) > > 2. the package-file-list problem has become a blocker since commit > 3c8f0d9 core/pkg-infra: restore completeness of packages files lists > with failures like > > comm: /home/jenkins/.../build/.files-list-staging.new: No such file or > directory > > Would it be an idea to write the statistics per package :-) inside the > package build folder and assemble them into one file only once all > packages are built? > > best regards, > Andreas > > > > > PS. Actually I just tried flock, which looks promising. If it works I'll > send a patch next week. > I 'd try it too, but the provided URL does not work :-( Regards, Vadim Kochan
Vadim,
On Fri, 22 Feb 2019 20:07:37 +0200
Vadim Kochan <vadim4j@gmail.com> wrote:
> I 'd try it too, but the provided URL does not work :-(
I updated the Wiki page with URLs on Github.com. We just shut down the
git.bootlin.com service a few days ago, so now my personal Buildroot
Git repo is at https://github.com/tpetazzoni/buildroot/.
I'm looking forward to your feedback!
Best regards,
Thomas
Hi Thomas, On Fri, Feb 22, 2019 at 09:29:12PM +0100, Thomas Petazzoni wrote: > Vadim, > > On Fri, 22 Feb 2019 20:07:37 +0200 > Vadim Kochan <vadim4j@gmail.com> wrote: > > > I 'd try it too, but the provided URL does not work :-( > > I updated the Wiki page with URLs on Github.com. We just shut down the > git.bootlin.com service a few days ago, so now my personal Buildroot > Git repo is at https://github.com/tpetazzoni/buildroot/. > > I'm looking forward to your feedback! > I think that in case of per-packages parallel build the logs became messy and meaningless so I send a patch which allows to do per-package logging too: https://patchwork.ozlabs.org/patch/1047549/ this is a POC so it just shows the idea. Regards, Vadim Kochan
Hello Vadim, On Mon, 25 Feb 2019 03:10:03 +0200 Vadim Kochan <vadim4j@gmail.com> wrote: > I think that in case of per-packages parallel build the logs became > messy and meaningless so I send a patch which allows to do per-package > logging too: > > https://patchwork.ozlabs.org/patch/1047549/ > > this is a POC so it just shows the idea. Thanks for proposing this! The problem of logging was discussed quite a lot back when I started to work on per-package directories. I think we investigated overriding the SHELL variable, but couldn't find an approach that worked properly, but maybe the approach was different than yours, I need to get back to the original discussion. Back then, what we concluded is that people could use the "make --output-sync=target" option, or "make -Otarget" in short. This option buffers the output of make on a per-target basis, and displays this output at once when the target has completed. The advantage is that the output is no longer garbled, but the drawback is that you don't see "live" what is happening: if a package takes 20 minutes to build, during 20 minutes you see nothing, and at the end of the 20 minutes, you get in one go the full build output of that package. Best regards, Thomas
Hi Thomas, On Mon, Feb 25, 2019 at 09:05:43AM +0100, Thomas Petazzoni wrote: > Hello Vadim, > > On Mon, 25 Feb 2019 03:10:03 +0200 > Vadim Kochan <vadim4j@gmail.com> wrote: > > > I think that in case of per-packages parallel build the logs became > > messy and meaningless so I send a patch which allows to do per-package > > logging too: > > > > https://patchwork.ozlabs.org/patch/1047549/ > > > > this is a POC so it just shows the idea. > > Thanks for proposing this! > > The problem of logging was discussed quite a lot back when I started to > work on per-package directories. I think we investigated overriding the > SHELL variable, but couldn't find an approach that worked properly, but > maybe the approach was different than yours, I need to get back to the > original discussion. > > Back then, what we concluded is that people could use the "make > --output-sync=target" option, or "make -Otarget" in short. This option > buffers the output of make on a per-target basis, and displays this > output at once when the target has completed. The advantage is that the > output is no longer garbled, but the drawback is that you don't see > "live" what is happening: if a package takes 20 minutes to build, > during 20 minutes you see nothing, and at the end of the 20 minutes, > you get in one go the full build output of that package. > I changed a bit an approach to log output per-package and per-step, so it solves the issue of refreshing log for particular step by simply removing the log file at the beginning of step. Actually these steps logs might be merged into one at the end of make. Regards, Vadim Kochan
Hi Thomas, All On Mon, Feb 25, 2019 at 09:05:43AM +0100, Thomas Petazzoni wrote: > Hello Vadim, > > On Mon, 25 Feb 2019 03:10:03 +0200 > Vadim Kochan <vadim4j@gmail.com> wrote: > > > I think that in case of per-packages parallel build the logs became > > messy and meaningless so I send a patch which allows to do per-package > > logging too: > > > > https://patchwork.ozlabs.org/patch/1047549/ > > > > this is a POC so it just shows the idea. > > Thanks for proposing this! > > The problem of logging was discussed quite a lot back when I started to > work on per-package directories. I think we investigated overriding the > SHELL variable, but couldn't find an approach that worked properly, but > maybe the approach was different than yours, I need to get back to the > original discussion. > > Back then, what we concluded is that people could use the "make > --output-sync=target" option, or "make -Otarget" in short. This option > buffers the output of make on a per-target basis, and displays this > output at once when the target has completed. The advantage is that the > output is no longer garbled, but the drawback is that you don't see > "live" what is happening: if a package takes 20 minutes to build, > during 20 minutes you see nothing, and at the end of the 20 minutes, > you get in one go the full build output of that package. > > Best regards, > I updated the patch: https://patchwork.ozlabs.org/patch/1047620/ where I added per-package-and-per-step logging, I think it is possible to merge them info one log file, but per-step approach allows to easy update logs for perticular step. But I 'd like to clarify if actually such approach make sense ? Also how do you think about to add (using the approach from the patch) prepending logs with a package name prefix (in bold) (not necessary for parellel building but enabled by config option): [${pkg_name}] $log_line after it may be possible to print the percentage of installed packages like: [${percent}][${pkg_name}] $log_line ofcourse all this is just for providing to user at runtime the info which package is building and how many packages (in percentages) are done. So, may be it looks useless, but I feel it would be nice to have. Regards, Vadim Kochan
Vadym, All, On 2019-03-01 16:50 +0200, Vadym Kochan spake thusly: > On Mon, Feb 25, 2019 at 09:05:43AM +0100, Thomas Petazzoni wrote: > > On Mon, 25 Feb 2019 03:10:03 +0200 > > Vadim Kochan <vadim4j@gmail.com> wrote: > > > > > I think that in case of per-packages parallel build the logs became > > > messy and meaningless so I send a patch which allows to do per-package > > > logging too: > > > > > > https://patchwork.ozlabs.org/patch/1047549/ > > > > > > this is a POC so it just shows the idea. > > The problem of logging was discussed quite a lot back when I started to > > work on per-package directories. I think we investigated overriding the > > SHELL variable, but couldn't find an approach that worked properly, but > > maybe the approach was different than yours, I need to get back to the > > original discussion. > > > > Back then, what we concluded is that people could use the "make > > --output-sync=target" option, or "make -Otarget" in short. This option > > buffers the output of make on a per-target basis, and displays this > > output at once when the target has completed. The advantage is that the > > output is no longer garbled, but the drawback is that you don't see > > "live" what is happening: if a package takes 20 minutes to build, > > during 20 minutes you see nothing, and at the end of the 20 minutes, > > you get in one go the full build output of that package. > > I updated the patch: > > https://patchwork.ozlabs.org/patch/1047620/ > > where I added per-package-and-per-step logging, I think it is possible > to merge them info one log file, but per-step approach allows to easy > update logs for perticular step. > > But I 'd like to clarify if actually such approach make sense ? As Thomas said, we already have had some discussion about the logging, and we also investigated using a shell wrapper, and a PoC similar to yours was even attempted. However, we eventually concluded that this was not going to work reliably, so we decided against at that time. I could not find that conclusion in the devdays meeting reports, so I guess it's burried somewhere in the mailing list. I'll try to unearth those later in the evening or the week-end... The first issue I can readily remember, was that the SHELL variable is passed down to all children, and some of them may re-use it to run commands, and the wrapper would make those fail. I don't have the details, though, as I said I'll try to dig them from the archives... But overall, I agree with Thomas, that people that still want nice logs call 'make -Otarget'. > Also how do you think about to add (using the approach from the patch) > prepending logs with a package name prefix (in bold) (not necessary for > parellel building but enabled by config option): > > [${pkg_name}] $log_line > > after it may be possible to print the percentage of installed packages > like: > > [${percent}][${pkg_name}] $log_line > > ofcourse all this is just for providing to user at runtime the info > which package is building and how many packages (in percentages) are > done. So, may be it looks useless, but I feel it would be nice to have. Sorry, but I think this is really getting too far. We already have a simple make wrapper, utils/brmake, which filters the build log and redirects all to a file, and just prints the '>>>' lines to stdout. I think this is as far as we should go with Buildroot. Regards, Yann E. MORIN.
On 01/03/2019 18:18, Yann E. MORIN wrote: > However, we eventually concluded that this was not going to work > reliably, so we decided against at that time. I could not find that > conclusion in the devdays meeting reports, so I guess it's burried > somewhere in the mailing list. I'll try to unearth those later in the > evening or the week-end... If I remember correct, didn't GNU make bypass the SHELL variable for simple commands? Regards, Arnout
On Mon, 4 Mar 2019 08:24:33 +0100 Arnout Vandecappelle <arnout@mind.be> wrote: > On 01/03/2019 18:18, Yann E. MORIN wrote: > > However, we eventually concluded that this was not going to work > > reliably, so we decided against at that time. I could not find that > > conclusion in the devdays meeting reports, so I guess it's burried > > somewhere in the mailing list. I'll try to unearth those later in the > > evening or the week-end... > > If I remember correct, didn't GNU make bypass the SHELL variable for simple > commands? Aah, yes, you have good memory, this definitely rings a bell. Thomas