| Message ID | 388a8da9b48e30391dd6f2cd35be53c2dafc4af0.1497794488.git.yann.morin.1998@free.fr |
|---|---|
| State | Changes Requested |
| Headers | show |
Hello, On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote: > Add a tools directory to store useful little scripts, tools, etc... > Users may add that directory in their PATH if they want to, but this > is not mandatory. > > Add the first tool: brmake, a small script that redirects the build > output log to a file, keeping just Buildroot's own messages, with the > date+time added at the start of the line. > > We need to unbuffer the output of make so that, when the script is > interrupted (SIGINT, ^C), there is no lingering output not yet digested > by the logger loop. > > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> > Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com> > Cc: Arnout Vandecappelle <arnout@mind.be> I tested the script and it works very well in general. - I successfully built the qemu_arm_versatile_defconfig using the script, pressing CTRL+C a few times and resuming by calling it again. - Also, by inserting an error to a package I got the expected behaviour: the error message is in the log file and the script returned the error code. - Tested using 'GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)' BUT if it is called without a .config file, it hangs forever waiting for input, and the menuconfig is sent to the log file. Maybe we can bailout if no .config is present. Also, 2 small nits below. > +++ b/tools/00-README.txt > @@ -0,0 +1,9 @@ > +This directory contains various useful scripts and tools for working > +with Buildroot. You need not add this directory in your PATH to use > +any of those tools, but you may do so if you want. > + > +brmake > + a script that can be run instead of make, that prepends the date in > + front of each line, redirects all of the build output to a file > + ("'br.log' in the current directory), and just ouputs the Buildroot Maybe this file will be dropped as discussed in http://patchwork.ozlabs.org/patch/777446/ If not, there is a typo ouputs -> outputs > + messages (those lines starting with >>>) on stdout. > diff --git a/tools/brmake b/tools/brmake > new file mode 100755 > index 0000000000..9f7b4bc65f > --- /dev/null > +++ b/tools/brmake > @@ -0,0 +1,37 @@ > +#!/bin/bash > +# (C) 2016, "Yann E. MORIN" <yann.morin.1998@free.fr> > +# License: WTFPL, https://spdx.org/licenses/WTFPL.html > + > +main() { > + local ret start d h m mf fs fs is unused. Regards, Ricardo
Ricardo, All, On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly: > On Sun, Jun 18, 2017 at 11:01 AM, Yann E. MORIN wrote: > > Add a tools directory to store useful little scripts, tools, etc... > > Users may add that directory in their PATH if they want to, but this > > is not mandatory. > > > > Add the first tool: brmake, a small script that redirects the build > > output log to a file, keeping just Buildroot's own messages, with the > > date+time added at the start of the line. > > > > We need to unbuffer the output of make so that, when the script is > > interrupted (SIGINT, ^C), there is no lingering output not yet digested > > by the logger loop. > > > > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> > > Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com> > > Cc: Arnout Vandecappelle <arnout@mind.be> > > I tested the script and it works very well in general. > > - I successfully built the qemu_arm_versatile_defconfig using the script, > pressing CTRL+C a few times and resuming by calling it again. > - Also, by inserting an error to a package I got the expected behaviour: the > error message is in the log file and the script returned the error code. > - Tested using 'GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)' > > BUT if it is called without a .config file, it hangs forever waiting for input, > and the menuconfig is sent to the log file. Indeed, my use-case is to only use it in an already-configured tree. > Maybe we can bailout if no .config is present. Hmmm... It's not trivial do detect, because the .config file is not necessarily in the current directory, for example: brmake -C /pat/to/buil-dir and I don't want to write a make option interpeter either... But I'll see what I can do. > Also, 2 small nits below. > > > +++ b/tools/00-README.txt > > @@ -0,0 +1,9 @@ > > +This directory contains various useful scripts and tools for working > > +with Buildroot. You need not add this directory in your PATH to use > > +any of those tools, but you may do so if you want. > > + > > +brmake > > + a script that can be run instead of make, that prepends the date in > > + front of each line, redirects all of the build output to a file > > + ("'br.log' in the current directory), and just ouputs the Buildroot > > Maybe this file will be dropped as discussed in > http://patchwork.ozlabs.org/patch/777446/ > If not, there is a typo > > ouputs -> outputs Yup, I already dropped the file locally. > > + messages (those lines starting with >>>) on stdout. > > diff --git a/tools/brmake b/tools/brmake > > new file mode 100755 > > index 0000000000..9f7b4bc65f > > --- /dev/null > > +++ b/tools/brmake > > @@ -0,0 +1,37 @@ > > +#!/bin/bash > > +# (C) 2016, "Yann E. MORIN" <yann.morin.1998@free.fr> > > +# License: WTFPL, https://spdx.org/licenses/WTFPL.html > > + > > +main() { > > + local ret start d h m mf fs > > fs is unused. OK. Thanks for the review and tests! :-) Regards, Yann E. MORIN. > Regards, > Ricardo
Hello, On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote: > On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly: [snip] >> BUT if it is called without a .config file, it hangs forever waiting for input, >> and the menuconfig is sent to the log file. > > Indeed, my use-case is to only use it in an already-configured tree. > >> Maybe we can bailout if no .config is present. > > Hmmm... It's not trivial do detect, because the .config file is not > necessarily in the current directory, for example: > > brmake -C /pat/to/buil-dir > > and I don't want to write a make option interpeter either... Oh, I see now it is not trivial. > > But I'll see what I can do. Only on the first use the user can get confused. I think adding a usage example in the file header is enough, saying the script must be run inside or pointing to (-C from make) an output directory already configured, or something like that. [snip] >> > + a script that can be run instead of make, that prepends the date in >> > + front of each line, redirects all of the build output to a file >> > + ("'br.log' in the current directory), and just ouputs the Buildroot >> >> Maybe this file will be dropped as discussed in >> http://patchwork.ozlabs.org/patch/777446/ >> If not, there is a typo >> >> ouputs -> outputs > > Yup, I already dropped the file locally. [snip] The purpose of the script (initially in 00-README.txt, with the typo fixed) could be placed in the script header as well. Regards, Ricardo
Ricardo, All, On 2017-06-20 23:06 -0300, Ricardo Martincoski spake thusly: > On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote: > > On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly: > [snip] > >> BUT if it is called without a .config file, it hangs forever waiting for input, > >> and the menuconfig is sent to the log file. > > > > Indeed, my use-case is to only use it in an already-configured tree. > > > >> Maybe we can bailout if no .config is present. > > > > Hmmm... It's not trivial do detect, because the .config file is not > > necessarily in the current directory, for example: > > > > brmake -C /pat/to/buil-dir > > > > and I don't want to write a make option interpeter either... > > Oh, I see now it is not trivial. Yet I think I was able to do it in a minimalist way that should cover 99.999999% of the cases (or so I hope, famous last words): https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/contrib&id=221c72431bb12e0dde4051d9317d9fee2bee6eb2 I'll resubmit soon (after a bit more testing...) > Only on the first use the user can get confused. > I think adding a usage example in the file header is enough, saying the script > must be run inside or pointing to (-C from make) an output directory already > configured, or something like that. > > [snip] > >> > + a script that can be run instead of make, that prepends the date in > >> > + front of each line, redirects all of the build output to a file > >> > + ("'br.log' in the current directory), and just ouputs the Buildroot > >> > >> Maybe this file will be dropped as discussed in > >> http://patchwork.ozlabs.org/patch/777446/ > >> If not, there is a typo > >> > >> ouputs -> outputs > > > > Yup, I already dropped the file locally. > [snip] > > The purpose of the script (initially in 00-README.txt, with the typo fixed) > could be placed in the script header as well. Yep, good idea. I'll do that. Thanks! Regards, Yann E. MORIN.
On 21-06-17 07:21, Yann E. MORIN wrote: > Ricardo, All, > > On 2017-06-20 23:06 -0300, Ricardo Martincoski spake thusly: >> On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote: >>> On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly: >> [snip] >>>> BUT if it is called without a .config file, it hangs forever waiting for input, >>>> and the menuconfig is sent to the log file. >>> >>> Indeed, my use-case is to only use it in an already-configured tree. >>> >>>> Maybe we can bailout if no .config is present. >>> >>> Hmmm... It's not trivial do detect, because the .config file is not >>> necessarily in the current directory, for example: >>> >>> brmake -C /pat/to/buil-dir >>> >>> and I don't want to write a make option interpeter either... >> >> Oh, I see now it is not trivial. How about instead make the script do make -C ${0%/*}/.. O=${PWD} "${@}" if $PWD is not the Buildroot dir. This still allows overriding either option on the command line. Not that this solves the issue... > Yet I think I was able to do it in a minimalist way that should cover > 99.999999% of the cases (or so I hope, famous last words): > https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/contrib&id=221c72431bb12e0dde4051d9317d9fee2bee6eb2 I haven't tried, but doesn't it do the wrong thing with make O=... -C ... ? But it still doesn't completely solve the issue, because "brmake menuconfig" still does the wrong thing... Unfortunately I don't know how to solve that (other than detecting menuconfig as an argument, obviously). On a related note, perhaps it's better to redirect stdin in case there is something interactive during the build (e.g. Qt license approval). Regards, Arnout > I'll resubmit soon (after a bit more testing...) > >> Only on the first use the user can get confused. >> I think adding a usage example in the file header is enough, saying the script >> must be run inside or pointing to (-C from make) an output directory already >> configured, or something like that. >> >> [snip] >>>>> + a script that can be run instead of make, that prepends the date in >>>>> + front of each line, redirects all of the build output to a file >>>>> + ("'br.log' in the current directory), and just ouputs the Buildroot >>>> >>>> Maybe this file will be dropped as discussed in >>>> http://patchwork.ozlabs.org/patch/777446/ >>>> If not, there is a typo >>>> >>>> ouputs -> outputs >>> >>> Yup, I already dropped the file locally. >> [snip] >> >> The purpose of the script (initially in 00-README.txt, with the typo fixed) >> could be placed in the script header as well. > > Yep, good idea. I'll do that. > > Thanks! > > Regards, > Yann E. MORIN. >
Arnout, All, On 2017-06-21 23:44 +0200, Arnout Vandecappelle spake thusly: > On 21-06-17 07:21, Yann E. MORIN wrote: > > Ricardo, All, > > > > On 2017-06-20 23:06 -0300, Ricardo Martincoski spake thusly: > >> On Mon, Jun 19, 2017 at 12:42 PM, Yann E. MORIN wrote: > >>> On 2017-06-18 22:11 -0300, Ricardo Martincoski spake thusly: > >> [snip] > >>>> BUT if it is called without a .config file, it hangs forever waiting for input, > >>>> and the menuconfig is sent to the log file. > >>> > >>> Indeed, my use-case is to only use it in an already-configured tree. > >>> > >>>> Maybe we can bailout if no .config is present. > >>> > >>> Hmmm... It's not trivial do detect, because the .config file is not > >>> necessarily in the current directory, for example: > >>> > >>> brmake -C /pat/to/buil-dir > >>> > >>> and I don't want to write a make option interpeter either... > >> > >> Oh, I see now it is not trivial. > > How about instead make the script do > > make -C ${0%/*}/.. O=${PWD} "${@}" What if one calls it with: brmake -C /path/to/my/O-dir But as much as we try to 'fix' it, we always uncover corner cases, becausze the above would consider ${PWD} to be the output dir if /path/to/my/O-dir is not already configured... :-/ Corner cases after corner cases, I think I'd just try to catch the obvious and trivial, and leave out the weird cases. So, either there is a .config in the CWD, or we abort. Deal? ;-) Regards, Yann E. MORIN.
diff --git a/docs/manual/writing-rules.txt b/docs/manual/writing-rules.txt index 3a2c8dcb37..c144874a0d 100644 --- a/docs/manual/writing-rules.txt +++ b/docs/manual/writing-rules.txt @@ -146,5 +146,6 @@ syntax, refer to http://www.methods.co.nz/asciidoc/userguide.html[]. === Support scripts -Some scripts in the +support/+ directory are written in Python and should follow -the https://www.python.org/dev/peps/pep-0008/[PEP8 Style Guide for Python Code]. +Some scripts in the +support/+ and +tools/+ directories are written in +Python and should follow the +https://www.python.org/dev/peps/pep-0008/[PEP8 Style Guide for Python Code]. diff --git a/tools/00-README.txt b/tools/00-README.txt new file mode 100644 index 0000000000..ffc0fbf5ef --- /dev/null +++ b/tools/00-README.txt @@ -0,0 +1,9 @@ +This directory contains various useful scripts and tools for working +with Buildroot. You need not add this directory in your PATH to use +any of those tools, but you may do so if you want. + +brmake + a script that can be run instead of make, that prepends the date in + front of each line, redirects all of the build output to a file + ("'br.log' in the current directory), and just ouputs the Buildroot + messages (those lines starting with >>>) on stdout. diff --git a/tools/brmake b/tools/brmake new file mode 100755 index 0000000000..9f7b4bc65f --- /dev/null +++ b/tools/brmake @@ -0,0 +1,37 @@ +#!/bin/bash +# (C) 2016, "Yann E. MORIN" <yann.morin.1998@free.fr> +# License: WTFPL, https://spdx.org/licenses/WTFPL.html + +main() { + local ret start d h m mf fs + + if ! which unbuffer >/dev/null 2>&1; then + printf "you need to install 'unbuffer' (from package expect or expect-dev)\n" >&2 + exit 1 + fi + + start=${SECONDS} + + ( exec 2>&1; unbuffer make "${@}"; ) \ + > >( while read line; do + printf "%(%Y-%m-%dT%H:%M:%S)T %s\n" -1 "${line}" + done \ + |tee -a br.log \ + |grep --colour=never -E '>>>' + ) + ret=${?} + + d=$((SECONDS-start)) + printf "Done in " + h=$((d/3600)) + d=$((d%3600)) + [ ${h} -eq 0 ] || { printf "%dh " ${h}; mf="02"; } + m=$((d/60)) + d=$((d%60)) + [ ${m} -eq 0 ] || { printf "%${mf}dmin " ${m}; sf="02"; } + printf "%${sf}ds\n" ${d} + + return ${ret} +} + +main "${@}"
Add a tools directory to store useful little scripts, tools, etc... Users may add that directory in their PATH if they want to, but this is not mandatory. Add the first tool: brmake, a small script that redirects the build output log to a file, keeping just Buildroot's own messages, with the date+time added at the start of the line. We need to unbuffer the output of make so that, when the script is interrupted (SIGINT, ^C), there is no lingering output not yet digested by the logger loop. Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com> Cc: Arnout Vandecappelle <arnout@mind.be> --- Changes v1 -> v2: - rename contrib -> tools (Arnout) - detect missing unbuffer (Arnout) Notes: - Arnout suggested using 'stdbuf' instead of 'unbuffer', but that has some shortcomings in weird corner-cases because it uses LD_PRELOAD to call a constructor that forces settings on stdout/stderr/stdin (it cost me three days to investigate a build failure because of that). So, I stuck with 'unbuffer'... - I also tried to use bash' coproc, but it seems there was a regression in later bash versions that causes an incorrect redirect. So, I stuck with 'unbuffer'... --- docs/manual/writing-rules.txt | 5 +++-- tools/00-README.txt | 9 +++++++++ tools/brmake | 37 +++++++++++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 2 deletions(-) create mode 100644 tools/00-README.txt create mode 100755 tools/brmake