[RFC,v2,1/1] utils/show-progress: add tool to show build progress at runtime
diff mbox series

Message ID 20190718073243.10767-1-vadim4j@gmail.com
State Changes Requested
Headers show
Series
  • [RFC,v2,1/1] utils/show-progress: add tool to show build progress at runtime
Related show

Commit Message

Vadim Kochan July 18, 2019, 7:32 a.m. UTC
This might be useful to watch the amount of built and selected
packages and some progress about it. So added python script which
prints progress and build stats. The sample of output is:

Press Ctrl-C to exit ...

Building: output/qemu_x86_64
 ##################################------------------------------ [ 42.42% 14/33]

Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
---

RFC v2:
    1) filter rootfs package type
    2) rename build-progress -> show-progress
    3) check empty version

 utils/show-progress | 91 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
 create mode 100755 utils/show-progress

Comments

Yann E. MORIN Aug. 4, 2019, 3:51 p.m. UTC | #1
Vadim, All,

On 2019-07-18 10:32 +0300, Vadim Kochan spake thusly:
> This might be useful to watch the amount of built and selected
> packages and some progress about it. So added python script which
> prints progress and build stats. The sample of output is:
> 
> Press Ctrl-C to exit ...
> 
> Building: output/qemu_x86_64
>  ##################################------------------------------ [ 42.42% 14/33]

With this defconfig:
    BR2_arm=y
    BR2_cortex_a7=y
    BR2_TOOLCHAIN_EXTERNAL=y
    # BR2_TARGET_ROOTFS_TAR is not set

and building out-of-tree, the script never reaches 100%:

    $ ../buildroot/utils/show-progress
    Press Ctrl-C to exit ...

    Building: .
     #########################################################################------- [ 91.67% 11/12]

Even though the build did finish...

> Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
[--SNIP--]
> diff --git a/utils/show-progress b/utils/show-progress
> new file mode 100755
> index 0000000000..ee1e1da289
> --- /dev/null
> +++ b/utils/show-progress
> @@ -0,0 +1,91 @@
> +#!/usr/bin/env python

This is actually a python3 script, and is not a valid pyhon2 one.

> +# Copyright (C) 2019 Vadim Kochan <vadim4j@gmail.com>

There is no reason nowadays to add copyright notices. The copyright is
defacto by the Bern convention. Furthermore, in an opensource project,
this notice is wrong as ssoon as other contributors do substantial
modification to the file.

And the git log is a much better way to extract authroship of a file.

[--SNIP--]
> +def progress(ready, total):
> +    perc = ready / total
> +    fill = round(perc * 80)
> +    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {}/{} ]'.format(perc, ready, total), end='')

I've locally changed it to:

    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {: >2d}/{}]'.format(perc, ready, total), end='')

> +    print("Press Ctrl-C to exit ...\n")
> +    print("Building: " + build_dir)

This print '.' as the build directory when doing an out-of-tree build
(see above). I've locally changed it to:

    print("Building: " + os.path.realpath(build_dir))

Regards,
Yann E. MORIN.
Vadim Kochan Aug. 6, 2019, 8:54 a.m. UTC | #2
Yann,

On Sun, Aug 4, 2019 at 6:52 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> Vadim, All,
>
> On 2019-07-18 10:32 +0300, Vadim Kochan spake thusly:
> > This might be useful to watch the amount of built and selected
> > packages and some progress about it. So added python script which
> > prints progress and build stats. The sample of output is:
> >
> > Press Ctrl-C to exit ...
> >
> > Building: output/qemu_x86_64
> >  ##################################------------------------------ [ 42.42% 14/33]
>
> With this defconfig:
>     BR2_arm=y
>     BR2_cortex_a7=y
>     BR2_TOOLCHAIN_EXTERNAL=y
>     # BR2_TARGET_ROOTFS_TAR is not set
>
> and building out-of-tree, the script never reaches 100%:
>
>     $ ../buildroot/utils/show-progress
>     Press Ctrl-C to exit ...
>
>     Building: .
>      #########################################################################------- [ 91.67% 11/12]
>
> Even though the build did finish...
>
> > Signed-off-by: Vadim Kochan <vadim4j@gmail.com>
> [--SNIP--]
> > diff --git a/utils/show-progress b/utils/show-progress
> > new file mode 100755
> > index 0000000000..ee1e1da289
> > --- /dev/null
> > +++ b/utils/show-progress
> > @@ -0,0 +1,91 @@
> > +#!/usr/bin/env python
>
> This is actually a python3 script, and is not a valid pyhon2 one.
>
> > +# Copyright (C) 2019 Vadim Kochan <vadim4j@gmail.com>
>
> There is no reason nowadays to add copyright notices. The copyright is
> defacto by the Bern convention. Furthermore, in an opensource project,
> this notice is wrong as ssoon as other contributors do substantial
> modification to the file.
>
> And the git log is a much better way to extract authroship of a file.
>
> [--SNIP--]
> > +def progress(ready, total):
> > +    perc = ready / total
> > +    fill = round(perc * 80)
> > +    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {}/{} ]'.format(perc, ready, total), end='')
>
> I've locally changed it to:
>
>     print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {: >2d}/{}]'.format(perc, ready, total), end='')
>
> > +    print("Press Ctrl-C to exit ...\n")
> > +    print("Building: " + build_dir)
>
> This print '.' as the build directory when doing an out-of-tree build
> (see above). I've locally changed it to:
>
>     print("Building: " + os.path.realpath(build_dir))
>
> Regards,
> Yann E. MORIN.
>

Yes, sorry for wasting your time. It really does not work if there is
package which has no installation
for target but only installed to the staging. I will fix this case after
"install_" properties will be merged to the "show-info" make helper.

I forgot to mark this patch for ignoring.

Regards,
Vadim Kochan

Patch
diff mbox series

diff --git a/utils/show-progress b/utils/show-progress
new file mode 100755
index 0000000000..ee1e1da289
--- /dev/null
+++ b/utils/show-progress
@@ -0,0 +1,91 @@ 
+#!/usr/bin/env python
+
+# Copyright (C) 2019 Vadim Kochan <vadim4j@gmail.com>
+
+import os
+import sys
+import json
+import time
+import logging
+import subprocess
+
+do_exit = False
+
+def usage():
+    print("""Usage: show-progress [-h] [PATH]
+show-progress shows progress about selected and built packages.
+If PATH is no specified then the current working one will be used as
+build output directory.
+
+Example usage:
+ $ show-progress output/qemu_x86_64
+
+""")
+    sys.exit(0)
+
+def get_pkgs_list(build_dir):
+    cmd = ["make", "-C", build_dir, "-s", "--no-print-directory", "show-info"]
+    results = []
+
+    with open(os.devnull, 'wb') as devnull:
+        p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=devnull,
+                             universal_newlines=True)
+        pkg_list = json.loads(p.communicate()[0])
+
+        for p in pkg_list:
+            if pkg_list[p]["type"] == "rootfs":
+                continue
+
+            ver = "" if pkg_list[p]["virtual"] else pkg_list[p]["version"]
+            ver = "-" + ver if ver != "" else ver
+
+            results.append(p + ver)
+
+    return results
+
+def is_pkg_built(build_dir, p):
+    for s in ["host", "target"]:
+        if os.path.exists(build_dir + "/build/" + p + "/.stamp_" + s + "_installed"):
+            return True
+
+    return False
+
+def progress(ready, total):
+    perc = ready / total
+    fill = round(perc * 80)
+    print('\r', '#' * fill + '-' * (80 - fill), '[{:>7.2%} {}/{} ]'.format(perc, ready, total), end='')
+    sys.stdout.flush()
+
+def main():
+    build_dir = "."
+
+    if "-h" in sys.argv or "--help" in sys.argv:
+        usage()
+
+    if len(sys.argv) > 1:
+        build_dir = sys.argv[1]
+
+    pkgs = get_pkgs_list(build_dir)
+    total = len(pkgs)
+
+    print("Press Ctrl-C to exit ...\n")
+    print("Building: " + build_dir)
+
+    while not do_exit:
+        ready = 0
+
+        for p in pkgs:
+            if is_pkg_built(build_dir, p):
+                ready = ready + 1
+
+        progress(ready, total)
+        time.sleep(1)
+
+        if ready == total:
+            print("Done")
+            break
+
+try:
+    main()
+except KeyboardInterrupt:
+    do_exit = True