| Message ID | 1474238918-1797-7-git-send-email-sjg@chromium.org |
|---|---|
| State | Accepted |
| Commit | 21f0eb332faddeb88a0f3e9788f83b6f9c22071c |
| Delegated to: | Simon Glass |
| Headers | show |
On Sun, Sep 18, 2016 at 04:48:31PM -0600, Simon Glass wrote: > On a machine with a lot of CPUs this prints a lot of useless lines of the > form: > > Cloning repo for thread <n> > > Adjust the output so that these all appear on one line, and disappear when > the cloning is complete. > > Note: This cloning is actually unnecessary and very wasteful on disk space > (about 3.5GB each time). It would be better to create symlinks. When do we actually need to clone? Or why, even?
Hi Tom, On 19 September 2016 at 13:14, Tom Rini <trini@konsulko.com> wrote: > On Sun, Sep 18, 2016 at 04:48:31PM -0600, Simon Glass wrote: > >> On a machine with a lot of CPUs this prints a lot of useless lines of the >> form: >> >> Cloning repo for thread <n> >> >> Adjust the output so that these all appear on one line, and disappear when >> the cloning is complete. >> >> Note: This cloning is actually unnecessary and very wasteful on disk space >> (about 3.5GB each time). It would be better to create symlinks. > > When do we actually need to clone? Or why, even? Each thread can check out a different commit. I think it might be possible to put the checkout in a separate thread so it can avoid contention. I'm not quite sure... Regards, Simon
On Thu, Sep 22, 2016 at 10:14:01PM -0600, Simon Glass wrote: > Hi Tom, > > On 19 September 2016 at 13:14, Tom Rini <trini@konsulko.com> wrote: > > On Sun, Sep 18, 2016 at 04:48:31PM -0600, Simon Glass wrote: > > > >> On a machine with a lot of CPUs this prints a lot of useless lines of the > >> form: > >> > >> Cloning repo for thread <n> > >> > >> Adjust the output so that these all appear on one line, and disappear when > >> the cloning is complete. > >> > >> Note: This cloning is actually unnecessary and very wasteful on disk space > >> (about 3.5GB each time). It would be better to create symlinks. > > > > When do we actually need to clone? Or why, even? > > Each thread can check out a different commit. I think it might be > possible to put the checkout in a separate thread so it can avoid > contention. I'm not quite sure... Ah, ok. So we have a few places under tools/ that should be doing 'git clone --local', at first glance.
Hi Tom, On 23 September 2016 at 12:42, Tom Rini <trini@konsulko.com> wrote: > On Thu, Sep 22, 2016 at 10:14:01PM -0600, Simon Glass wrote: >> Hi Tom, >> >> On 19 September 2016 at 13:14, Tom Rini <trini@konsulko.com> wrote: >> > On Sun, Sep 18, 2016 at 04:48:31PM -0600, Simon Glass wrote: >> > >> >> On a machine with a lot of CPUs this prints a lot of useless lines of the >> >> form: >> >> >> >> Cloning repo for thread <n> >> >> >> >> Adjust the output so that these all appear on one line, and disappear when >> >> the cloning is complete. >> >> >> >> Note: This cloning is actually unnecessary and very wasteful on disk space >> >> (about 3.5GB each time). It would be better to create symlinks. >> > >> > When do we actually need to clone? Or why, even? >> >> Each thread can check out a different commit. I think it might be >> possible to put the checkout in a separate thread so it can avoid >> contention. I'm not quite sure... > > Ah, ok. So we have a few places under tools/ that should be doing 'git > clone --local', at first glance. Actually I think I am wrong about this. The 3.5GB is due to 32 copies of the code, not the cloning. In fact --local is the default and that is what buildman seems to use. So in fact I'm not sure we can change it. Each thread can be building a different commit. Regards, Simon
On Fri, Sep 23, 2016 at 03:09:53PM -0600, Simon Glass wrote: > Hi Tom, > > On 23 September 2016 at 12:42, Tom Rini <trini@konsulko.com> wrote: > > On Thu, Sep 22, 2016 at 10:14:01PM -0600, Simon Glass wrote: > >> Hi Tom, > >> > >> On 19 September 2016 at 13:14, Tom Rini <trini@konsulko.com> wrote: > >> > On Sun, Sep 18, 2016 at 04:48:31PM -0600, Simon Glass wrote: > >> > > >> >> On a machine with a lot of CPUs this prints a lot of useless lines of the > >> >> form: > >> >> > >> >> Cloning repo for thread <n> > >> >> > >> >> Adjust the output so that these all appear on one line, and disappear when > >> >> the cloning is complete. > >> >> > >> >> Note: This cloning is actually unnecessary and very wasteful on disk space > >> >> (about 3.5GB each time). It would be better to create symlinks. > >> > > >> > When do we actually need to clone? Or why, even? > >> > >> Each thread can check out a different commit. I think it might be > >> possible to put the checkout in a separate thread so it can avoid > >> contention. I'm not quite sure... > > > > Ah, ok. So we have a few places under tools/ that should be doing 'git > > clone --local', at first glance. > > Actually I think I am wrong about this. The 3.5GB is due to 32 copies > of the code, not the cloning. In fact --local is the default and that > is what buildman seems to use. > > So in fact I'm not sure we can change it. Each thread can be building > a different commit. Ah, OK, oh well.
On 18 September 2016 at 16:48, Simon Glass <sjg@chromium.org> wrote: > On a machine with a lot of CPUs this prints a lot of useless lines of the > form: > > Cloning repo for thread <n> > > Adjust the output so that these all appear on one line, and disappear when > the cloning is complete. > > Note: This cloning is actually unnecessary and very wasteful on disk space > (about 3.5GB each time). It would be better to create symlinks. > > Signed-off-by: Simon Glass <sjg@chromium.org> > --- > > tools/buildman/builder.py | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) Applied to u-boot-dm.
diff --git a/tools/buildman/builder.py b/tools/buildman/builder.py index 2aca189..7c237a5 100644 --- a/tools/buildman/builder.py +++ b/tools/buildman/builder.py @@ -1379,8 +1379,10 @@ class Builder: if os.path.exists(git_dir): gitutil.Fetch(git_dir, thread_dir) else: - Print('Cloning repo for thread %d' % thread_num) + Print('\rCloning repo for thread %d' % thread_num, + newline=False) gitutil.Clone(src_dir, thread_dir) + Print('\r%s\r' % (' ' * 30), newline=False) def _PrepareWorkingSpace(self, max_threads, setup_git): """Prepare the working directory for use.
On a machine with a lot of CPUs this prints a lot of useless lines of the form: Cloning repo for thread <n> Adjust the output so that these all appear on one line, and disappear when the cloning is complete. Note: This cloning is actually unnecessary and very wasteful on disk space (about 3.5GB each time). It would be better to create symlinks. Signed-off-by: Simon Glass <sjg@chromium.org> --- tools/buildman/builder.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)