| Message ID | 20210107171719.477856-1-pbonzini@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC] ci: ensure that all jobs use a shallow clone | expand |
On Thu, Jan 07, 2021 at 06:17:19PM +0100, Paolo Bonzini wrote: > We are still using quite a bit of bandwidth to run CI jobs, even though > GitLab has switched to gitlab.com to fetch the sources. This is in > part because we are handling submodules ourselves and therefore those > do not use shallow clones. I presume the impact is going to vary for GitLab vs the other CI systems. For gitlab, any bandwidth used should be entirely within gitlab's own network infra since its pulling repos from gitlab.com/qemu-project For Cirrus/Shippable/Travis the bandwidth used would be more of a potential performance problem. > Observe GitLab's GIT_DEPTH environment variable in > scripts/git-submodule.sh so that CI systems can use a shallow clone > for submodules. Set a depth of 3 for all CI systems, for both the main > repository (using whatever native configuration mechanism is there) > and the submodules. > > Reported-by: Stefan Hajnoczi <stefanha@redhat.com> > Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> > --- > Completely untested, just to propose the idea. The check-patch and check-dco jobs need the git history depth to be long enough to cover the entire contents of the branch that is being tested vs common ancestor against git master. This could potentially be 100's of commits deep for some of the very largest branches we see merged in qemu Travis has issues with git cloning and concurrent pushes. eg if you push branch A, it schedules a CI job. Then you push branch B before jobs for A have started. When the job for A starts, it will be unable to checkout the commit for branch A's merge because the history will be too shallow. I'm not sure if the other CI systems have similar issues or not. > > .cirrus.yml | 1 + > .gitlab-ci.yml | 3 +++ > .shippable.yml | 2 ++ > .travis.yml | 2 ++ > scripts/git-submodule.sh | 2 +- > 5 files changed, 9 insertions(+), 1 deletion(-) > > diff --git a/.cirrus.yml b/.cirrus.yml > index 6f2a958472..47c2467d04 100644 > --- a/.cirrus.yml > +++ b/.cirrus.yml > @@ -1,5 +1,6 @@ > env: > - CIRRUS_CLONE_DEPTH: 1 > + CIRRUS_CLONE_DEPTH: 3 > + GIT_DEPTH: 3 > > freebsd_12_task: > freebsd_instance: > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 01c9e46410..ff19a9c313 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -7,6 +7,9 @@ stages: > - build > - test > > +variables: > + GIT_DEPTH: 3 > + > include: > - local: '/.gitlab-ci.d/edk2.yml' > - local: '/.gitlab-ci.d/opensbi.yml' > diff --git a/.shippable.yml b/.shippable.yml > index 14350e6de8..f2d59b478e 100644 > --- a/.shippable.yml > +++ b/.shippable.yml > @@ -1,9 +1,11 @@ > language: c > +depth: 3 > git: > submodules: false > env: > global: > - LC_ALL=C > + - GIT_DEPTH=3 > matrix: > - IMAGE=debian-amd64 > TARGET_LIST=x86_64-softmmu,x86_64-linux-user > diff --git a/.travis.yml b/.travis.yml > index f2a101936c..f2a785a42b 100644 > --- a/.travis.yml > +++ b/.travis.yml > @@ -80,9 +80,11 @@ env: > - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime" > - CCACHE_MAXSIZE=1G > - G_MESSAGES_DEBUG=error > + - GIT_DEPTH=3 > > > git: > + depth: 3 > # we want to do this ourselves > submodules: false > > diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh > index 65ed877aef..2886087a12 100755 > --- a/scripts/git-submodule.sh > +++ b/scripts/git-submodule.sh > @@ -75,7 +75,7 @@ update) > exit 0 > fi > > - $GIT submodule update --init $modules 1>/dev/null > + $GIT submodule update ${GIT_DEPTH:+--depth "$GIT_DEPTH"} --init $modules 1>/dev/null > test $? -ne 0 && error "failed to update modules" > > $GIT submodule status $modules > "${substat}" > -- > 2.29.2 > > Regards, Daniel
On 07/01/2021 19.28, Daniel P. Berrangé wrote: [...] > Travis has issues with git cloning and concurrent pushes. > > eg if you push branch A, it schedules a CI job. Then you > push branch B before jobs for A have started. > > When the job for A starts, it will be unable to checkout > the commit for branch A's merge because the history will > be too shallow. I think we have to get rid of travis.yml soon anyway, since they changed their business model, and soon it won't be possible anymore to use it for QEMU. travis-ci.org is getting decommissioned and on travis-ci.com you can only get free CI minutes for non-sponsored FOSS projects. So let's simply not worry about Travis-CI anymore. >> .cirrus.yml | 1 + >> .gitlab-ci.yml | 3 +++ >> .shippable.yml | 2 ++ Maybe we could rather disable shippable now that we support the cross container builds on gitlab-ci, too? Thomas
Il gio 7 gen 2021, 20:05 Thomas Huth <thuth@redhat.com> ha scritto: > on travis-ci.com you can > only get free CI minutes for non-sponsored FOSS projects. > So let's simply not worry about Travis-CI anymore. > > Maybe we could rather disable shippable now that we support the cross > container builds on gitlab-ci, too? > With pleasure, starting this discussion was an intended possible side effect of the patch. :) The main issue with Travis is the non-x86 builders, which have no alternative yet. Paolo > Thomas > >
On Thu, Jan 07, 2021 at 08:23:49PM +0100, Paolo Bonzini wrote: > Il gio 7 gen 2021, 20:05 Thomas Huth <thuth@redhat.com> ha scritto: > > > on travis-ci.com you can > > only get free CI minutes for non-sponsored FOSS projects. > > So let's simply not worry about Travis-CI anymore. > > > > Maybe we could rather disable shippable now that we support the cross > > container builds on gitlab-ci, too? > > > > With pleasure, starting this discussion was an intended possible side > effect of the patch. :) > > The main issue with Travis is the non-x86 builders, which have no > alternative yet. The free travis accounts get 10000 credits. Linux jobs take 10 credits per minute of execution time. Free credits are a one-off grant which don't auto renew. You have to make an email request to travis to beg for more each time they run out. The non-x86 CI jobs we have in travis consume 91 minutes of time in total. So we'll burn through all free credits in 11 jobs and then our CI will stop. IOW, despite travis giving us non-x86 builders, it is doomed to be unusuable, unless we can convince them to give us a *massively* larger free credit allowance on the qemu account. It would need to be on the order x100 larger, and auto-renewing once a month to cope with our rate of builds for the non-x86 CI jobs. I think we need to just delete the travis config and accept that we can't run those jobs unless we provide our own hardware for non-x86 in GitLab CI. IIUC, the latter is something we're planning anyway. So I support removing travis and shippable configs and focusing only on GitLab and Cirrus Regards, Daniel
On 08/01/2021 11.16, Daniel P. Berrangé wrote: > IOW, despite travis giving us non-x86 builders, it is doomed to be > unusuable, unless we can convince them to give us a *massively* > larger free credit allowance on the qemu account. I think convincing them to do this will be very hard. I've tried to apply to the free OSS credits for my account, and in the mail that I've got back after a couple of weeks, they wrote that "Project must not be sponsored by a commercial company or organization (monetary or with employees paid to work on the project)" to qualify for the free credits. That doesn't sound like they'll accept QEMU, I think. Thomas
On Fri, Jan 08, 2021 at 12:06:01PM +0100, Thomas Huth wrote: > On 08/01/2021 11.16, Daniel P. Berrangé wrote: > > > IOW, despite travis giving us non-x86 builders, it is doomed to be > > unusuable, unless we can convince them to give us a *massively* > > larger free credit allowance on the qemu account. > I think convincing them to do this will be very hard. I've tried to apply to > the free OSS credits for my account, and in the mail that I've got back > after a couple of weeks, they wrote that "Project must not be sponsored by a > commercial company or organization (monetary or with employees paid to work > on the project)" to qualify for the free credits. That doesn't sound like > they'll accept QEMU, I think. Yeah that makes their OSS program unviable for any project which isn't essentially just a single person free time effort. Lets drop Travis and not waste any more time on this. Regards, Daniel
diff --git a/.cirrus.yml b/.cirrus.yml index 6f2a958472..47c2467d04 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -1,5 +1,6 @@ env: - CIRRUS_CLONE_DEPTH: 1 + CIRRUS_CLONE_DEPTH: 3 + GIT_DEPTH: 3 freebsd_12_task: freebsd_instance: diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 01c9e46410..ff19a9c313 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -7,6 +7,9 @@ stages: - build - test +variables: + GIT_DEPTH: 3 + include: - local: '/.gitlab-ci.d/edk2.yml' - local: '/.gitlab-ci.d/opensbi.yml' diff --git a/.shippable.yml b/.shippable.yml index 14350e6de8..f2d59b478e 100644 --- a/.shippable.yml +++ b/.shippable.yml @@ -1,9 +1,11 @@ language: c +depth: 3 git: submodules: false env: global: - LC_ALL=C + - GIT_DEPTH=3 matrix: - IMAGE=debian-amd64 TARGET_LIST=x86_64-softmmu,x86_64-linux-user diff --git a/.travis.yml b/.travis.yml index f2a101936c..f2a785a42b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -80,9 +80,11 @@ env: - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime" - CCACHE_MAXSIZE=1G - G_MESSAGES_DEBUG=error + - GIT_DEPTH=3 git: + depth: 3 # we want to do this ourselves submodules: false diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh index 65ed877aef..2886087a12 100755 --- a/scripts/git-submodule.sh +++ b/scripts/git-submodule.sh @@ -75,7 +75,7 @@ update) exit 0 fi - $GIT submodule update --init $modules 1>/dev/null + $GIT submodule update ${GIT_DEPTH:+--depth "$GIT_DEPTH"} --init $modules 1>/dev/null test $? -ne 0 && error "failed to update modules" $GIT submodule status $modules > "${substat}"
We are still using quite a bit of bandwidth to run CI jobs, even though GitLab has switched to gitlab.com to fetch the sources. This is in part because we are handling submodules ourselves and therefore those do not use shallow clones. Observe GitLab's GIT_DEPTH environment variable in scripts/git-submodule.sh so that CI systems can use a shallow clone for submodules. Set a depth of 3 for all CI systems, for both the main repository (using whatever native configuration mechanism is there) and the submodules. Reported-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> --- Completely untested, just to propose the idea. .cirrus.yml | 1 + .gitlab-ci.yml | 3 +++ .shippable.yml | 2 ++ .travis.yml | 2 ++ scripts/git-submodule.sh | 2 +- 5 files changed, 9 insertions(+), 1 deletion(-)