| Message ID | 20240308150748.894699-1-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | [RFC] gitlab: aggressively avoid extra GIT data | expand |
On 08/03/2024 16.07, Alex Bennée wrote: > This avoids fetching blobs and tree references for branches we are not > going to worry about. Also skip tag references which are similarly not > useful and keep the default --prune. This keeps the .git data to > around 100M rather than the ~400M even a shallow clone takes. > > So we can check the savings we also run a quick du while setting up > the build. > > We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the > Windows build (git too old?) and the migration legacy test where we > build an older QEMU alongside the main one. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > --- > .gitlab-ci.d/base.yml | 4 ++++ > .gitlab-ci.d/buildtest-template.yml | 1 + > .gitlab-ci.d/buildtest.yml | 2 ++ > .gitlab-ci.d/windows.yml | 2 ++ > 4 files changed, 9 insertions(+) > > diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml > index ef173a34e6..7cb800fc3b 100644 > --- a/.gitlab-ci.d/base.yml > +++ b/.gitlab-ci.d/base.yml > @@ -24,6 +24,10 @@ variables: > # Each script line from will be in a collapsible section in the job output > # and show the duration of each line. > FF_SCRIPT_SECTIONS: 1 > + # The project has a fairly fat GIT repo so we try and avoid bringing in things > + # we don't need. The --filter options avoid blobs and tree references we aren't going to use > + # and we also avoid fetching tags. > + GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune The original settings included --quiet according to the docs: https://docs.gitlab.com/ee/ci/runners/configure_runners.html#git-fetch-extra-flags Should we include --quite here, too? Thomas
diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index ef173a34e6..7cb800fc3b 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -24,6 +24,10 @@ variables: # Each script line from will be in a collapsible section in the job output # and show the duration of each line. FF_SCRIPT_SECTIONS: 1 + # The project has a fairly fat GIT repo so we try and avoid bringing in things + # we don't need. The --filter options avoid blobs and tree references we aren't going to use + # and we also avoid fetching tags. + GIT_FETCH_EXTRA_FLAGS: --filter=blob:none --filter=tree:0 --no-tags --prune interruptible: true diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index 4fbfeb6667..22045add80 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -14,6 +14,7 @@ - export CCACHE_DIR="$CCACHE_BASEDIR/ccache" - export CCACHE_MAXSIZE="500M" - export PATH="$CCACHE_WRAPPERSDIR:$PATH" + - du -sh .git - mkdir build - cd build - ccache --zero-stats diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index a1c030337b..def469a9dc 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -187,6 +187,8 @@ build-previous-qemu: variables: IMAGE: opensuse-leap TARGETS: x86_64-softmmu aarch64-softmmu + # Override the default flags as we need more to grab the old version + GIT_FETCH_EXTRA_FLAGS: --prune before_script: - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)" - git remote add upstream https://gitlab.com/qemu-project/qemu diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml index f116b8012d..f9da745042 100644 --- a/.gitlab-ci.d/windows.yml +++ b/.gitlab-ci.d/windows.yml @@ -28,6 +28,8 @@ msys2-64bit: # qTests don't run successfully with "--without-default-devices", # so let's exclude the qtests from CI for now. TEST_ARGS: --no-suite qtest + # The Windows git is a bit older so override the default + GIT_FETCH_EXTRA_FLAGS: --no-tags --prune artifacts: name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG" expire_in: 7 days
This avoids fetching blobs and tree references for branches we are not going to worry about. Also skip tag references which are similarly not useful and keep the default --prune. This keeps the .git data to around 100M rather than the ~400M even a shallow clone takes. So we can check the savings we also run a quick du while setting up the build. We also have to have special settings of GIT_FETCH_EXTRA_FLAGS for the Windows build (git too old?) and the migration legacy test where we build an older QEMU alongside the main one. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- .gitlab-ci.d/base.yml | 4 ++++ .gitlab-ci.d/buildtest-template.yml | 1 + .gitlab-ci.d/buildtest.yml | 2 ++ .gitlab-ci.d/windows.yml | 2 ++ 4 files changed, 9 insertions(+)