| Message ID | 20210126065757.403853-1-thuth@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | gitlab-ci.yml: Avoid recompiling the sources in the test jobs | expand |
On 1/26/21 7:57 AM, Thomas Huth wrote: > Currently, our check-system-* jobs are recompiling the whole sources > again. This happens due to the fact that the jobs are checking out > the whole source tree and required submodules again, and only try > to use the "build" directory with the binaries and object files as an > artifact from the previous stage - which simply does not work right > anymore (with the current version of meson). Due to some changed > time stamps, meson/ninja are always trying to rebuild the whole tree. > > In the long run, we could likely use "meson test --no-rebuild", but > there is still some work going on in that area to improve the user > experience. So until this has been done, simply avoid recompiling the > sources with a trick: pass NINJA=":" to the make process in the test > jobs. Also check out the submodules manually before updating the > timestamps in the build folder, so that the binaries are definitely > newer that all the source files. > This saves ca. 10 - 15 minutes of precious CI cycles in each run. > > Suggested-by: Paolo Bonzini <pbonzini@redhat.com> > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > .gitlab-ci.yml | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index de3a3d25b5..16fea25ba9 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -38,9 +38,11 @@ include: > stage: test > image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest > script: > + - scripts/git-submodule.sh update > + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') > - cd build > - find . -type f -exec touch {} + > - - make $MAKE_CHECK_ARGS > + - make NINJA=":" $MAKE_CHECK_ARGS This ninja trick deserves a comment in the YAML file. Otherwise: Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> > > .acceptance_template: &acceptance_definition > cache: >
On 26/01/2021 08.36, Philippe Mathieu-Daudé wrote: > On 1/26/21 7:57 AM, Thomas Huth wrote: >> Currently, our check-system-* jobs are recompiling the whole sources >> again. This happens due to the fact that the jobs are checking out >> the whole source tree and required submodules again, and only try >> to use the "build" directory with the binaries and object files as an >> artifact from the previous stage - which simply does not work right >> anymore (with the current version of meson). Due to some changed >> time stamps, meson/ninja are always trying to rebuild the whole tree. >> >> In the long run, we could likely use "meson test --no-rebuild", but >> there is still some work going on in that area to improve the user >> experience. So until this has been done, simply avoid recompiling the >> sources with a trick: pass NINJA=":" to the make process in the test >> jobs. Also check out the submodules manually before updating the >> timestamps in the build folder, so that the binaries are definitely >> newer that all the source files. >> This saves ca. 10 - 15 minutes of precious CI cycles in each run. >> >> Suggested-by: Paolo Bonzini <pbonzini@redhat.com> >> Signed-off-by: Thomas Huth <thuth@redhat.com> >> --- >> .gitlab-ci.yml | 4 +++- >> 1 file changed, 3 insertions(+), 1 deletion(-) >> >> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml >> index de3a3d25b5..16fea25ba9 100644 >> --- a/.gitlab-ci.yml >> +++ b/.gitlab-ci.yml >> @@ -38,9 +38,11 @@ include: >> stage: test >> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest >> script: >> + - scripts/git-submodule.sh update >> + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') >> - cd build >> - find . -type f -exec touch {} + >> - - make $MAKE_CHECK_ARGS >> + - make NINJA=":" $MAKE_CHECK_ARGS > > This ninja trick deserves a comment in the YAML file. I'll add: # Avoid recompiling by hiding ninja with NINJA=":" Ok? Thomas
On 1/26/21 10:08 AM, Thomas Huth wrote: > On 26/01/2021 08.36, Philippe Mathieu-Daudé wrote: >> On 1/26/21 7:57 AM, Thomas Huth wrote: >>> Currently, our check-system-* jobs are recompiling the whole sources >>> again. This happens due to the fact that the jobs are checking out >>> the whole source tree and required submodules again, and only try >>> to use the "build" directory with the binaries and object files as an >>> artifact from the previous stage - which simply does not work right >>> anymore (with the current version of meson). Due to some changed >>> time stamps, meson/ninja are always trying to rebuild the whole tree. >>> >>> In the long run, we could likely use "meson test --no-rebuild", but >>> there is still some work going on in that area to improve the user >>> experience. So until this has been done, simply avoid recompiling the >>> sources with a trick: pass NINJA=":" to the make process in the test >>> jobs. Also check out the submodules manually before updating the >>> timestamps in the build folder, so that the binaries are definitely >>> newer that all the source files. >>> This saves ca. 10 - 15 minutes of precious CI cycles in each run. >>> >>> Suggested-by: Paolo Bonzini <pbonzini@redhat.com> >>> Signed-off-by: Thomas Huth <thuth@redhat.com> >>> --- >>> .gitlab-ci.yml | 4 +++- >>> 1 file changed, 3 insertions(+), 1 deletion(-) >>> >>> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml >>> index de3a3d25b5..16fea25ba9 100644 >>> --- a/.gitlab-ci.yml >>> +++ b/.gitlab-ci.yml >>> @@ -38,9 +38,11 @@ include: >>> stage: test >>> image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest >>> script: >>> + - scripts/git-submodule.sh update >>> + $(grep GIT_SUBMODULES build/config-host.mak | sed >>> 's/GIT_SUBMODULES=//') >>> - cd build >>> - find . -type f -exec touch {} + >>> - - make $MAKE_CHECK_ARGS >>> + - make NINJA=":" $MAKE_CHECK_ARGS >> >> This ninja trick deserves a comment in the YAML file. > > I'll add: > > # Avoid recompiling by hiding ninja with NINJA=":" > > Ok? Perfect, thanks! Phil.
On 1/26/21 3:57 AM, Thomas Huth wrote: > Currently, our check-system-* jobs are recompiling the whole sources > again. This happens due to the fact that the jobs are checking out > the whole source tree and required submodules again, and only try > to use the "build" directory with the binaries and object files as an > artifact from the previous stage - which simply does not work right > anymore (with the current version of meson). Due to some changed > time stamps, meson/ninja are always trying to rebuild the whole tree. > > In the long run, we could likely use "meson test --no-rebuild", but > there is still some work going on in that area to improve the user > experience. So until this has been done, simply avoid recompiling the > sources with a trick: pass NINJA=":" to the make process in the test > jobs. Also check out the submodules manually before updating the > timestamps in the build folder, so that the binaries are definitely > newer that all the source files. > This saves ca. 10 - 15 minutes of precious CI cycles in each run. Thanks for working on this! > > Suggested-by: Paolo Bonzini <pbonzini@redhat.com> > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > .gitlab-ci.yml | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index de3a3d25b5..16fea25ba9 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -38,9 +38,11 @@ include: > stage: test > image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest > script: > + - scripts/git-submodule.sh update > + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') > - cd build > - find . -type f -exec touch {} + > - - make $MAKE_CHECK_ARGS > + - make NINJA=":" $MAKE_CHECK_ARGS > > .acceptance_template: &acceptance_definition > cache:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index de3a3d25b5..16fea25ba9 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -38,9 +38,11 @@ include: stage: test image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest script: + - scripts/git-submodule.sh update + $(grep GIT_SUBMODULES build/config-host.mak | sed 's/GIT_SUBMODULES=//') - cd build - find . -type f -exec touch {} + - - make $MAKE_CHECK_ARGS + - make NINJA=":" $MAKE_CHECK_ARGS .acceptance_template: &acceptance_definition cache:
Currently, our check-system-* jobs are recompiling the whole sources again. This happens due to the fact that the jobs are checking out the whole source tree and required submodules again, and only try to use the "build" directory with the binaries and object files as an artifact from the previous stage - which simply does not work right anymore (with the current version of meson). Due to some changed time stamps, meson/ninja are always trying to rebuild the whole tree. In the long run, we could likely use "meson test --no-rebuild", but there is still some work going on in that area to improve the user experience. So until this has been done, simply avoid recompiling the sources with a trick: pass NINJA=":" to the make process in the test jobs. Also check out the submodules manually before updating the timestamps in the build folder, so that the binaries are definitely newer that all the source files. This saves ca. 10 - 15 minutes of precious CI cycles in each run. Suggested-by: Paolo Bonzini <pbonzini@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com> --- .gitlab-ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)