| Message ID | 20230227151110.31455-2-farosas@suse.de |
|---|---|
| State | New |
| Headers | show |
| Series | ci: Speed up container stage | expand |
Fabiano Rosas <farosas@suse.de> writes: > Our dockerfiles no longer reference layers from other qemu images so > we can now use 'docker build' on them. > > Also reinstate the caching that was disabled due to bad interactions > with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use > BUILDKIT in GitLab either"). We now believe those issues to be fixed. > > The COMMON_TAG needed to be fixed for the caching to work. The > docker.py script was not using the variable, but constructing the > correct URL directly. > > Signed-off-by: Fabiano Rosas <farosas@suse.de> > --- > .gitlab-ci.d/container-template.yml | 9 ++++----- > 1 file changed, 4 insertions(+), 5 deletions(-) > > diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml > index c434b9c8f3..519b8a9482 100644 > --- a/.gitlab-ci.d/container-template.yml > +++ b/.gitlab-ci.d/container-template.yml > @@ -6,17 +6,16 @@ > - docker:dind > before_script: > - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" > + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > - apk add python3 > - docker info > - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > script: > - echo "TAG:$TAG" > - echo "COMMON_TAG:$COMMON_TAG" > - - ./tests/docker/docker.py --engine docker build > - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" > - -r $CI_REGISTRY/qemu-project/qemu > - - docker tag "qemu/$NAME" "$TAG" > + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" > + --build-arg BUILDKIT_INLINE_CACHE=1 > + -f "tests/docker/dockerfiles/$NAME.docker" "." I wonder why this doesn't injest a bunch of context. If I run: docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" . it attempts to bring my entire build directory in as build context. This is why we use the - < docker form in the Makefile. > - docker push "$TAG" > after_script: > - docker logout So what I don't understand is if I do: docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - < /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker I still see pretty much a full rebuild of the image.
Alex Bennée <alex.bennee@linaro.org> writes: > Fabiano Rosas <farosas@suse.de> writes: > >> Our dockerfiles no longer reference layers from other qemu images so >> we can now use 'docker build' on them. >> >> Also reinstate the caching that was disabled due to bad interactions >> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use >> BUILDKIT in GitLab either"). We now believe those issues to be fixed. >> >> The COMMON_TAG needed to be fixed for the caching to work. The >> docker.py script was not using the variable, but constructing the >> correct URL directly. >> >> Signed-off-by: Fabiano Rosas <farosas@suse.de> >> --- >> .gitlab-ci.d/container-template.yml | 9 ++++----- >> 1 file changed, 4 insertions(+), 5 deletions(-) >> >> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml >> index c434b9c8f3..519b8a9482 100644 >> --- a/.gitlab-ci.d/container-template.yml >> +++ b/.gitlab-ci.d/container-template.yml >> @@ -6,17 +6,16 @@ >> - docker:dind >> before_script: >> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" >> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" >> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" >> - apk add python3 >> - docker info >> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" >> script: >> - echo "TAG:$TAG" >> - echo "COMMON_TAG:$COMMON_TAG" >> - - ./tests/docker/docker.py --engine docker build >> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" >> - -r $CI_REGISTRY/qemu-project/qemu >> - - docker tag "qemu/$NAME" "$TAG" >> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" >> + --build-arg BUILDKIT_INLINE_CACHE=1 >> + -f "tests/docker/dockerfiles/$NAME.docker" "." > > I wonder why this doesn't injest a bunch of context. If I run: > > docker build --cache-from > registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg > BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t > qemu > /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" . > > it attempts to bring my entire build directory in as build context. This > is why we use the - < docker form in the Makefile. > >> - docker push "$TAG" >> after_script: >> - docker logout > > So what I don't understand is if I do: > > docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross > docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg > BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - < > /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker > > I still see pretty much a full rebuild of the image. For reference at the end: 🕙19:37:36 alex@zen:container-diff.git on master took 19s ➜ ./out/container-diff diff daemon://registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross daemon://qemu/debian-alpha-cross --type=file -----File----- These entries have been added to qemu/debian-alpha-cross: FILE SIZE /usr/share/doc/base-files/FAQ 6B /usr/share/info/dir 781B These entries have been deleted from qemu/debian-alpha-cross: None These entries have been changed between registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross and qemu/debian-alpha-cross: FILE SIZE1 SIZE2 /var/cache/debconf/templates.dat 762.2K 764.1K /var/cache/debconf/templates.dat-old 762.2K 764.1K /var/lib/dpkg/available 70.7K 70.7K /var/log/dpkg.log 12.9K 20.2K /var/log/apt/term.log 7.4K 15.5K /var/cache/ldconfig/aux-cache 5.5K 5.5K /var/cache/debconf/config.dat-old 4.5K 4.7K /var/log/apt/history.log 1.5K 2K /etc/shadow 501B 501B /etc/shadow- 501B 501B /etc/apt/sources.list 430B 430B
Alex Bennée <alex.bennee@linaro.org> writes: > Fabiano Rosas <farosas@suse.de> writes: > >> Our dockerfiles no longer reference layers from other qemu images so >> we can now use 'docker build' on them. >> >> Also reinstate the caching that was disabled due to bad interactions >> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use >> BUILDKIT in GitLab either"). We now believe those issues to be fixed. >> >> The COMMON_TAG needed to be fixed for the caching to work. The >> docker.py script was not using the variable, but constructing the >> correct URL directly. >> >> Signed-off-by: Fabiano Rosas <farosas@suse.de> >> --- >> .gitlab-ci.d/container-template.yml | 9 ++++----- >> 1 file changed, 4 insertions(+), 5 deletions(-) >> >> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml >> index c434b9c8f3..519b8a9482 100644 >> --- a/.gitlab-ci.d/container-template.yml >> +++ b/.gitlab-ci.d/container-template.yml >> @@ -6,17 +6,16 @@ >> - docker:dind >> before_script: >> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" >> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" >> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" >> - apk add python3 >> - docker info >> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" >> script: >> - echo "TAG:$TAG" >> - echo "COMMON_TAG:$COMMON_TAG" >> - - ./tests/docker/docker.py --engine docker build >> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" >> - -r $CI_REGISTRY/qemu-project/qemu >> - - docker tag "qemu/$NAME" "$TAG" >> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" >> + --build-arg BUILDKIT_INLINE_CACHE=1 >> + -f "tests/docker/dockerfiles/$NAME.docker" "." > > I wonder why this doesn't injest a bunch of context. If I run: > > docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu > /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" . > > it attempts to bring my entire build directory in as build context. This > is why we use the - < docker form in the Makefile. > I only see that without DOCKER_BUILDKIT=1. With the variable set it does like in the CI jobs. I presume it is being set automatically by gitlab, but we could add it to the script to be explicit. >> - docker push "$TAG" >> after_script: >> - docker logout > > So what I don't understand is if I do: > > docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross > docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg > BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - < > /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker > > I still see pretty much a full rebuild of the image. I don't use docker and podman does not support caching. I have fresh-installed docker today and indeed it seems to not use the cache at every build. We're missing something. Sometimes it works: === $ docker system prune -a -f Deleted build cache objects: xzbxmzaib3a8s0ufetop5ikhi 0ce9qln4ipd2vgf9xw9to0fdb se4hq3rce3lad20t9sqqnubob Total reclaimed space: 5.845kB $ docker images -a REPOSITORY TAG IMAGE ID CREATED SIZE $ DOCKER_BUILDKIT=1 docker build --tag registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --cache-from registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --build-arg BUILDKIT_INLINE_CACHE=1 -f debian-amd64.docker . [+] Building 57.8s (12/12) FINISHED => [internal] load build definition from debian-amd64.docker 0.2s => => transferring dockerfile: 5.90kB 0.0s => [internal] load .dockerignore 0.3s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/debian:11-slim 1.8s => importing cache manifest from registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest 0.7s => [1/6] FROM docker.io/library/debian:11-slim@sha256:8eaee63a5ea83744e62d5bf88e7d472d7f19b5feda3bfc6a2304cc074f269269 0.0s => CACHED [2/6] RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -y eatmydata && eatmydata apt-get dist-upgrade -y && eatmydata apt-get i 0.0s => CACHED [3/6] RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -y --no-install-recommends cscope global linux-headers-amd64 0.0s => CACHED [4/6] RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap 0.0s => CACHED [5/6] RUN cd /usr/src/netmap && git checkout v11.3 0.0s => CACHED [6/6] RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install 54.9s => => pulling sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa 1.2s => => pulling sha256:f63092d28e024c119d6a4e257d0c62bb2117be93605f1f2363b1d9cb8fc9dc6c 44.6s => => pulling sha256:30fcedea9de9f22da43e20a79b40516c6e4a55e70eea401acacca18912e86769 4.5s => => pulling sha256:6f498daa6d20055a31e4681ec878e86129c3c1eaf5cf40f7ed87631accef1906 3.6s => => pulling sha256:79b1b0d7bce9a16b4a01e2055cd402d93e04efded50e2ffcf6be0d7c67e4895a 0.6s => => pulling sha256:250b927c5ce2d6d3e9d24bf085f649ce951966c9ec9c9e00ab8292dfb02c8b17 0.7s => exporting to image 0.1s => => exporting layers 0.0s => => writing image sha256:3663223c0a31c2be678c2fe110ba284b80b8d04d2d3290c14be7a4d7320f8a67 0.0s => => naming to registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest 0.0s => exporting cache 0.0s => => preparing build cache for export === There are issues pointing to some general unreliability of the cache: https://github.com/moby/buildkit/issues/2274 https://github.com/moby/buildkit/issues/1981 What is your docker version? Mine is 20.10.17-ce and Gitlab's is 23.0.1. The issues above are on 20.10.11.
Fabiano Rosas <farosas@suse.de> writes: > Alex Bennée <alex.bennee@linaro.org> writes: > >> Fabiano Rosas <farosas@suse.de> writes: >> >>> Our dockerfiles no longer reference layers from other qemu images so >>> we can now use 'docker build' on them. >>> >>> Also reinstate the caching that was disabled due to bad interactions >>> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use >>> BUILDKIT in GitLab either"). We now believe those issues to be fixed. >>> >>> The COMMON_TAG needed to be fixed for the caching to work. The >>> docker.py script was not using the variable, but constructing the >>> correct URL directly. >>> >>> Signed-off-by: Fabiano Rosas <farosas@suse.de> >>> --- >>> .gitlab-ci.d/container-template.yml | 9 ++++----- >>> 1 file changed, 4 insertions(+), 5 deletions(-) >>> >>> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml >>> index c434b9c8f3..519b8a9482 100644 >>> --- a/.gitlab-ci.d/container-template.yml >>> +++ b/.gitlab-ci.d/container-template.yml >>> @@ -6,17 +6,16 @@ >>> - docker:dind >>> before_script: >>> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" >>> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" >>> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" >>> - apk add python3 >>> - docker info >>> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" >>> script: >>> - echo "TAG:$TAG" >>> - echo "COMMON_TAG:$COMMON_TAG" >>> - - ./tests/docker/docker.py --engine docker build >>> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" >>> - -r $CI_REGISTRY/qemu-project/qemu >>> - - docker tag "qemu/$NAME" "$TAG" >>> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" >>> + --build-arg BUILDKIT_INLINE_CACHE=1 >>> + -f "tests/docker/dockerfiles/$NAME.docker" "." >> >> I wonder why this doesn't injest a bunch of context. If I run: >> >> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu >> /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" . >> >> it attempts to bring my entire build directory in as build context. This >> is why we use the - < docker form in the Makefile. >> > > I only see that without DOCKER_BUILDKIT=1. With the variable set it does > like in the CI jobs. I presume it is being set automatically by gitlab, > but we could add it to the script to be explicit. > >>> - docker push "$TAG" >>> after_script: >>> - docker logout >> >> So what I don't understand is if I do: >> >> docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross >> docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg >> BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - < >> /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker >> >> I still see pretty much a full rebuild of the image. > > I don't use docker and podman does not support caching. I have > fresh-installed docker today and indeed it seems to not use the cache at > every build. We're missing something. > > Sometimes it works: > Oops, sorry about the long lines, here it is again: === $ docker system prune -a -f Deleted build cache objects: xzbxmzaib3a8s0ufetop5ikhi 0ce9qln4ipd2vgf9xw9to0fdb se4hq3rce3lad20t9sqqnubob Total reclaimed space: 5.845kB $ docker images -a REPOSITORY TAG IMAGE ID CREATED SIZE $ DOCKER_BUILDKIT=1 docker build --tag registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --cache-from registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest --build-arg BUILDKIT_INLINE_CACHE=1 -f debian-amd64.docker . [+] Building 57.8s (12/12) FINISHED => [internal] load build definition from debian-amd64.docker => => transferring dockerfile: 5.90kB => [internal] load .dockerignore => => transferring context: 2B => [internal] load metadata for docker.io/library/debian:11-slim => importing cache manifest from registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest => [1/6] FROM docker.io/library/debian:11-slim@sha256:8eaee63a5ea83744e62d5bf88e7d472d7f19b5feda3bfc6a2304cc074f269269 => CACHED [2/6] RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install -y eatmydata && eatmydata apt-get dist-upgrade -y && eatmydata apt-get i => CACHED [3/6] RUN DEBIAN_FRONTEND=noninteractive eatmydata apt install -y --no-install-recommends cscope global linux-headers-amd64 => CACHED [4/6] RUN git clone https://github.com/luigirizzo/netmap.git /usr/src/netmap => CACHED [5/6] RUN cd /usr/src/netmap && git checkout v11.3 => CACHED [6/6] RUN cd /usr/src/netmap/LINUX && ./configure --no-drivers --no-apps --kernel-dir=$(ls -d /usr/src/linux-headers-*-amd64) && make install => => pulling sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa => => pulling sha256:f63092d28e024c119d6a4e257d0c62bb2117be93605f1f2363b1d9cb8fc9dc6c => => pulling sha256:30fcedea9de9f22da43e20a79b40516c6e4a55e70eea401acacca18912e86769 => => pulling sha256:6f498daa6d20055a31e4681ec878e86129c3c1eaf5cf40f7ed87631accef1906 => => pulling sha256:79b1b0d7bce9a16b4a01e2055cd402d93e04efded50e2ffcf6be0d7c67e4895a => => pulling sha256:250b927c5ce2d6d3e9d24bf085f649ce951966c9ec9c9e00ab8292dfb02c8b17 => exporting to image => => exporting layers => => writing image sha256:3663223c0a31c2be678c2fe110ba284b80b8d04d2d3290c14be7a4d7320f8a67 => => naming to registry.gitlab.com/farosas/qemu/qemu/debian-amd64:latest => exporting cache => => preparing build cache for export === > > There are issues pointing to some general unreliability of the cache: > > https://github.com/moby/buildkit/issues/2274 > https://github.com/moby/buildkit/issues/1981 > > What is your docker version? Mine is 20.10.17-ce and Gitlab's is > 23.0.1. The issues above are on 20.10.11.
On Mon, Feb 27, 2023 at 06:12:30PM -0300, Fabiano Rosas wrote: > Alex Bennée <alex.bennee@linaro.org> writes: > > > Fabiano Rosas <farosas@suse.de> writes: > > > >> Our dockerfiles no longer reference layers from other qemu images so > >> we can now use 'docker build' on them. > >> > >> Also reinstate the caching that was disabled due to bad interactions > >> with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use > >> BUILDKIT in GitLab either"). We now believe those issues to be fixed. > >> > >> The COMMON_TAG needed to be fixed for the caching to work. The > >> docker.py script was not using the variable, but constructing the > >> correct URL directly. > >> > >> Signed-off-by: Fabiano Rosas <farosas@suse.de> > >> --- > >> .gitlab-ci.d/container-template.yml | 9 ++++----- > >> 1 file changed, 4 insertions(+), 5 deletions(-) > >> > >> diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml > >> index c434b9c8f3..519b8a9482 100644 > >> --- a/.gitlab-ci.d/container-template.yml > >> +++ b/.gitlab-ci.d/container-template.yml > >> @@ -6,17 +6,16 @@ > >> - docker:dind > >> before_script: > >> - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > >> - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" > >> + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > >> - apk add python3 > >> - docker info > >> - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > >> script: > >> - echo "TAG:$TAG" > >> - echo "COMMON_TAG:$COMMON_TAG" > >> - - ./tests/docker/docker.py --engine docker build > >> - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" > >> - -r $CI_REGISTRY/qemu-project/qemu > >> - - docker tag "qemu/$NAME" "$TAG" > >> + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" > >> + --build-arg BUILDKIT_INLINE_CACHE=1 > >> + -f "tests/docker/dockerfiles/$NAME.docker" "." > > > > I wonder why this doesn't injest a bunch of context. If I run: > > > > docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg BUILDKIT_INLINE_CACHE=1 --build-arg USER=alex --build-arg UID=1000 -t qemu > > /debian-alpha-cross -f "/home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker" . > > > > it attempts to bring my entire build directory in as build context. This > > is why we use the - < docker form in the Makefile. > > > > I only see that without DOCKER_BUILDKIT=1. With the variable set it does > like in the CI jobs. I presume it is being set automatically by gitlab, > but we could add it to the script to be explicit. > > >> - docker push "$TAG" > >> after_script: > >> - docker logout > > > > So what I don't understand is if I do: > > > > docker pull registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross > > docker build --cache-from registry.gitlab.com/stsquad/qemu/qemu/debian-alpha-cross --build-arg > > BUILDKIT_INLINE_CACHE=1 -t qemu/debian-alpha-cross - < > > /home/alex/lsrc/qemu.git/tests/docker/dockerfiles/debian-alpha-cross.docker > > > > I still see pretty much a full rebuild of the image. > > I don't use docker and podman does not support caching. I have > fresh-installed docker today and indeed it seems to not use the cache at > every build. We're missing something. Empirically in gitlab I see it working as desired after applying this patch. My first job building alpine container does all commands: https://gitlab.com/berrange/qemu/-/jobs/3845066410 My second job building alpine container uses the cache: https://gitlab.com/berrange/qemu/-/jobs/3845340414 IME of using this in libvirt the caching almost always did its job as expected in gitlab. IOW I wouldn't worry too much about local behaviour possibly differing. With regards, Daniel
On Mon, Feb 27, 2023 at 12:11:10PM -0300, Fabiano Rosas wrote: > Our dockerfiles no longer reference layers from other qemu images so > we can now use 'docker build' on them. > > Also reinstate the caching that was disabled due to bad interactions > with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use > BUILDKIT in GitLab either"). We now believe those issues to be fixed. > > The COMMON_TAG needed to be fixed for the caching to work. The > docker.py script was not using the variable, but constructing the > correct URL directly. > > Signed-off-by: Fabiano Rosas <farosas@suse.de> > --- > .gitlab-ci.d/container-template.yml | 9 ++++----- > 1 file changed, 4 insertions(+), 5 deletions(-) Tested-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> > > diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml > index c434b9c8f3..519b8a9482 100644 > --- a/.gitlab-ci.d/container-template.yml > +++ b/.gitlab-ci.d/container-template.yml > @@ -6,17 +6,16 @@ > - docker:dind > before_script: > - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" > + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > - apk add python3 > - docker info > - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > script: > - echo "TAG:$TAG" > - echo "COMMON_TAG:$COMMON_TAG" > - - ./tests/docker/docker.py --engine docker build > - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" > - -r $CI_REGISTRY/qemu-project/qemu > - - docker tag "qemu/$NAME" "$TAG" > + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" > + --build-arg BUILDKIT_INLINE_CACHE=1 > + -f "tests/docker/dockerfiles/$NAME.docker" "." > - docker push "$TAG" > after_script: > - docker logout > -- > 2.35.3 > With regards, Daniel
Fabiano Rosas <farosas@suse.de> writes: > Alex Bennée <alex.bennee@linaro.org> writes: > >> Fabiano Rosas <farosas@suse.de> writes: >> <snip> > > There are issues pointing to some general unreliability of the cache: > > https://github.com/moby/buildkit/issues/2274 > https://github.com/moby/buildkit/issues/1981 > > What is your docker version? Mine is 20.10.17-ce and Gitlab's is > 23.0.1. The issues above are on 20.10.11. Ahh Debian stable strikes again: 19:34:42 [root@zen:~] # docker --version Docker version 20.10.5+dfsg1, build 55c4c88 Bookworm is only on 20.10.23+dfsg1-1 and others Maybe it's time to enable the docker repos again? Debian does offer podman but I had issues with mapping user ids between container and volume with it. All the user files appeared own by root.
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index c434b9c8f3..519b8a9482 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -6,17 +6,16 @@ - docker:dind before_script: - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" - apk add python3 - docker info - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" script: - echo "TAG:$TAG" - echo "COMMON_TAG:$COMMON_TAG" - - ./tests/docker/docker.py --engine docker build - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" - -r $CI_REGISTRY/qemu-project/qemu - - docker tag "qemu/$NAME" "$TAG" + - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG" + --build-arg BUILDKIT_INLINE_CACHE=1 + -f "tests/docker/dockerfiles/$NAME.docker" "." - docker push "$TAG" after_script: - docker logout
Our dockerfiles no longer reference layers from other qemu images so we can now use 'docker build' on them. Also reinstate the caching that was disabled due to bad interactions with certain runners. See commit 6ddc3dc7a8 ("tests/docker: don't use BUILDKIT in GitLab either"). We now believe those issues to be fixed. The COMMON_TAG needed to be fixed for the caching to work. The docker.py script was not using the variable, but constructing the correct URL directly. Signed-off-by: Fabiano Rosas <farosas@suse.de> --- .gitlab-ci.d/container-template.yml | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)