| Message ID | 20230224180857.1050220-8-alex.bennee@linaro.org |
|---|---|
| State | New |
| Headers | show |
| Series | testing/next: docker.py removal and kaniko updates | expand |
On Fri, Feb 24, 2023 at 06:08:57PM +0000, Alex Bennée wrote: > Apparently the docker-in-docker approach has some flaws including > needing privileged mode to run and being quite slow. An alternative > approach is to use Google's kaniko tool. It also works across > different gitlab executors. Interesting, I've not come across this tool before. > Following the gitlab example code we drop all the direct docker calls > and usage of the script and make a direct call to kaniko and hope the > images are cacheable by others. > > Signed-off-by: Alex Bennée <alex.bennee@linaro.org> > --- > .gitlab-ci.d/container-template.yml | 23 +++++++++-------------- > 1 file changed, 9 insertions(+), 14 deletions(-) > > diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml > index c434b9c8f3..c2d7950df8 100644 > --- a/.gitlab-ci.d/container-template.yml > +++ b/.gitlab-ci.d/container-template.yml > @@ -1,22 +1,17 @@ > .container_job_template: > extends: .base_job_template > - image: docker:stable > + image: > + name: gcr.io/kaniko-project/executor:v1.9.0-debug > + entrypoint: [""] > stage: containers > - services: > - - docker:dind > before_script: > - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" > - - apk add python3 > - - docker info > - - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > script: > - echo "TAG:$TAG" > - echo "COMMON_TAG:$COMMON_TAG" > - - ./tests/docker/docker.py --engine docker build > - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" > - -r $CI_REGISTRY/qemu-project/qemu > - - docker tag "qemu/$NAME" "$TAG" > - - docker push "$TAG" > - after_script: > - - docker logout > + - /kaniko/executor > + --reproducible > + --context "${CI_PROJECT_DIR}" > + --dockerfile "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker" > + --destination "${TAG}" AFAICT from reading the docs, this does not allow for caching of layers from the previous built image, so will always do a from scratch build. This feels like it would kill any performance benefits over docker. https://github.com/GoogleContainerTools/kaniko#caching "Users can opt into caching by setting the --cache=true flag. A remote repository for storing cached layers can be provided via the --cache-repo flag. If this flag isn't provided, a cached repo will be inferred from the --destination provided." So it would sound like we need --cache=true adding to this command, and in theory it should prime its cache from ${TAG}, but i'm not entirely confident in that without testing. Also does not seem like its psosible to have the dual cache COMMON_TAG + TAG, which is particularly important for forks, since COMMON_TAG is more likely to be up2date. We could live with the latter restriction if we *always* used COMMON_TAG as the cache, because 99% of the time forks are going to just be using dockerfiles that match upstream. Only those few contributors who have a branch that's changing the dockerfiles would benefit from pointing the cache at their fork. So we could achieve that I believe using - /kaniko/executor --reproducible --context "${CI_PROJECT_DIR}" --cache=true --cache-repo "${COMMON_TAG}" --dockerfile "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker" --destination "${TAG}" With regards, Daniel
diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index c434b9c8f3..c2d7950df8 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -1,22 +1,17 @@ .container_job_template: extends: .base_job_template - image: docker:stable + image: + name: gcr.io/kaniko-project/executor:v1.9.0-debug + entrypoint: [""] stage: containers - services: - - docker:dind before_script: - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" - - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/$NAME:latest" - - apk add python3 - - docker info - - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" script: - echo "TAG:$TAG" - echo "COMMON_TAG:$COMMON_TAG" - - ./tests/docker/docker.py --engine docker build - -t "qemu/$NAME" -f "tests/docker/dockerfiles/$NAME.docker" - -r $CI_REGISTRY/qemu-project/qemu - - docker tag "qemu/$NAME" "$TAG" - - docker push "$TAG" - after_script: - - docker logout + - /kaniko/executor + --reproducible + --context "${CI_PROJECT_DIR}" + --dockerfile "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker" + --destination "${TAG}"
Apparently the docker-in-docker approach has some flaws including needing privileged mode to run and being quite slow. An alternative approach is to use Google's kaniko tool. It also works across different gitlab executors. Following the gitlab example code we drop all the direct docker calls and usage of the script and make a direct call to kaniko and hope the images are cacheable by others. Signed-off-by: Alex Bennée <alex.bennee@linaro.org> --- .gitlab-ci.d/container-template.yml | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-)