mbox series

[0/7] testing/next: docker.py removal and kaniko updates

Message ID 20230224180857.1050220-1-alex.bennee@linaro.org
Headers show
Series testing/next: docker.py removal and kaniko updates | expand

Message

Alex Bennée Feb. 24, 2023, 6:08 p.m. UTC
This series attempts to remove our dependence on the docker.py script
and build things directly with the appropriate tool. I've been
noodling around with how we build images on gitlab to see if they can
cache better because the normal case should be we don't need to
rebuild everything if the upstream distro hasn't updated its package
list.

Anyway what do people think?

Alex Bennée (7):
  configure: expose the direct container command
  tests/dockerfiles: unify debian-toolchain references
  tests/lcitool: append user setting stanza to dockerfiles
  tests/docker: add USER stanzas to non-lci images
  tests/docker: use direct RUNC call to build containers
  tests/docker: use direct RUNC call to run test jobs
  tests/gitlab: use kaniko to build images

 configure                                     |  3 +++
 .gitlab-ci.d/cirrus/freebsd-12.vars           |  5 ++++
 .gitlab-ci.d/cirrus/freebsd-13.vars           |  5 ++++
 .gitlab-ci.d/cirrus/macos-12.vars             |  5 ++++
 .gitlab-ci.d/container-template.yml           | 23 +++++++---------
 tests/docker/Makefile.include                 | 27 +++++++++++--------
 tests/docker/dockerfiles/alpine.docker        |  5 ++++
 tests/docker/dockerfiles/centos8.docker       |  5 ++++
 .../dockerfiles/debian-all-test-cross.docker  |  5 ++++
 .../dockerfiles/debian-alpha-cross.docker     |  5 ++++
 .../dockerfiles/debian-amd64-cross.docker     |  5 ++++
 tests/docker/dockerfiles/debian-amd64.docker  |  5 ++++
 .../dockerfiles/debian-arm64-cross.docker     |  5 ++++
 .../dockerfiles/debian-armel-cross.docker     |  5 ++++
 .../dockerfiles/debian-armhf-cross.docker     |  5 ++++
 .../dockerfiles/debian-hexagon-cross.docker   |  5 ++++
 .../dockerfiles/debian-hppa-cross.docker      |  5 ++++
 .../dockerfiles/debian-loongarch-cross.docker |  5 ++++
 .../dockerfiles/debian-m68k-cross.docker      |  5 ++++
 .../dockerfiles/debian-mips-cross.docker      |  5 ++++
 .../dockerfiles/debian-mips64-cross.docker    |  5 ++++
 .../dockerfiles/debian-mips64el-cross.docker  |  5 ++++
 .../dockerfiles/debian-mipsel-cross.docker    |  5 ++++
 tests/docker/dockerfiles/debian-native.docker |  5 ++++
 .../debian-powerpc-test-cross.docker          |  6 ++++-
 .../dockerfiles/debian-ppc64el-cross.docker   |  5 ++++
 .../dockerfiles/debian-riscv64-cross.docker   |  5 ++++
 .../debian-riscv64-test-cross.docker          |  5 ++++
 .../dockerfiles/debian-s390x-cross.docker     |  5 ++++
 .../dockerfiles/debian-sh4-cross.docker       |  5 ++++
 .../dockerfiles/debian-sparc64-cross.docker   |  5 ++++
 .../dockerfiles/debian-toolchain.docker       |  9 +++++--
 .../dockerfiles/debian-tricore-cross.docker   |  5 ++++
 .../dockerfiles/debian-xtensa-cross.docker    |  5 ++++
 .../dockerfiles/fedora-cris-cross.docker      |  5 ++++
 .../dockerfiles/fedora-i386-cross.docker      |  5 ++++
 .../dockerfiles/fedora-win32-cross.docker     |  5 ++++
 .../dockerfiles/fedora-win64-cross.docker     |  5 ++++
 tests/docker/dockerfiles/fedora.docker        |  5 ++++
 tests/docker/dockerfiles/opensuse-leap.docker |  5 ++++
 tests/docker/dockerfiles/python.docker        |  5 ++++
 tests/docker/dockerfiles/ubuntu2004.docker    |  5 ++++
 tests/docker/dockerfiles/ubuntu2204.docker    |  5 ++++
 tests/lcitool/refresh                         | 11 +++++++-
 44 files changed, 240 insertions(+), 29 deletions(-)

Comments

Philippe Mathieu-Daudé Feb. 28, 2023, 11:58 a.m. UTC | #1
On 24/2/23 19:08, Alex Bennée wrote:
> This series attempts to remove our dependence on the docker.py script
> and build things directly with the appropriate tool. I've been
> noodling around with how we build images on gitlab to see if they can
> cache better because the normal case should be we don't need to
> rebuild everything if the upstream distro hasn't updated its package
> list.
> 
> Anyway what do people think?

Removing dind limitation is interesting.

Unrelated, can we tag registry.gitlab.com/qemu-project's
docker images along with releases?
Daniel P. Berrangé Feb. 28, 2023, 12:58 p.m. UTC | #2
On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote:
> On 24/2/23 19:08, Alex Bennée wrote:
> > This series attempts to remove our dependence on the docker.py script
> > and build things directly with the appropriate tool. I've been
> > noodling around with how we build images on gitlab to see if they can
> > cache better because the normal case should be we don't need to
> > rebuild everything if the upstream distro hasn't updated its package
> > list.
> > 
> > Anyway what do people think?
> 
> Removing dind limitation is interesting.
> 
> Unrelated, can we tag registry.gitlab.com/qemu-project's
> docker images along with releases?

Can you elaborate on this ?

We're only using the images for CI purposes and they must always reflect
the current state of git master. We're using a fixed docker tag 'latest',
as that avoids the container registry growing arbitrarily large.

Our CI rules should prevent the pipelines running on stable branches,
so we shouldn't need container tags for stable.

With regards,
Daniel
Philippe Mathieu-Daudé Feb. 28, 2023, 1:29 p.m. UTC | #3
On 28/2/23 13:58, Daniel P. Berrangé wrote:
> On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote:
>> On 24/2/23 19:08, Alex Bennée wrote:
>>> This series attempts to remove our dependence on the docker.py script
>>> and build things directly with the appropriate tool. I've been
>>> noodling around with how we build images on gitlab to see if they can
>>> cache better because the normal case should be we don't need to
>>> rebuild everything if the upstream distro hasn't updated its package
>>> list.
>>>
>>> Anyway what do people think?
>>
>> Removing dind limitation is interesting.
>>
>> Unrelated, can we tag registry.gitlab.com/qemu-project's
>> docker images along with releases?
> 
> Can you elaborate on this ?
> 
> We're only using the images for CI purposes and they must always reflect
> the current state of git master. We're using a fixed docker tag 'latest',
> as that avoids the container registry growing arbitrarily large.
> 
> Our CI rules should prevent the pipelines running on stable branches,
> so we shouldn't need container tags for stable.

I'm not suggesting keeping jobs to build, but doing a snapshot of the
released containers.

I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch
it again. This is useful when bisecting pre-v8, but also to build pre-v8
and do performance comparison. One shouldn't have to upgrade such
container (in particular when package mirror disappear), since it
already contains all we need.

Personally I'd like to keep generic and problematic ones:
- qemu/fedora
- qemu/fedora-win64-cross
- qemu/debian-xtensa-cross
Daniel P. Berrangé Feb. 28, 2023, 1:34 p.m. UTC | #4
On Tue, Feb 28, 2023 at 02:29:12PM +0100, Philippe Mathieu-Daudé wrote:
> On 28/2/23 13:58, Daniel P. Berrangé wrote:
> > On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote:
> > > On 24/2/23 19:08, Alex Bennée wrote:
> > > > This series attempts to remove our dependence on the docker.py script
> > > > and build things directly with the appropriate tool. I've been
> > > > noodling around with how we build images on gitlab to see if they can
> > > > cache better because the normal case should be we don't need to
> > > > rebuild everything if the upstream distro hasn't updated its package
> > > > list.
> > > > 
> > > > Anyway what do people think?
> > > 
> > > Removing dind limitation is interesting.
> > > 
> > > Unrelated, can we tag registry.gitlab.com/qemu-project's
> > > docker images along with releases?
> > 
> > Can you elaborate on this ?
> > 
> > We're only using the images for CI purposes and they must always reflect
> > the current state of git master. We're using a fixed docker tag 'latest',
> > as that avoids the container registry growing arbitrarily large.
> > 
> > Our CI rules should prevent the pipelines running on stable branches,
> > so we shouldn't need container tags for stable.
> 
> I'm not suggesting keeping jobs to build, but doing a snapshot of the
> released containers.
> 
> I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch
> it again. This is useful when bisecting pre-v8, but also to build pre-v8
> and do performance comparison. One shouldn't have to upgrade such
> container (in particular when package mirror disappear), since it
> already contains all we need.

The main risk with this is the impact on our storage quota. With the
OSS Program membership IIUC we get Ultimate level features which
is 250 GB of storage, across git repos, pipeline cache/artifacts/logs,
container registry.

Currently they have no way to enforce that since their accounting of
usage is not accurate enough. They're working on fixing that so at
somepoint we'll be subject to the 250 GB limit.

What I don't know is how much storage we're currently using across
the /qemu-project namespace, and what extra is implied by taking
a snapshot of our container registry 3 times a year. I'm expecting
it to probably be in the high 10's of GB though for the container
registry.

With regards,
Daniel
Alex Bennée Feb. 28, 2023, 1:57 p.m. UTC | #5
Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Feb 28, 2023 at 02:29:12PM +0100, Philippe Mathieu-Daudé wrote:
>> On 28/2/23 13:58, Daniel P. Berrangé wrote:
>> > On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote:
>> > > On 24/2/23 19:08, Alex Bennée wrote:
>> > > > This series attempts to remove our dependence on the docker.py script
>> > > > and build things directly with the appropriate tool. I've been
>> > > > noodling around with how we build images on gitlab to see if they can
>> > > > cache better because the normal case should be we don't need to
>> > > > rebuild everything if the upstream distro hasn't updated its package
>> > > > list.
>> > > > 
>> > > > Anyway what do people think?
>> > > 
>> > > Removing dind limitation is interesting.
>> > > 
>> > > Unrelated, can we tag registry.gitlab.com/qemu-project's
>> > > docker images along with releases?
>> > 
>> > Can you elaborate on this ?
>> > 
>> > We're only using the images for CI purposes and they must always reflect
>> > the current state of git master. We're using a fixed docker tag 'latest',
>> > as that avoids the container registry growing arbitrarily large.
>> > 
>> > Our CI rules should prevent the pipelines running on stable branches,
>> > so we shouldn't need container tags for stable.
>> 
>> I'm not suggesting keeping jobs to build, but doing a snapshot of the
>> released containers.
>> 
>> I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch
>> it again. This is useful when bisecting pre-v8, but also to build pre-v8
>> and do performance comparison. One shouldn't have to upgrade such
>> container (in particular when package mirror disappear), since it
>> already contains all we need.
>
> The main risk with this is the impact on our storage quota. With the
> OSS Program membership IIUC we get Ultimate level features which
> is 250 GB of storage, across git repos, pipeline cache/artifacts/logs,
> container registry.
>
> Currently they have no way to enforce that since their accounting of
> usage is not accurate enough. They're working on fixing that so at
> somepoint we'll be subject to the 250 GB limit.
>
> What I don't know is how much storage we're currently using across
> the /qemu-project namespace, and what extra is implied by taking
> a snapshot of our container registry 3 times a year. I'm expecting
> it to probably be in the high 10's of GB though for the container
> registry.

Currently we are using:

86.6 Gb of artefacts
28.5 Gb of containers
220 Mb of file uploads
24.8Mb of git repo

We could probably cut down a lot of usage of artefacts by either not
building full fat ones with debug symbols and passing between layers or
tweaking the build system to prevent re-building of object files if the
final binary is present in the file system.
Philippe Mathieu-Daudé Feb. 28, 2023, 2:33 p.m. UTC | #6
On 28/2/23 14:57, Alex Bennée wrote:
> 
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
>> On Tue, Feb 28, 2023 at 02:29:12PM +0100, Philippe Mathieu-Daudé wrote:
>>> On 28/2/23 13:58, Daniel P. Berrangé wrote:
>>>> On Tue, Feb 28, 2023 at 12:58:54PM +0100, Philippe Mathieu-Daudé wrote:
>>>>> On 24/2/23 19:08, Alex Bennée wrote:
>>>>>> This series attempts to remove our dependence on the docker.py script
>>>>>> and build things directly with the appropriate tool. I've been
>>>>>> noodling around with how we build images on gitlab to see if they can
>>>>>> cache better because the normal case should be we don't need to
>>>>>> rebuild everything if the upstream distro hasn't updated its package
>>>>>> list.
>>>>>>
>>>>>> Anyway what do people think?
>>>>>
>>>>> Removing dind limitation is interesting.
>>>>>
>>>>> Unrelated, can we tag registry.gitlab.com/qemu-project's
>>>>> docker images along with releases?
>>>>
>>>> Can you elaborate on this ?
>>>>
>>>> We're only using the images for CI purposes and they must always reflect
>>>> the current state of git master. We're using a fixed docker tag 'latest',
>>>> as that avoids the container registry growing arbitrarily large.
>>>>
>>>> Our CI rules should prevent the pipelines running on stable branches,
>>>> so we shouldn't need container tags for stable.
>>>
>>> I'm not suggesting keeping jobs to build, but doing a snapshot of the
>>> released containers.
>>>
>>> I.e. when we release 8.0, we should tag qemu/fedora:v8.0 and never touch
>>> it again. This is useful when bisecting pre-v8, but also to build pre-v8
>>> and do performance comparison. One shouldn't have to upgrade such
>>> container (in particular when package mirror disappear), since it
>>> already contains all we need.
>>
>> The main risk with this is the impact on our storage quota. With the
>> OSS Program membership IIUC we get Ultimate level features which
>> is 250 GB of storage, across git repos, pipeline cache/artifacts/logs,
>> container registry.
>>
>> Currently they have no way to enforce that since their accounting of
>> usage is not accurate enough. They're working on fixing that so at
>> somepoint we'll be subject to the 250 GB limit.
>>
>> What I don't know is how much storage we're currently using across
>> the /qemu-project namespace, and what extra is implied by taking
>> a snapshot of our container registry 3 times a year. I'm expecting
>> it to probably be in the high 10's of GB though for the container
>> registry.

Maybe we can keep (a subset of) the stable release container images
and store them on a registry outside of gitlab. That would be nice,
but if I'm the only one interested and nobody felt this was useful
before, no need to spend more energy on this topic.

> Currently we are using:
> 
> 86.6 Gb of artefacts

^ transient

> 28.5 Gb of containers

^ constant

> 220 Mb of file uploads
> 24.8Mb of git repo
> 
> We could probably cut down a lot of usage of artefacts by either not
> building full fat ones with debug symbols and passing between layers or
> tweaking the build system to prevent re-building of object files if the
> final binary is present in the file system.
>