[for-7.0] gitlab-ci: Add cirrus-ci based tests for NetBSD and OpenBSD

Cirrus-CI provides KVM in their Linux containers, so we can also run
our VM-based NetBSD and OpenBSD build jobs there.
Since the VM installation might take a while, we only run the "help"
target on the first invocation to avoid timeouts, and then only check
the build during the next run, once the base image has been cached.
For the the build tests, we also only use very a limited set of target
CPUs since compiling in these VMs is not very fast (especially the
build on OpenBSD seems to be incredibly slow).
For the time being, the jobs are also marked as manually only, since
this double-indirect setup (with the cirrus-run script and VMs in
the Cirrus-CI containers) might fail more often than the other jobs.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 If you're interested, you can see some runs here:
 NetBSD: https://gitlab.com/thuth/qemu/-/jobs/1865965079
 OpenBSD: https://gitlab.com/thuth/qemu/-/jobs/1866116634

 .gitlab-ci.d/cirrus.yml           | 35 +++++++++++++++++++++++++++++++
 .gitlab-ci.d/cirrus/kvm-build.yml | 31 +++++++++++++++++++++++++++
 2 files changed, 66 insertions(+)
 create mode 100644 .gitlab-ci.d/cirrus/kvm-build.yml

On Thu, Dec 09, 2021 at 11:31:24AM +0100, Thomas Huth wrote:
> Cirrus-CI provides KVM in their Linux containers, so we can also run
> our VM-based NetBSD and OpenBSD build jobs there.
> Since the VM installation might take a while, we only run the "help"
> target on the first invocation to avoid timeouts, and then only check
> the build during the next run, once the base image has been cached.
> For the the build tests, we also only use very a limited set of target
> CPUs since compiling in these VMs is not very fast (especially the
> build on OpenBSD seems to be incredibly slow).
> For the time being, the jobs are also marked as manually only, since
> this double-indirect setup (with the cirrus-run script and VMs in
> the Cirrus-CI containers) might fail more often than the other jobs.

I think they'll have to be manual forever basically unless
something changes in cirrus.

Historically we've had trouble with the cirrus jobs timing out.
This was ultimately a result of the fact that only 2 cirrus jobs
can run concurrently, and we had duplicate jobs being scheduled
on 'master' and 'staging'. This resulted in 4 jobs being queued
and most of the time, and because each job took > 30 minutes,
two of them would frequently hit the gitlab job 1 hour timeout.

Unless we can ensure that /all/ our cirrus jobs will reliably
completed in about 20 minutes in normal case (30 mins if
cirrus is being slow), then we can't have more than 2 cirrus
jobs as one or more will end up going over the 1 hour cutoff.

The idea of having NetBSD/OpenBSD jobs is good, but I think
it feels like a case where we're going to need to look at
using custom runners if we want them triggere on 'staging'.

Manual jobs could be ok for contributors forks at most.

> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  If you're interested, you can see some runs here:
>  NetBSD: https://gitlab.com/thuth/qemu/-/jobs/1865965079
>  OpenBSD: https://gitlab.com/thuth/qemu/-/jobs/1866116634
> 
>  .gitlab-ci.d/cirrus.yml           | 35 +++++++++++++++++++++++++++++++
>  .gitlab-ci.d/cirrus/kvm-build.yml | 31 +++++++++++++++++++++++++++
>  2 files changed, 66 insertions(+)
>  create mode 100644 .gitlab-ci.d/cirrus/kvm-build.yml
> 
> diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
> index 90bbe9f8ff..f2d4c0ca4f 100644
> --- a/.gitlab-ci.d/cirrus.yml
> +++ b/.gitlab-ci.d/cirrus.yml
> @@ -64,3 +64,38 @@ x64-macos-11-base-build:
>      PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin
>      PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig
>      TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64
> +
> +
> +# The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job
> +.cirrus_kvm_job:
> +  stage: build
> +  image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
> +  needs: []
> +  timeout: 80m
> +  allow_failure: true
> +  script:
> +    - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g"
> +          -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g"
> +          -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g"
> +          -e "s|[@]NAME@|$NAME|g"
> +          -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g"
> +          -e "s|[@]TEST_TARGETS@|$TEST_TARGETS|g"
> +      <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml
> +    - cat .gitlab-ci.d/cirrus/$NAME.yml
> +    - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
> +  rules:
> +    - when: manual
> +
> +x86-netbsd:
> +  extends: .cirrus_kvm_job
> +  variables:
> +    NAME: netbsd
> +    CONFIGURE_ARGS: --target-list=x86_64-softmmu,ppc64-softmmu,aarch64-softmmu
> +    TEST_TARGETS: check
> +
> +x86-openbsd:
> +  extends: .cirrus_kvm_job
> +  variables:
> +    NAME: openbsd
> +    CONFIGURE_ARGS: --target-list=i386-softmmu,riscv64-softmmu,mips64-softmmu
> +    TEST_TARGETS: check
> diff --git a/.gitlab-ci.d/cirrus/kvm-build.yml b/.gitlab-ci.d/cirrus/kvm-build.yml
> new file mode 100644
> index 0000000000..4334fabf39
> --- /dev/null
> +++ b/.gitlab-ci.d/cirrus/kvm-build.yml
> @@ -0,0 +1,31 @@
> +container:
> +  image: fedora:35
> +  cpu: 4
> +  memory: 8Gb
> +  kvm: true
> +
> +env:
> +  CIRRUS_CLONE_DEPTH: 1
> +  CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@"
> +  CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@"
> +  CI_COMMIT_SHA: "@CI_COMMIT_SHA@"
> +
> +@NAME@_task:
> +  @NAME@_vm_cache:
> +    folder: $HOME/.cache/qemu-vm
> +  install_script:
> +    - dnf update -y
> +    - dnf install -y git make openssh-clients qemu-img qemu-system-x86 wget
> +  clone_script:
> +    - git clone --depth 100 "$CI_REPOSITORY_URL" .
> +    - git fetch origin "$CI_COMMIT_REF_NAME"
> +    - git reset --hard "$CI_COMMIT_SHA"
> +  build_script:
> +    - if [ -f $HOME/.cache/qemu-vm/images/@NAME@.img ]; then
> +        make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN)
> +          EXTRA_CONFIGURE_OPTS="@CONFIGURE_ARGS@"
> +          BUILD_TARGET="@TEST_TARGETS@" ;
> +      else
> +        make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN) BUILD_TARGET=help
> +          EXTRA_CONFIGURE_OPTS="--disable-system --disable-user --disable-tools" ;
> +      fi
> -- 
> 2.27.0
> 
> 

Regards,
Daniel

On 10/12/2021 13.10, Daniel P. Berrangé wrote:
> On Thu, Dec 09, 2021 at 11:31:24AM +0100, Thomas Huth wrote:
>> Cirrus-CI provides KVM in their Linux containers, so we can also run
>> our VM-based NetBSD and OpenBSD build jobs there.
>> Since the VM installation might take a while, we only run the "help"
>> target on the first invocation to avoid timeouts, and then only check
>> the build during the next run, once the base image has been cached.
>> For the the build tests, we also only use very a limited set of target
>> CPUs since compiling in these VMs is not very fast (especially the
>> build on OpenBSD seems to be incredibly slow).
>> For the time being, the jobs are also marked as manually only, since
>> this double-indirect setup (with the cirrus-run script and VMs in
>> the Cirrus-CI containers) might fail more often than the other jobs.
> 
> I think they'll have to be manual forever basically unless
> something changes in cirrus.
> 
> Historically we've had trouble with the cirrus jobs timing out.
> This was ultimately a result of the fact that only 2 cirrus jobs
> can run concurrently, and we had duplicate jobs being scheduled
> on 'master' and 'staging'. This resulted in 4 jobs being queued
> and most of the time, and because each job took > 30 minutes,
> two of them would frequently hit the gitlab job 1 hour timeout.
> 
> Unless we can ensure that /all/ our cirrus jobs will reliably
> completed in about 20 minutes in normal case (30 mins if
> cirrus is being slow), then we can't have more than 2 cirrus
> jobs as one or more will end up going over the 1 hour cutoff.
> 
> The idea of having NetBSD/OpenBSD jobs is good, but I think
> it feels like a case where we're going to need to look at
> using custom runners if we want them trigger on 'staging'.
> 
> Manual jobs could be ok for contributors forks at most.

Agreed. I'll rephrase the last paragraph a little bit:

The jobs are marked as "manual" only, since this double-indirect setup (with 
the cirrus-run script and VMs in the Cirrus-CI containers) might fail more 
often than the other jobs, and since we can trigger a limited amount of 
Cirrus-CI jobs at a time anyway (due to the restrictions in the free tier of 
Cirrus). Thus these jobs are rather added as convenience for contributors 
who would like to run the NetBSD/OpenBSD tests without the need of 
downloading and installing the corresponding VM images on their local machines.

  Thomas