Message ID | 20211209103124.121942-1-thuth@redhat.com |
---|---|
State | New |
Headers | show |
Series | [for-7.0] gitlab-ci: Add cirrus-ci based tests for NetBSD and OpenBSD | expand |
On Thu, Dec 09, 2021 at 11:31:24AM +0100, Thomas Huth wrote: > Cirrus-CI provides KVM in their Linux containers, so we can also run > our VM-based NetBSD and OpenBSD build jobs there. > Since the VM installation might take a while, we only run the "help" > target on the first invocation to avoid timeouts, and then only check > the build during the next run, once the base image has been cached. > For the the build tests, we also only use very a limited set of target > CPUs since compiling in these VMs is not very fast (especially the > build on OpenBSD seems to be incredibly slow). > For the time being, the jobs are also marked as manually only, since > this double-indirect setup (with the cirrus-run script and VMs in > the Cirrus-CI containers) might fail more often than the other jobs. I think they'll have to be manual forever basically unless something changes in cirrus. Historically we've had trouble with the cirrus jobs timing out. This was ultimately a result of the fact that only 2 cirrus jobs can run concurrently, and we had duplicate jobs being scheduled on 'master' and 'staging'. This resulted in 4 jobs being queued and most of the time, and because each job took > 30 minutes, two of them would frequently hit the gitlab job 1 hour timeout. Unless we can ensure that /all/ our cirrus jobs will reliably completed in about 20 minutes in normal case (30 mins if cirrus is being slow), then we can't have more than 2 cirrus jobs as one or more will end up going over the 1 hour cutoff. The idea of having NetBSD/OpenBSD jobs is good, but I think it feels like a case where we're going to need to look at using custom runners if we want them triggere on 'staging'. Manual jobs could be ok for contributors forks at most. > Signed-off-by: Thomas Huth <thuth@redhat.com> > --- > If you're interested, you can see some runs here: > NetBSD: https://gitlab.com/thuth/qemu/-/jobs/1865965079 > OpenBSD: https://gitlab.com/thuth/qemu/-/jobs/1866116634 > > .gitlab-ci.d/cirrus.yml | 35 +++++++++++++++++++++++++++++++ > .gitlab-ci.d/cirrus/kvm-build.yml | 31 +++++++++++++++++++++++++++ > 2 files changed, 66 insertions(+) > create mode 100644 .gitlab-ci.d/cirrus/kvm-build.yml > > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml > index 90bbe9f8ff..f2d4c0ca4f 100644 > --- a/.gitlab-ci.d/cirrus.yml > +++ b/.gitlab-ci.d/cirrus.yml > @@ -64,3 +64,38 @@ x64-macos-11-base-build: > PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin > PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig > TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64 > + > + > +# The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job > +.cirrus_kvm_job: > + stage: build > + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master > + needs: [] > + timeout: 80m > + allow_failure: true > + script: > + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" > + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" > + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g" > + -e "s|[@]NAME@|$NAME|g" > + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g" > + -e "s|[@]TEST_TARGETS@|$TEST_TARGETS|g" > + <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml > + - cat .gitlab-ci.d/cirrus/$NAME.yml > + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml > + rules: > + - when: manual > + > +x86-netbsd: > + extends: .cirrus_kvm_job > + variables: > + NAME: netbsd > + CONFIGURE_ARGS: --target-list=x86_64-softmmu,ppc64-softmmu,aarch64-softmmu > + TEST_TARGETS: check > + > +x86-openbsd: > + extends: .cirrus_kvm_job > + variables: > + NAME: openbsd > + CONFIGURE_ARGS: --target-list=i386-softmmu,riscv64-softmmu,mips64-softmmu > + TEST_TARGETS: check > diff --git a/.gitlab-ci.d/cirrus/kvm-build.yml b/.gitlab-ci.d/cirrus/kvm-build.yml > new file mode 100644 > index 0000000000..4334fabf39 > --- /dev/null > +++ b/.gitlab-ci.d/cirrus/kvm-build.yml > @@ -0,0 +1,31 @@ > +container: > + image: fedora:35 > + cpu: 4 > + memory: 8Gb > + kvm: true > + > +env: > + CIRRUS_CLONE_DEPTH: 1 > + CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@" > + CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@" > + CI_COMMIT_SHA: "@CI_COMMIT_SHA@" > + > +@NAME@_task: > + @NAME@_vm_cache: > + folder: $HOME/.cache/qemu-vm > + install_script: > + - dnf update -y > + - dnf install -y git make openssh-clients qemu-img qemu-system-x86 wget > + clone_script: > + - git clone --depth 100 "$CI_REPOSITORY_URL" . > + - git fetch origin "$CI_COMMIT_REF_NAME" > + - git reset --hard "$CI_COMMIT_SHA" > + build_script: > + - if [ -f $HOME/.cache/qemu-vm/images/@NAME@.img ]; then > + make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN) > + EXTRA_CONFIGURE_OPTS="@CONFIGURE_ARGS@" > + BUILD_TARGET="@TEST_TARGETS@" ; > + else > + make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN) BUILD_TARGET=help > + EXTRA_CONFIGURE_OPTS="--disable-system --disable-user --disable-tools" ; > + fi > -- > 2.27.0 > > Regards, Daniel
On 10/12/2021 13.10, Daniel P. Berrangé wrote: > On Thu, Dec 09, 2021 at 11:31:24AM +0100, Thomas Huth wrote: >> Cirrus-CI provides KVM in their Linux containers, so we can also run >> our VM-based NetBSD and OpenBSD build jobs there. >> Since the VM installation might take a while, we only run the "help" >> target on the first invocation to avoid timeouts, and then only check >> the build during the next run, once the base image has been cached. >> For the the build tests, we also only use very a limited set of target >> CPUs since compiling in these VMs is not very fast (especially the >> build on OpenBSD seems to be incredibly slow). >> For the time being, the jobs are also marked as manually only, since >> this double-indirect setup (with the cirrus-run script and VMs in >> the Cirrus-CI containers) might fail more often than the other jobs. > > I think they'll have to be manual forever basically unless > something changes in cirrus. > > Historically we've had trouble with the cirrus jobs timing out. > This was ultimately a result of the fact that only 2 cirrus jobs > can run concurrently, and we had duplicate jobs being scheduled > on 'master' and 'staging'. This resulted in 4 jobs being queued > and most of the time, and because each job took > 30 minutes, > two of them would frequently hit the gitlab job 1 hour timeout. > > Unless we can ensure that /all/ our cirrus jobs will reliably > completed in about 20 minutes in normal case (30 mins if > cirrus is being slow), then we can't have more than 2 cirrus > jobs as one or more will end up going over the 1 hour cutoff. > > The idea of having NetBSD/OpenBSD jobs is good, but I think > it feels like a case where we're going to need to look at > using custom runners if we want them trigger on 'staging'. > > Manual jobs could be ok for contributors forks at most. Agreed. I'll rephrase the last paragraph a little bit: The jobs are marked as "manual" only, since this double-indirect setup (with the cirrus-run script and VMs in the Cirrus-CI containers) might fail more often than the other jobs, and since we can trigger a limited amount of Cirrus-CI jobs at a time anyway (due to the restrictions in the free tier of Cirrus). Thus these jobs are rather added as convenience for contributors who would like to run the NetBSD/OpenBSD tests without the need of downloading and installing the corresponding VM images on their local machines. Thomas
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml index 90bbe9f8ff..f2d4c0ca4f 100644 --- a/.gitlab-ci.d/cirrus.yml +++ b/.gitlab-ci.d/cirrus.yml @@ -64,3 +64,38 @@ x64-macos-11-base-build: PATH_EXTRA: /usr/local/opt/ccache/libexec:/usr/local/opt/gettext/bin PKG_CONFIG_PATH: /usr/local/opt/curl/lib/pkgconfig:/usr/local/opt/ncurses/lib/pkgconfig:/usr/local/opt/readline/lib/pkgconfig TEST_TARGETS: check-unit check-block check-qapi-schema check-softfloat check-qtest-x86_64 + + +# The following jobs run VM-based tests via KVM on a Linux-based Cirrus-CI job +.cirrus_kvm_job: + stage: build + image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master + needs: [] + timeout: 80m + allow_failure: true + script: + - sed -e "s|[@]CI_REPOSITORY_URL@|$CI_REPOSITORY_URL|g" + -e "s|[@]CI_COMMIT_REF_NAME@|$CI_COMMIT_REF_NAME|g" + -e "s|[@]CI_COMMIT_SHA@|$CI_COMMIT_SHA|g" + -e "s|[@]NAME@|$NAME|g" + -e "s|[@]CONFIGURE_ARGS@|$CONFIGURE_ARGS|g" + -e "s|[@]TEST_TARGETS@|$TEST_TARGETS|g" + <.gitlab-ci.d/cirrus/kvm-build.yml >.gitlab-ci.d/cirrus/$NAME.yml + - cat .gitlab-ci.d/cirrus/$NAME.yml + - cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml + rules: + - when: manual + +x86-netbsd: + extends: .cirrus_kvm_job + variables: + NAME: netbsd + CONFIGURE_ARGS: --target-list=x86_64-softmmu,ppc64-softmmu,aarch64-softmmu + TEST_TARGETS: check + +x86-openbsd: + extends: .cirrus_kvm_job + variables: + NAME: openbsd + CONFIGURE_ARGS: --target-list=i386-softmmu,riscv64-softmmu,mips64-softmmu + TEST_TARGETS: check diff --git a/.gitlab-ci.d/cirrus/kvm-build.yml b/.gitlab-ci.d/cirrus/kvm-build.yml new file mode 100644 index 0000000000..4334fabf39 --- /dev/null +++ b/.gitlab-ci.d/cirrus/kvm-build.yml @@ -0,0 +1,31 @@ +container: + image: fedora:35 + cpu: 4 + memory: 8Gb + kvm: true + +env: + CIRRUS_CLONE_DEPTH: 1 + CI_REPOSITORY_URL: "@CI_REPOSITORY_URL@" + CI_COMMIT_REF_NAME: "@CI_COMMIT_REF_NAME@" + CI_COMMIT_SHA: "@CI_COMMIT_SHA@" + +@NAME@_task: + @NAME@_vm_cache: + folder: $HOME/.cache/qemu-vm + install_script: + - dnf update -y + - dnf install -y git make openssh-clients qemu-img qemu-system-x86 wget + clone_script: + - git clone --depth 100 "$CI_REPOSITORY_URL" . + - git fetch origin "$CI_COMMIT_REF_NAME" + - git reset --hard "$CI_COMMIT_SHA" + build_script: + - if [ -f $HOME/.cache/qemu-vm/images/@NAME@.img ]; then + make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN) + EXTRA_CONFIGURE_OPTS="@CONFIGURE_ARGS@" + BUILD_TARGET="@TEST_TARGETS@" ; + else + make vm-build-@NAME@ J=$(getconf _NPROCESSORS_ONLN) BUILD_TARGET=help + EXTRA_CONFIGURE_OPTS="--disable-system --disable-user --disable-tools" ; + fi
Cirrus-CI provides KVM in their Linux containers, so we can also run our VM-based NetBSD and OpenBSD build jobs there. Since the VM installation might take a while, we only run the "help" target on the first invocation to avoid timeouts, and then only check the build during the next run, once the base image has been cached. For the the build tests, we also only use very a limited set of target CPUs since compiling in these VMs is not very fast (especially the build on OpenBSD seems to be incredibly slow). For the time being, the jobs are also marked as manually only, since this double-indirect setup (with the cirrus-run script and VMs in the Cirrus-CI containers) might fail more often than the other jobs. Signed-off-by: Thomas Huth <thuth@redhat.com> --- If you're interested, you can see some runs here: NetBSD: https://gitlab.com/thuth/qemu/-/jobs/1865965079 OpenBSD: https://gitlab.com/thuth/qemu/-/jobs/1866116634 .gitlab-ci.d/cirrus.yml | 35 +++++++++++++++++++++++++++++++ .gitlab-ci.d/cirrus/kvm-build.yml | 31 +++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) create mode 100644 .gitlab-ci.d/cirrus/kvm-build.yml