| Message ID | 20181020224254.5895-1-arnout@mind.be |
|---|---|
| State | Superseded |
| Headers | show |
| Series | .gitlab-ci.yml: do runtime tests only on explicit trigger | expand |
Hello, On Sat, Oct 20, 2018 at 07:42 PM, Arnout Vandecappelle (Essensium/Mind) wrote: [snip] > +++ b/.gitlab-ci.yml > @@ -62,6 +62,11 @@ check-package: > - output/build/packages-file-list.txt > > .runtime_test: &runtime_test > + # Running the runtime tests for every push is too much, so limit to > + # explicit triggers through the API. > + only: > + - triggers > + - tags > # Keep build directories so the rootfs can be an artifact of the job. The > # runner will clean up those files for us. > # Multiply every emulator timeout by 10 to avoid sporadic failures in > @@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig > zynq_zed_defconfig: *defconfig > zynq_zybo_defconfig: *defconfig > zynqmp_zcu106_defconfig: *defconfig > -tests.boot.test_atf.TestATFAllwinner: *runtime_test > -tests.boot.test_atf.TestATFMarvell: *runtime_test > -tests.boot.test_atf.TestATFVexpress: *runtime_test [snip] You forgot to re-run 'make .gitlab-ci.yml'. We don't want to remove the test list. > +++ b/.gitlab-ci.yml.in > @@ -62,6 +62,11 @@ check-package: > - output/build/packages-file-list.txt > > .runtime_test: &runtime_test > + # Running the runtime tests for every push is too much, so limit to > + # explicit triggers through the API. > + only: > + - triggers > + - tags We could also allow a regexp on the branch name to trigger it: - /^.*-runtime_test$/ Using the same commit, I pushed 2 branches: branch name: gitlabci-triggers, #jobs = 4 https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520 branch name: gitlabci-triggers-runtime_test, #jobs = 72 https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541 > # Keep build directories so the rootfs can be an artifact of the job. The > # runner will clean up those files for us. > # Multiply every emulator timeout by 10 to avoid sporadic failures in > -- Regards, Ricardo
On 21/10/2018 03:54, Ricardo Martincoski wrote: > Hello, > > On Sat, Oct 20, 2018 at 07:42 PM, Arnout Vandecappelle (Essensium/Mind) wrote: > > [snip] >> +++ b/.gitlab-ci.yml >> @@ -62,6 +62,11 @@ check-package: >> - output/build/packages-file-list.txt >> >> .runtime_test: &runtime_test >> + # Running the runtime tests for every push is too much, so limit to >> + # explicit triggers through the API. >> + only: >> + - triggers >> + - tags >> # Keep build directories so the rootfs can be an artifact of the job. The >> # runner will clean up those files for us. >> # Multiply every emulator timeout by 10 to avoid sporadic failures in >> @@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig >> zynq_zed_defconfig: *defconfig >> zynq_zybo_defconfig: *defconfig >> zynqmp_zcu106_defconfig: *defconfig >> -tests.boot.test_atf.TestATFAllwinner: *runtime_test >> -tests.boot.test_atf.TestATFMarvell: *runtime_test >> -tests.boot.test_atf.TestATFVexpress: *runtime_test > [snip] > > You forgot to re-run 'make .gitlab-ci.yml'. > We don't want to remove the test list. Oops. Actually I didn't forget, but it failed (silently) to generate correctly because nsoe2 wasn't installed. So I've submitted a patch that errors out if the listing fails. > >> +++ b/.gitlab-ci.yml.in >> @@ -62,6 +62,11 @@ check-package: >> - output/build/packages-file-list.txt >> >> .runtime_test: &runtime_test >> + # Running the runtime tests for every push is too much, so limit to >> + # explicit triggers through the API. >> + only: >> + - triggers >> + - tags > > We could also allow a regexp on the branch name to trigger it: > - /^.*-runtime_test$/ > > Using the same commit, I pushed 2 branches: > branch name: gitlabci-triggers, #jobs = 4 > https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520 > branch name: gitlabci-triggers-runtime_test, #jobs = 72 > https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541 Looks nice, but that's really material for a separate patch (and probably should be done for defconfigs too then). Would it be possible to also define branches for a specific test or defconfig? Could you take care of this? Regards, Arnout
Hello, On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote: [snip] >>> .runtime_test: &runtime_test >>> + # Running the runtime tests for every push is too much, so limit to >>> + # explicit triggers through the API. >>> + only: >>> + - triggers >>> + - tags >> >> We could also allow a regexp on the branch name to trigger it: >> - /^.*-runtime_test$/ >> >> Using the same commit, I pushed 2 branches: >> branch name: gitlabci-triggers, #jobs = 4 >> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520 >> branch name: gitlabci-triggers-runtime_test, #jobs = 72 >> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541 > > Looks nice, but that's really material for a separate patch (and probably Indeed. > should be done for defconfigs too then). OK. > > Would it be possible to also define branches for a specific test or defconfig? Yes. Using the same commit, I pushed 3 branches and a tag: git push gitlab HEAD:test2 #jobs = 4 https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315 git push gitlab HEAD:test2-TestSyslogNg #jobs = 5 https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342 git push gitlab HEAD:test2-runtime_test #jobs = 72 https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317 git push gitlab test2-tag #jobs = 260 https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321 We will need some extra 'sed' trickery to create a regexp for each test case: -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test +tests.package.test_syslog_ng.TestSyslogNg: + <<: *runtime_test + only: + - triggers + - tags + - /^.*-runtime_test$/ + - /^.*-TestSyslogNg$/ Unfortunately the inheritance using the anchor runtime_test does not allow to merge two 'only' keywords, the 'only' declared on the test case key overrides entirely the 'only' declared on the hidden key '.runtime_test' (or the other way around if << is placed at the end), so we would need to repeat those lines for every test case listed. But maybe we don't care that .gitlab-ci.yml grows because those lines would be automatically generated by 'make .gitlab-ci.yml'. > > Could you take care of this? Sure. Regards, Ricardo
On 10/22/18 3:35 AM, Ricardo Martincoski wrote: > Hello, > > On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote: > > [snip] >>>> .runtime_test: &runtime_test >>>> + # Running the runtime tests for every push is too much, so limit to >>>> + # explicit triggers through the API. >>>> + only: >>>> + - triggers >>>> + - tags >>> >>> We could also allow a regexp on the branch name to trigger it: >>> - /^.*-runtime_test$/ >>> >>> Using the same commit, I pushed 2 branches: >>> branch name: gitlabci-triggers, #jobs = 4 >>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723520 >>> branch name: gitlabci-triggers-runtime_test, #jobs = 72 >>> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33723541 >> >> Looks nice, but that's really material for a separate patch (and probably > > Indeed. > >> should be done for defconfigs too then). > > OK. > >> >> Would it be possible to also define branches for a specific test or defconfig? > > Yes. > > Using the same commit, I pushed 3 branches and a tag: > git push gitlab HEAD:test2 #jobs = 4 > https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315 > git push gitlab HEAD:test2-TestSyslogNg #jobs = 5 > https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342 > git push gitlab HEAD:test2-runtime_test #jobs = 72 > https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317 > git push gitlab test2-tag #jobs = 260 > https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321 Nice! I suppose test2-tag is actually a tag, not a branch, so it matches the 'tags' trigger? > > We will need some extra 'sed' trickery to create a regexp for each test case: > -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test > +tests.package.test_syslog_ng.TestSyslogNg: > + <<: *runtime_test > + only: > + - triggers > + - tags > + - /^.*-runtime_test$/ > + - /^.*-TestSyslogNg$/ Maybe you can use variables in there? So, in the only specification of .runtime_test: only: - triggers - tags - /^.*-runtime_test$/ - /^.*-${CI_JOB_NAME}$/ Hm, unlikely, otherwise the other $ would have to be quoted. Apparently nowadays there is also support for more complex only specifications using variables. Perhaps you could take a look at that? I'm afraid though that it's always an AND of the conditions you specify, so no way to specify that it's for tags OR for things matching the CI_JOB_NAME. Regards, Arnout > > Unfortunately the inheritance using the anchor runtime_test does not allow to > merge two 'only' keywords, the 'only' declared on the test case key overrides > entirely the 'only' declared on the hidden key '.runtime_test' (or the other way > around if << is placed at the end), so we would need to repeat those lines for > every test case listed. > But maybe we don't care that .gitlab-ci.yml grows because those lines would be > automatically generated by 'make .gitlab-ci.yml'. > >> >> Could you take care of this? > > Sure. > > Regards, > Ricardo >
Hello, On Mon, Oct 22, 2018 at 08:08 PM, Arnout Vandecappelle wrote: > On 10/22/18 3:35 AM, Ricardo Martincoski wrote: >> >> On Sun, Oct 21, 2018 at 08:23 AM, Arnout Vandecappelle wrote: [snip] >>> Would it be possible to also define branches for a specific test or defconfig? >> >> Yes. >> >> Using the same commit, I pushed 3 branches and a tag: >> git push gitlab HEAD:test2 #jobs = 4 >> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765315 >> git push gitlab HEAD:test2-TestSyslogNg #jobs = 5 >> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765342 >> git push gitlab HEAD:test2-runtime_test #jobs = 72 >> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765317 >> git push gitlab test2-tag #jobs = 260 >> https://gitlab.com/RicardoMartincoski/buildroot/pipelines/33765321 > > Nice! I suppose test2-tag is actually a tag, not a branch, so it matches the > 'tags' trigger? Yes. It is a tag. > >> >> We will need some extra 'sed' trickery to create a regexp for each test case: >> -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test >> +tests.package.test_syslog_ng.TestSyslogNg: >> + <<: *runtime_test >> + only: >> + - triggers >> + - tags >> + - /^.*-runtime_test$/ >> + - /^.*-TestSyslogNg$/ > > Maybe you can use variables in there? So, in the only specification of > .runtime_test: > > only: > - triggers > - tags > - /^.*-runtime_test$/ > - /^.*-${CI_JOB_NAME}$/ > > Hm, unlikely, otherwise the other $ would have to be quoted. Maybe using %variable%. I didn't tested yet. > > Apparently nowadays there is also support for more complex only specifications > using variables. Perhaps you could take a look at that? I'm afraid though that > it's always an AND of the conditions you specify, so no way to specify that it's > for tags OR for things matching the CI_JOB_NAME. I will take a look. Regards, Ricardo
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index d6a2319ba0..d621692ce1 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -62,6 +62,11 @@ check-package: - output/build/packages-file-list.txt .runtime_test: &runtime_test + # Running the runtime tests for every push is too much, so limit to + # explicit triggers through the API. + only: + - triggers + - tags # Keep build directories so the rootfs can be an artifact of the job. The # runner will clean up those files for us. # Multiply every emulator timeout by 10 to avoid sporadic failures in @@ -261,71 +266,3 @@ zynq_zc706_defconfig: *defconfig zynq_zed_defconfig: *defconfig zynq_zybo_defconfig: *defconfig zynqmp_zcu106_defconfig: *defconfig -tests.boot.test_atf.TestATFAllwinner: *runtime_test -tests.boot.test_atf.TestATFMarvell: *runtime_test -tests.boot.test_atf.TestATFVexpress: *runtime_test -tests.core.test_file_capabilities.TestFileCapabilities: *runtime_test -tests.core.test_hardening.TestFortifyConserv: *runtime_test -tests.core.test_hardening.TestFortifyNone: *runtime_test -tests.core.test_hardening.TestRelro: *runtime_test -tests.core.test_hardening.TestRelroPartial: *runtime_test -tests.core.test_hardening.TestSspNone: *runtime_test -tests.core.test_hardening.TestSspStrong: *runtime_test -tests.core.test_post_scripts.TestPostScripts: *runtime_test -tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test -tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test -tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test -tests.core.test_timezone.TestNoTimezone: *runtime_test -tests.fs.test_ext.TestExt2: *runtime_test -tests.fs.test_ext.TestExt2r1: *runtime_test -tests.fs.test_ext.TestExt3: *runtime_test -tests.fs.test_ext.TestExt4: *runtime_test -tests.fs.test_iso9660.TestIso9660Grub2External: *runtime_test -tests.fs.test_iso9660.TestIso9660Grub2ExternalCompress: *runtime_test -tests.fs.test_iso9660.TestIso9660Grub2Internal: *runtime_test -tests.fs.test_iso9660.TestIso9660SyslinuxExternal: *runtime_test -tests.fs.test_iso9660.TestIso9660SyslinuxExternalCompress: *runtime_test -tests.fs.test_iso9660.TestIso9660SyslinuxInternal: *runtime_test -tests.fs.test_jffs2.TestJffs2: *runtime_test -tests.fs.test_squashfs.TestSquashfs: *runtime_test -tests.fs.test_ubi.TestUbi: *runtime_test -tests.fs.test_yaffs2.TestYaffs2: *runtime_test -tests.init.test_busybox.TestInitSystemBusyboxRo: *runtime_test -tests.init.test_busybox.TestInitSystemBusyboxRoNet: *runtime_test -tests.init.test_busybox.TestInitSystemBusyboxRw: *runtime_test -tests.init.test_busybox.TestInitSystemBusyboxRwNet: *runtime_test -tests.init.test_none.TestInitSystemNone: *runtime_test -tests.init.test_systemd.TestInitSystemSystemdRoFull: *runtime_test -tests.init.test_systemd.TestInitSystemSystemdRoIfupdown: *runtime_test -tests.init.test_systemd.TestInitSystemSystemdRoNetworkd: *runtime_test -tests.init.test_systemd.TestInitSystemSystemdRwFull: *runtime_test -tests.init.test_systemd.TestInitSystemSystemdRwIfupdown: *runtime_test -tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test -tests.package.test_dropbear.TestDropbear: *runtime_test -tests.package.test_ipython.TestIPythonPy2: *runtime_test -tests.package.test_ipython.TestIPythonPy3: *runtime_test -tests.package.test_python.TestPython2: *runtime_test -tests.package.test_python.TestPython3: *runtime_test -tests.package.test_python_autobahn.TestPythonPy2Autobahn: *runtime_test -tests.package.test_python_autobahn.TestPythonPy3Autobahn: *runtime_test -tests.package.test_python_cryptography.TestPythonPy2Cryptography: *runtime_test -tests.package.test_python_cryptography.TestPythonPy3Cryptography: *runtime_test -tests.package.test_python_incremental.TestPythonPy2Incremental: *runtime_test -tests.package.test_python_incremental.TestPythonPy3Incremental: *runtime_test -tests.package.test_python_twisted.TestPythonPy2Twisted: *runtime_test -tests.package.test_python_twisted.TestPythonPy3Twisted: *runtime_test -tests.package.test_python_txaio.TestPythonPy2Txaio: *runtime_test -tests.package.test_python_txaio.TestPythonPy3Txaio: *runtime_test -tests.package.test_python_txtorcon.TestPythonPy2Txtorcon: *runtime_test -tests.package.test_python_txtorcon.TestPythonPy3Txtorcon: *runtime_test -tests.package.test_rust.TestRust: *runtime_test -tests.package.test_rust.TestRustBin: *runtime_test -tests.package.test_syslog_ng.TestSyslogNg: *runtime_test -tests.toolchain.test_external.TestExternalToolchainBuildrootMusl: *runtime_test -tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc: *runtime_test -tests.toolchain.test_external.TestExternalToolchainCCache: *runtime_test -tests.toolchain.test_external.TestExternalToolchainCtngMusl: *runtime_test -tests.toolchain.test_external.TestExternalToolchainLinaroArm: *runtime_test -tests.toolchain.test_external.TestExternalToolchainSourceryArmv4: *runtime_test -tests.toolchain.test_external.TestExternalToolchainSourceryArmv5: *runtime_test -tests.toolchain.test_external.TestExternalToolchainSourceryArmv7: *runtime_test diff --git a/.gitlab-ci.yml.in b/.gitlab-ci.yml.in index 38e24c8d01..db526c4b5a 100644 --- a/.gitlab-ci.yml.in +++ b/.gitlab-ci.yml.in @@ -62,6 +62,11 @@ check-package: - output/build/packages-file-list.txt .runtime_test: &runtime_test + # Running the runtime tests for every push is too much, so limit to + # explicit triggers through the API. + only: + - triggers + - tags # Keep build directories so the rootfs can be an artifact of the job. The # runner will clean up those files for us. # Multiply every emulator timeout by 10 to avoid sporadic failures in
When the runtime tests were first introduced, they still ran pretty quickly. Nowadays, however, there are a lot of runtime tests, and some of them take a really long time. So running them on every push is really too much. Just like we do for the defconfigs, run them on explicit trigger only. The explicit trigger is now done every week, but it can be increased to e.g. twice or three times per week. Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> --- .gitlab-ci.yml | 73 ++++------------------------------------------- .gitlab-ci.yml.in | 5 ++++ 2 files changed, 10 insertions(+), 68 deletions(-)