| Message ID | 20200830155953.1650346-1-arnout@mind.be |
|---|---|
| State | Rejected |
| Headers | show |
| Series | support/misc/gitlab-ci.yml.in: include Branches workflow | expand |
Arnout, All, +Romain and +Jugurtha, as we've had some discussions on that topic publicly and privately before. On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly: > Recently, gitlab-ci has gained a lot more flexibility in when pipelines > are created. As a side-effect of this, double pipelines may be created > when a `when` clause is used in some job. Avoid this by adding a > workflow that launches it only on branches. See [1] and [2]. > > Note that in reality, the duplicate pipelines only occur for merge > requests. Since we don't have merge requests, we don't have this > problem. Howver, gitlab also displays an annoying error message on the > pipeline page [3], and it sends an error mail to the triggerer (i.e., > Arnout), so it's still useful to do this. > > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines > [2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates > [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds > > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> > --- > I'd like to get this one in master, because the e-mails are annoying me > :-) I'd like to suggest an alternative: move all the conditions into the generating script, now that we do have a script that generates the pipeline description. Having the conditions in the script will help handle such cases, by only ever emitting those jobs we actually want to run. This would allow for more flexibility than the limited micro-language used by gitlab-ci.yml. Also, this will allow us to not care about the evolutions of that micro- language. To that effect, I already started workign on that two weeks ago, and I have omething that works as before (even better I think): https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond I just need to cleanup that last commit (and remember why I started doing it. meh...). I hope I'll be able to send it tonight. Regards, Yann E. MORIN. > --- > support/misc/gitlab-ci.yml.in | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in > index dddebf09e9..8a031898ef 100644 > --- a/support/misc/gitlab-ci.yml.in > +++ b/support/misc/gitlab-ci.yml.in > @@ -1,6 +1,9 @@ > # Configuration for Gitlab-CI. > # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines > > +include: > + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' > + > image: buildroot/base:20200814.2228 > > .check_base: > -- > 2.26.2 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
On 31/08/2020 14:43, Yann E. MORIN wrote: > Arnout, All, > > +Romain and +Jugurtha, as we've had some discussions on that topic > publicly and privately before. > > On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly: >> Recently, gitlab-ci has gained a lot more flexibility in when pipelines >> are created. As a side-effect of this, double pipelines may be created >> when a `when` clause is used in some job. Avoid this by adding a >> workflow that launches it only on branches. See [1] and [2]. >> >> Note that in reality, the duplicate pipelines only occur for merge >> requests. Since we don't have merge requests, we don't have this >> problem. Howver, gitlab also displays an annoying error message on the >> pipeline page [3], and it sends an error mail to the triggerer (i.e., >> Arnout), so it's still useful to do this. >> >> [1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines >> [2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates >> [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds >> >> Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> >> --- >> I'd like to get this one in master, because the e-mails are annoying me >> :-) > > I'd like to suggest an alternative: move all the conditions into the > generating script, now that we do have a script that generates the > pipeline description. Note that it's not the complicated conditions on the defconfig and test jobs that are the issue. The warning is also issued for check-flake8... So I think the workflow will still be needed (or alternatively, every job should have a condition to run only on branches). Also, I would like to move the "constant" bits back out to the top-level gitlab-ci.yml, so the first page already shows the quick test results. > Having the conditions in the script will help handle such cases, by only > ever emitting those jobs we actually want to run. This would allow for > more flexibility than the limited micro-language used by gitlab-ci.yml. > > Also, this will allow us to not care about the evolutions of that micro- > language. > > To that effect, I already started workign on that two weeks ago, and I > have omething that works as before (even better I think): > > https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond > > I just need to cleanup that last commit (and remember why I started > doing it. meh...). I hope I'll be able to send it tonight. I think that's a great idea. However, that sounds like a bigger change which probably shouldn't go into master. So I'd prefer to still merge my patch as well. Regards, Arnout > > Regards, > Yann E. MORIN. > >> --- >> support/misc/gitlab-ci.yml.in | 3 +++ >> 1 file changed, 3 insertions(+) >> >> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in >> index dddebf09e9..8a031898ef 100644 >> --- a/support/misc/gitlab-ci.yml.in >> +++ b/support/misc/gitlab-ci.yml.in >> @@ -1,6 +1,9 @@ >> # Configuration for Gitlab-CI. >> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines >> >> +include: >> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' >> + >> image: buildroot/base:20200814.2228 >> >> .check_base: >> -- >> 2.26.2 >> >> _______________________________________________ >> buildroot mailing list >> buildroot@busybox.net >> http://lists.busybox.net/mailman/listinfo/buildroot >
Arnout, All, On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly: > Note that it's not the complicated conditions on the defconfig and test jobs > that are the issue. The warning is also issued for check-flake8... So I think > the workflow will still be needed (or alternatively, every job should have a > condition to run only on branches). Sorry, but I don't follow: if the jobs are only generated when we want them, then they no longer have conditions in the generated gitlab-ci.yml to begin with; that's the whole point of moving the conditional handling into the script. > On 31/08/2020 14:43, Yann E. MORIN wrote: > > I'd like to suggest an alternative: move all the conditions into the > > generating script, now that we do have a script that generates the > > pipeline description. [--SNIP--] > Also, I would like to move the "constant" bits back out to the top-level > gitlab-ci.yml, so the first page already shows the quick test results. Sorry, but again they are not "constant". For example, check-flake8, check-package and check-DEVELOPERS are not run when one pushes a *_defconfig or *_test branch (or however they are named) to trigger the build of jsut that defconfig or just that runtime test. > > Having the conditions in the script will help handle such cases, by only > > ever emitting those jobs we actually want to run. This would allow for > > more flexibility than the limited micro-language used by gitlab-ci.yml. > > > > Also, this will allow us to not care about the evolutions of that micro- > > language. > I think that's a great idea. However, that sounds like a bigger change which > probably shouldn't go into master. So I'd prefer to still merge my patch as well. See an actual comment on that patch, below... > >> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in > >> index dddebf09e9..8a031898ef 100644 > >> --- a/support/misc/gitlab-ci.yml.in > >> +++ b/support/misc/gitlab-ci.yml.in > >> @@ -1,6 +1,9 @@ > >> # Configuration for Gitlab-CI. > >> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines > >> > >> +include: > >> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' Where does that file come from? How are we guaranteed that its content does not change and break our pipelines? If that template is interesting, then we should carry it rather than depend on an external, un-managed entity... Regards, Yann E. MORIN. > >> image: buildroot/base:20200814.2228 > >> > >> .check_base: > >> -- > >> 2.26.2 > >> > >> _______________________________________________ > >> buildroot mailing list > >> buildroot@busybox.net > >> http://lists.busybox.net/mailman/listinfo/buildroot > >
Arnout, Jugurtha, All, On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly: > On 31/08/2020 14:43, Yann E. MORIN wrote: > > +Romain and +Jugurtha, as we've had some discussions on that topic > > publicly and privately before. [--SNIP--] > > To that effect, I already started workign on that two weeks ago, and I > > have omething that works as before (even better I think): > > https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/gitlab-ci-cond > > I just need to cleanup that last commit (and remember why I started > > doing it. meh...). I hope I'll be able to send it tonight. And of course, as usual, I did not have time to properly finish that branch... Still, it is very close to be ready, with all the test cases now working as expected: Random branch, only basic tests https://gitlab.com/ymorin/buildroot/-/commits/yem/gitlab-ci-cond https://gitlab.com/ymorin/buildroot/-/pipelines/184068345 All defconfigs: https://gitlab.com/ymorin/buildroot/-/commits/FOO-defconfigs https://gitlab.com/ymorin/buildroot/-/pipelines/184069188 One defconfig: https://gitlab.com/ymorin/buildroot/-/commits/FOO-warp7_defconfig https://gitlab.com/ymorin/buildroot/-/pipelines/184068523 All runtime tests: https://gitlab.com/ymorin/buildroot/-/commits/FOO-runtime-tests https://gitlab.com/ymorin/buildroot/-/pipelines/184070380 One runtime test: https://gitlab.com/ymorin/buildroot/-/commits/FOO-tests.package.test_openssh https://gitlab.com/ymorin/buildroot/-/pipelines/184068633 I still have to test a maintenance branch (by faking one with a branch named 'fooo.no.x' to match the pattern), but I'm afraid I've reached my allowed number of jobs for now... Regards, Yann E. MORIN.
On 31/08/2020 21:52, Yann E. MORIN wrote: > Arnout, All, > > On 2020-08-31 14:58 +0200, Arnout Vandecappelle spake thusly: >> Note that it's not the complicated conditions on the defconfig and test jobs >> that are the issue. The warning is also issued for check-flake8... So I think >> the workflow will still be needed (or alternatively, every job should have a >> condition to run only on branches). > > Sorry, but I don't follow: if the jobs are only generated when we want > them, then they no longer have conditions in the generated gitlab-ci.yml > to begin with; that's the whole point of moving the conditional handling > into the script. Ah, OK, I thought it was just about generating the defconfig and tests parts. My bad. >> On 31/08/2020 14:43, Yann E. MORIN wrote: [snip] > > See an actual comment on that patch, below... > >>>> diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in >>>> index dddebf09e9..8a031898ef 100644 >>>> --- a/support/misc/gitlab-ci.yml.in >>>> +++ b/support/misc/gitlab-ci.yml.in >>>> @@ -1,6 +1,9 @@ >>>> # Configuration for Gitlab-CI. >>>> # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines >>>> >>>> +include: >>>> + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' > > Where does that file come from? How are we guaranteed that its content > does not change and break our pipelines? > > If that template is interesting, then we should carry it rather than > depend on an external, un-managed entity... It is part of the gitlab installation. That means that indeed it is not under our control, but neither is the rest of gitlab (e.g., the part that generates those warnings :-). In other words: it's about as reliable as the rules clauses we have now. Note that the workflow file is indeed very simple, and we could simply instantiate it here. Again, though, in order to expedite the 2020.08 release, I propose to apply this simple patch to master, and soon after merge your series that makes it script-based - possibly backporting it to the stable branches, though I don't really think that is needed. Regards, Arnout
Arnout, All, On 2020-08-30 17:59 +0200, Arnout Vandecappelle (Essensium/Mind) spake thusly: > Recently, gitlab-ci has gained a lot more flexibility in when pipelines > are created. As a side-effect of this, double pipelines may be created > when a `when` clause is used in some job. Avoid this by adding a > workflow that launches it only on branches. See [1] and [2]. > > Note that in reality, the duplicate pipelines only occur for merge > requests. Since we don't have merge requests, we don't have this > problem. Howver, gitlab also displays an annoying error message on the > pipeline page [3], and it sends an error mail to the triggerer (i.e., > Arnout), so it's still useful to do this. > > [1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines > [2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates > [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds > > Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> We've no merged a better script that contains all the logic about generating the child pipeline, so I believe this patch of yours is no longer needed. Regards, Yann E. MORIN. > --- > I'd like to get this one in master, because the e-mails are annoying me > :-) > --- > support/misc/gitlab-ci.yml.in | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in > index dddebf09e9..8a031898ef 100644 > --- a/support/misc/gitlab-ci.yml.in > +++ b/support/misc/gitlab-ci.yml.in > @@ -1,6 +1,9 @@ > # Configuration for Gitlab-CI. > # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines > > +include: > + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' > + > image: buildroot/base:20200814.2228 > > .check_base: > -- > 2.26.2 > > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
diff --git a/support/misc/gitlab-ci.yml.in b/support/misc/gitlab-ci.yml.in index dddebf09e9..8a031898ef 100644 --- a/support/misc/gitlab-ci.yml.in +++ b/support/misc/gitlab-ci.yml.in @@ -1,6 +1,9 @@ # Configuration for Gitlab-CI. # Builds appear on https://gitlab.com/buildroot.org/buildroot/pipelines +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' + image: buildroot/base:20200814.2228 .check_base:
Recently, gitlab-ci has gained a lot more flexibility in when pipelines are created. As a side-effect of this, double pipelines may be created when a `when` clause is used in some job. Avoid this by adding a workflow that launches it only on branches. See [1] and [2]. Note that in reality, the duplicate pipelines only occur for merge requests. Since we don't have merge requests, we don't have this problem. Howver, gitlab also displays an annoying error message on the pipeline page [3], and it sends an error mail to the triggerer (i.e., Arnout), so it's still useful to do this. [1] https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines [2] https://docs.gitlab.com/ee/ci/yaml/README.html#workflowrules-templates [3] https://gitlab.com/buildroot.org/buildroot/-/pipelines/183589361/builds Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> --- I'd like to get this one in master, because the e-mails are annoying me :-) --- support/misc/gitlab-ci.yml.in | 3 +++ 1 file changed, 3 insertions(+)