| Message ID | 20200312193616.438922-5-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | QEMU Gating CI | expand |
On Thu, 12 Mar 2020 at 19:37, Cleber Rosa <crosa@redhat.com> wrote: > > This script is intended to be used right after a push to a branch. > > By default, it will look for the pipeline associated with the commit > that is the HEAD of the *local* staging branch. It can be used as a > one time check, or with the `--wait` option to wait until the pipeline > completes. > > If the pipeline is successful, then a merge of the staging branch into > the master branch should be the next step. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++ > 1 file changed, 148 insertions(+) > create mode 100755 contrib/ci/scripts/gitlab-pipeline-status > > diff --git a/contrib/ci/scripts/gitlab-pipeline-status b/contrib/ci/scripts/gitlab-pipeline-status > new file mode 100755 > index 0000000000..83d412daec > --- /dev/null > +++ b/contrib/ci/scripts/gitlab-pipeline-status > @@ -0,0 +1,148 @@ > +#!/usr/bin/env python3 > + > +""" > +Checks the GitLab pipeline status for a given commit commit > +""" All new files, and particularly new scripts and source files, should have the usual copyright-and-license comment at the top, please. thanks -- PMM
On Fri, Mar 13, 2020 at 01:56:51PM +0000, Peter Maydell wrote: > > All new files, and particularly new scripts and source > files, should have the usual copyright-and-license > comment at the top, please. > > thanks > -- PMM > My bad. I'll wait for further comments and included that in a v2. Cheers, - Cleber.
On Thu, Mar 12, 2020 at 03:36:15PM -0400, Cleber Rosa wrote: > This script is intended to be used right after a push to a branch. > > By default, it will look for the pipeline associated with the commit > that is the HEAD of the *local* staging branch. It can be used as a > one time check, or with the `--wait` option to wait until the pipeline > completes. > > If the pipeline is successful, then a merge of the staging branch into > the master branch should be the next step. On IRC yesterday we were discussing the current .gitlab-ci.yml status, and how frustrating it is that every time we get it green, more code is soon merged that turns it red again. It feels like it should be an easy win to declare that the current GitLab CI jobs are to be used as a gating tests for merges to master. As & when custom runners come online, their jobs can simply augment the existing jobs. IOW, use of GitLab CI for gating master shouldn't be dependant on setup of custom runners which we've been waiting on for a long term. Peter indicated that his main requirement is a way to automate the task of kicking off the CI job & getting its status. It seems like the script in this patch should fullfill that requirement. Assuming Peter (or whomever is going todo the merge) has a fork of https://gitlab.com/qemu-project/qemu then they need to find the "ID" number of their fork. This is visible at the top for the page for their fork eg mine: https://gitlab.com/berrange/qemu reports: "Project ID: 18588805 " Assuming the fork is configured as a git remote called "gitlab", then to use GitLab CI as gating test, all that appears to be needed is $ git push gitlab $ ./contrib/ci/scripts/gitlab-pipeline-status --wait -p 18588805 If this is an acceptable level of automation for Peter, then can we get this specific patch merged right now and make current GitLab CI be gating for master. With GitLab as gating, then we have further incentive to move all the jobs currently on Travis CI and Shippable, over to use GitLab too, and also use cirrus-run to make Cirrus CI jobs be proxied from GitLab. All this can be in parallel with adding custom GitLab runners for expanding testing coverage still further. > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++ > 1 file changed, 148 insertions(+) > create mode 100755 contrib/ci/scripts/gitlab-pipeline-status > > diff --git a/contrib/ci/scripts/gitlab-pipeline-status b/contrib/ci/scripts/gitlab-pipeline-status > new file mode 100755 > index 0000000000..83d412daec > --- /dev/null > +++ b/contrib/ci/scripts/gitlab-pipeline-status > @@ -0,0 +1,148 @@ > +#!/usr/bin/env python3 > + > +""" > +Checks the GitLab pipeline status for a given commit commit > +""" > + > +# pylint: disable=C0103 > + > +import argparse > +import http.client > +import json > +import os > +import subprocess > +import time > +import sys > + > + > +def get_local_staging_branch_commit(): > + """ > + Returns the commit sha1 for the *local* branch named "staging" > + """ > + result = subprocess.run(['git', 'rev-parse', 'staging'], > + stdin=subprocess.DEVNULL, > + stdout=subprocess.PIPE, > + stderr=subprocess.DEVNULL, > + cwd=os.path.dirname(__file__), > + universal_newlines=True).stdout.strip() > + if result == 'staging': > + raise ValueError("There's no local staging branch") > + if len(result) != 40: > + raise ValueError("Branch staging HEAD doesn't look like a sha1") > + return result > + > + > +def get_pipeline_status(project_id, commit_sha1): > + """ > + Returns the JSON content of the pipeline status API response > + """ > + url = '/api/v4/projects/{}/pipelines?sha={}'.format(project_id, > + commit_sha1) > + connection = http.client.HTTPSConnection('gitlab.com') > + connection.request('GET', url=url) > + response = connection.getresponse() > + if response.code != http.HTTPStatus.OK: > + raise ValueError("Failed to receive a successful response") > + json_response = json.loads(response.read()) > + # afaict, there should one one pipeline for the same project + commit > + # if this assumption is false, we can add further filters to the > + # url, such as username, and order_by. > + if not json_response: > + raise ValueError("No pipeline found") > + return json_response[0] > + > + > +def wait_on_pipeline_success(timeout, interval, > + project_id, commit_sha): > + """ > + Waits for the pipeline to end up to the timeout given > + """ > + start = time.time() > + while True: > + if time.time() >= (start + timeout): > + print("Waiting on the pipeline success timed out") > + return False > + > + status = get_pipeline_status(project_id, commit_sha) > + if status['status'] == 'running': > + time.sleep(interval) > + print('running...') > + continue > + > + if status['status'] == 'success': > + return True > + > + msg = "Pipeline ended unsuccessfully, check: %s" % status['web_url'] > + print(msg) > + return False > + > + > +def main(): > + """ > + Script entry point > + """ > + parser = argparse.ArgumentParser( > + prog='pipeline-status', > + description='check or wait on a pipeline status') > + > + parser.add_argument('-t', '--timeout', type=int, default=7200, > + help=('Amount of time (in seconds) to wait for the ' > + 'pipeline to complete. Defaults to ' > + '%(default)s')) > + parser.add_argument('-i', '--interval', type=int, default=60, > + help=('Amount of time (in seconds) to wait between ' > + 'checks of the pipeline status. Defaults ' > + 'to %(default)s')) > + parser.add_argument('-w', '--wait', action='store_true', default=False, > + help=('Wether to wait, instead of checking only once ' > + 'the status of a pipeline')) > + parser.add_argument('-p', '--project-id', type=int, default=11167699, > + help=('The GitLab project ID. Defaults to the project ' > + 'for https://gitlab.com/qemu-project/qemu, that ' > + 'is, "%(default)s"')) > + try: > + default_commit = get_local_staging_branch_commit() > + commit_required = False > + except ValueError: > + default_commit = '' > + commit_required = True > + parser.add_argument('-c', '--commit', required=commit_required, > + default=default_commit, > + help=('Look for a pipeline associated with the given ' > + 'commit. If one is not explicitly given, the ' > + 'commit associated with the local branch named ' > + '"staging" is used. Default: %(default)s')) > + parser.add_argument('--verbose', action='store_true', default=False, > + help=('A minimal verbosity level that prints the ' > + 'overall result of the check/wait')) > + > + args = parser.parse_args() > + > + try: > + if args.wait: > + success = wait_on_pipeline_success( > + args.timeout, > + args.interval, > + args.project_id, > + args.commit) > + else: > + status = get_pipeline_status(args.project_id, > + args.commit) > + success = status['status'] == 'success' > + except Exception as error: # pylint: disable=W0703 > + success = False > + if args.verbose: > + print("ERROR: %s" % error.args[0]) > + > + if success: > + if args.verbose: > + print('success') > + sys.exit(0) > + else: > + if args.verbose: > + print('failure') > + sys.exit(1) > + > + > +if __name__ == '__main__': > + main() > -- > 2.24.1 > > Regards, Daniel
On Thu, Jun 18, 2020 at 12:45:24PM +0100, Daniel P. Berrangé wrote: > On Thu, Mar 12, 2020 at 03:36:15PM -0400, Cleber Rosa wrote: > > This script is intended to be used right after a push to a branch. > > > > By default, it will look for the pipeline associated with the commit > > that is the HEAD of the *local* staging branch. It can be used as a > > one time check, or with the `--wait` option to wait until the pipeline > > completes. > > > > If the pipeline is successful, then a merge of the staging branch into > > the master branch should be the next step. > > On IRC yesterday we were discussing the current .gitlab-ci.yml status, > and how frustrating it is that every time we get it green, more code is > soon merged that turns it red again. > Hi Daniel, I know this is nothing new to you, but "green" turning "red" is something that can be minimized, but never completely abolished. We've had discussions on how we could minimize those, and suggestions on how to address those occurrences. The points that I remember about it are: 1. For jobs supposed to run on containers and vms, use images prepared to run the builds and tests, that is, don't do the prep steps (package installations and others) within the job itself - this is currently missing. 2. For bare metal, have the setup either documented or scripted and run before the jobs (in the patch series I sent, these were done in a number of Ansible based playbooks). 3. Have a mechanism for promoting non-gating jobs to gating jobs. Keeping track of their reliability over a period of time is a way to forecast it future behavior, as much as it can be done. 4. Have a MAINTAINERS entry for those jobs, and have maintainers attemtp to address issues within a specified amount of time; if that fails, have a mechanism to downgrade those jobs to non-gating. A patch removing/moving entries from .gitlab-ci*.yml would do that and of course sping another set of CI jobs bringing things back to green. > It feels like it should be an easy win to declare that the current GitLab > CI jobs are to be used as a gating tests for merges to master. > I'm fine with that approach if most people agree. This would mean point #1 is ignored for now. For everyone's sake, points #3 and #4 are a must IMO. Point #2 will applied as we introduced our non-shared gitlab runners. > As & when custom runners come online, their jobs can simply augment the > existing jobs. IOW, use of GitLab CI for gating master shouldn't be > dependant on setup of custom runners which we've been waiting on for a > long term. > > Peter indicated that his main requirement is a way to automate the task > of kicking off the CI job & getting its status. It seems like the script > in this patch should fullfill that requirement. > > Assuming Peter (or whomever is going todo the merge) has a fork of > > https://gitlab.com/qemu-project/qemu > > then they need to find the "ID" number of their fork. This is > visible at the top for the page for their fork eg mine: > > https://gitlab.com/berrange/qemu > > reports: > > "Project ID: 18588805 " > > Assuming the fork is configured as a git remote called "gitlab", then > to use GitLab CI as gating test, all that appears to be needed is > > $ git push gitlab > $ ./contrib/ci/scripts/gitlab-pipeline-status --wait -p 18588805 > > If this is an acceptable level of automation for Peter, then can we > get this specific patch merged right now and make current GitLab CI > be gating for master. > > > With GitLab as gating, then we have further incentive to move all > the jobs currently on Travis CI and Shippable, over to use GitLab > too, and also use cirrus-run to make Cirrus CI jobs be proxied > from GitLab. All this can be in parallel with adding custom GitLab > runners for expanding testing coverage still further. > The script should indeed work with the workflow you described. I'll just run a few tests, to make sure that are no caveats, and I'll let you and Peter know about it. Thanks, - Cleber.
On Mon, Jun 22, 2020 at 10:21:03AM -0400, Cleber Rosa wrote: > > The script should indeed work with the workflow you described. I'll just > run a few tests, to make sure that are no caveats, and I'll let you and > Peter know about it. > FIY, I've cherry-pick just the patch with the scripts (for no reason other than having some content) and pushed it to a different branch than staging: https://gitlab.com/cleber.gnu/qemuci/-/commits/exp/script_shared_runners Then I ran: ./contrib/ci/scripts/gitlab-pipeline-status --wait --verbose -p 15173319 -c ef12b411985baab9071e5fab1851acdd07d9bec8 Which worked as expected. BTW, the commit wouldn't be necessary if I was checking on a "staging" branch. The pipeline triggered (and being waited on by the script is at): https://gitlab.com/cleber.gnu/qemuci/-/pipelines/159334485 So I believe it's now a matter of sorting out job the inclusion/exclusion aspects, that is, if we start with all/some of the current jobs and some or none of the jobs intended to run on the s390/aarch64/x86_64 (non-shared) runners. Thanks, - Cleber.
On 6/23/20 7:59 PM, Cleber Rosa wrote: > On Mon, Jun 22, 2020 at 10:21:03AM -0400, Cleber Rosa wrote: >> >> The script should indeed work with the workflow you described. I'll just >> run a few tests, to make sure that are no caveats, and I'll let you and >> Peter know about it. >> > > FIY, I've cherry-pick just the patch with the scripts (for no reason other > than having some content) and pushed it to a different branch than staging: > > https://gitlab.com/cleber.gnu/qemuci/-/commits/exp/script_shared_runners > > Then I ran: > > ./contrib/ci/scripts/gitlab-pipeline-status --wait --verbose -p 15173319 -c ef12b411985baab9071e5fab1851acdd07d9bec8 > > Which worked as expected. Great news! Can you respin as a new series with this as single patch, and explanation in the cover about how to setup it? This is the last piece missing for Peter to use the effort done by Alex/Daniel/Thomas on the GitLab side :) > BTW, the commit wouldn't be necessary if I > was checking on a "staging" branch. The pipeline triggered (and being > waited on by the script is at): > > https://gitlab.com/cleber.gnu/qemuci/-/pipelines/159334485 > > So I believe it's now a matter of sorting out job the > inclusion/exclusion aspects, that is, if we start with all/some of the > current jobs and some or none of the jobs intended to run on the > s390/aarch64/x86_64 (non-shared) runners. > > Thanks, > - Cleber. >
diff --git a/contrib/ci/scripts/gitlab-pipeline-status b/contrib/ci/scripts/gitlab-pipeline-status new file mode 100755 index 0000000000..83d412daec --- /dev/null +++ b/contrib/ci/scripts/gitlab-pipeline-status @@ -0,0 +1,148 @@ +#!/usr/bin/env python3 + +""" +Checks the GitLab pipeline status for a given commit commit +""" + +# pylint: disable=C0103 + +import argparse +import http.client +import json +import os +import subprocess +import time +import sys + + +def get_local_staging_branch_commit(): + """ + Returns the commit sha1 for the *local* branch named "staging" + """ + result = subprocess.run(['git', 'rev-parse', 'staging'], + stdin=subprocess.DEVNULL, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + cwd=os.path.dirname(__file__), + universal_newlines=True).stdout.strip() + if result == 'staging': + raise ValueError("There's no local staging branch") + if len(result) != 40: + raise ValueError("Branch staging HEAD doesn't look like a sha1") + return result + + +def get_pipeline_status(project_id, commit_sha1): + """ + Returns the JSON content of the pipeline status API response + """ + url = '/api/v4/projects/{}/pipelines?sha={}'.format(project_id, + commit_sha1) + connection = http.client.HTTPSConnection('gitlab.com') + connection.request('GET', url=url) + response = connection.getresponse() + if response.code != http.HTTPStatus.OK: + raise ValueError("Failed to receive a successful response") + json_response = json.loads(response.read()) + # afaict, there should one one pipeline for the same project + commit + # if this assumption is false, we can add further filters to the + # url, such as username, and order_by. + if not json_response: + raise ValueError("No pipeline found") + return json_response[0] + + +def wait_on_pipeline_success(timeout, interval, + project_id, commit_sha): + """ + Waits for the pipeline to end up to the timeout given + """ + start = time.time() + while True: + if time.time() >= (start + timeout): + print("Waiting on the pipeline success timed out") + return False + + status = get_pipeline_status(project_id, commit_sha) + if status['status'] == 'running': + time.sleep(interval) + print('running...') + continue + + if status['status'] == 'success': + return True + + msg = "Pipeline ended unsuccessfully, check: %s" % status['web_url'] + print(msg) + return False + + +def main(): + """ + Script entry point + """ + parser = argparse.ArgumentParser( + prog='pipeline-status', + description='check or wait on a pipeline status') + + parser.add_argument('-t', '--timeout', type=int, default=7200, + help=('Amount of time (in seconds) to wait for the ' + 'pipeline to complete. Defaults to ' + '%(default)s')) + parser.add_argument('-i', '--interval', type=int, default=60, + help=('Amount of time (in seconds) to wait between ' + 'checks of the pipeline status. Defaults ' + 'to %(default)s')) + parser.add_argument('-w', '--wait', action='store_true', default=False, + help=('Wether to wait, instead of checking only once ' + 'the status of a pipeline')) + parser.add_argument('-p', '--project-id', type=int, default=11167699, + help=('The GitLab project ID. Defaults to the project ' + 'for https://gitlab.com/qemu-project/qemu, that ' + 'is, "%(default)s"')) + try: + default_commit = get_local_staging_branch_commit() + commit_required = False + except ValueError: + default_commit = '' + commit_required = True + parser.add_argument('-c', '--commit', required=commit_required, + default=default_commit, + help=('Look for a pipeline associated with the given ' + 'commit. If one is not explicitly given, the ' + 'commit associated with the local branch named ' + '"staging" is used. Default: %(default)s')) + parser.add_argument('--verbose', action='store_true', default=False, + help=('A minimal verbosity level that prints the ' + 'overall result of the check/wait')) + + args = parser.parse_args() + + try: + if args.wait: + success = wait_on_pipeline_success( + args.timeout, + args.interval, + args.project_id, + args.commit) + else: + status = get_pipeline_status(args.project_id, + args.commit) + success = status['status'] == 'success' + except Exception as error: # pylint: disable=W0703 + success = False + if args.verbose: + print("ERROR: %s" % error.args[0]) + + if success: + if args.verbose: + print('success') + sys.exit(0) + else: + if args.verbose: + print('failure') + sys.exit(1) + + +if __name__ == '__main__': + main()
This script is intended to be used right after a push to a branch. By default, it will look for the pipeline associated with the commit that is the HEAD of the *local* staging branch. It can be used as a one time check, or with the `--wait` option to wait until the pipeline completes. If the pipeline is successful, then a merge of the staging branch into the master branch should be the next step. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- contrib/ci/scripts/gitlab-pipeline-status | 148 ++++++++++++++++++++++ 1 file changed, 148 insertions(+) create mode 100755 contrib/ci/scripts/gitlab-pipeline-status