| Message ID | 20201019015003.1527746-4-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | GitLab Custom Runners and Jobs (was: QEMU Gating CI) | expand |
On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote: > To have the jobs dispatched to custom runners, gitlab-runner must > be installed, active as a service and properly configured. The > variables file and playbook introduced here should help with those > steps. > > The playbook introduced here covers a number of different Linux > distributions and FreeBSD, and are intended to provide a reproducible > environment. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> > --- In general, there's been put quite some effort into the playbooks - sorry I'm late to the game - is there a plan to introduce QEMU as a project to lcitool? We've taken care of most of the bits in the playbooks that are being introduced and for the remaining ones I think it would be that big of an overhaul to do the adjustments. One major re-factor though would IMO be to break the dependency lcitool has on the machine naming, kind of restricting it to a limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which makes it inconvenient to prepare physical hosts. More comments inline... > docs/devel/ci.rst | 63 ++++++++++++++++++++++++++ > scripts/ci/setup/.gitignore | 1 + > scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++ > scripts/ci/setup/vars.yml.template | 13 ++++++ > 4 files changed, 149 insertions(+) > create mode 100644 scripts/ci/setup/.gitignore > create mode 100644 scripts/ci/setup/gitlab-runner.yml > create mode 100644 scripts/ci/setup/vars.yml.template > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst > index 208b5e399b..a234a5e24c 100644 > --- a/docs/devel/ci.rst > +++ b/docs/devel/ci.rst > @@ -84,3 +84,66 @@ To run the playbook, execute:: > > cd scripts/ci/setup > ansible-playbook -i inventory build-environment.yml > + > +gitlab-runner setup and registration > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +The gitlab-runner agent needs to be installed on each machine that > +will run jobs. The association between a machine and a GitLab project > +happens with a registration token. To find the registration token for > +your repository/project, navigate on GitLab's web UI to: > + > + * Settings (the gears like icon), then > + * CI/CD, then > + * Runners, and click on the "Expand" button, then > + * Under "Set up a specific Runner manually", look for the value under > + "Use the following registration token during setup" > + > +Copy the ``scripts/ci/setup/vars.yml.template`` file to > +``scripts/ci/setup/vars.yml``. Then, set the > +``gitlab_runner_registration_token`` variable to the value obtained > +earlier. > + > +.. note:: gitlab-runner is not available from the standard location > + for all OS and architectures combinations. For some systems, > + a custom build may be necessary. Some builds are avaiable > + at https://cleber.fedorapeople.org/gitlab-runner/ and this > + URI may be used as a value on ``vars.yml`` Yes, this can be suboptimal...Would it make sense to fall back to build the binary of a given version from git as a fallback during this playbook if the necessary arch version isn't provided the official way? Just an idea, I'd like to avoid the need for you to become the maintainer of the binaries and keep up with the releases. > + > +To run the playbook, execute:: > + > + cd scripts/ci/setup > + ansible-playbook -i inventory gitlab-runner.yml > + > +.. note:: there are currently limitations to gitlab-runner itself when > + setting up a service under FreeBSD systems. You will need to > + perform steps 4 to 10 manually, as described at Which one of them is considered an automation problem? In lcitool we made gitlab-runner completely automated on all distros, including FreeBSD: 4) log file permissions - you're creating the user, you can as well create the file with correct permissions 5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml it will pull a bunch of dependencies which ensure the dir exists 6) gitlab_runner service script should IMO provided by this automation as a template and install to the correct location 7) ensure the service script is executable - template module can do that 8) register the runner - you're doing that 9) enable the service - Ansible's service module is generic and supports init/OpenRC 10) I don't see a step 10 IOW I think there should be as little manual intervention as possible and in this case I don't think any manual steps are needed by the user. Regards, Erik > + https://docs.gitlab.com/runner/install/freebsd.html > + > +Following the registration, it's necessary to configure the runner tags, > +and optionally other configurations on the GitLab UI. Navigate to: > + > + * Settings (the gears like icon), then > + * CI/CD, then > + * Runners, and click on the "Expand" button, then > + * "Runners activated for this project", then > + * Click on the "Edit" icon (next to the "Lock" Icon) > + > +Under tags, add values matching the jobs a runner should run. For a > +FreeBSD 12.1 x86_64 system, the tags should be set as:: > + > + freebsd12.1,x86_64 > + > +Because the job definition at ``.gitlab-ci.d/custom-runners.yml`` > +would contain:: > + > + freebsd-12.1-x86_64-all: > + tags: > + - freebsd_12.1 > + - x86_64 > + > +It's also recommended to: > + > + * increase the "Maximum job timeout" to something like ``2h`` > + * uncheck the "Run untagged jobs" check box > + * give it a better Description > diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore > new file mode 100644 > index 0000000000..f112d05dd0 > --- /dev/null > +++ b/scripts/ci/setup/.gitignore > @@ -0,0 +1 @@ > +vars.yml > \ No newline at end of file > diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml > new file mode 100644 > index 0000000000..c2f52dad10 > --- /dev/null > +++ b/scripts/ci/setup/gitlab-runner.yml > @@ -0,0 +1,72 @@ > +--- > +- name: Installation of gitlab-runner > + hosts: all > + vars_files: > + - vars.yml > + tasks: > + - debug: > + msg: 'Checking for a valid GitLab registration token' > + failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'" > + > + - name: Checks the availability of official gitlab-runner builds in the archive > + uri: > + url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-linux-386 > + method: HEAD > + status_code: > + - 200 > + - 403 Why is 403 an acceptable success status code? > + register: gitlab_runner_available_archive > + > + - name: Update base url > + set_fact: > + gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner- > + when: gitlab_runner_available_archive.status == 200 > + - debug: > + msg: Base gitlab-runner url is {{ gitlab_runner_base_url }} > + > + - name: Set OS name (FreeBSD) > + set_fact: > + gitlab_runner_os: freebsd > + when: "ansible_facts['system'] == 'FreeBSD'" > + > + - name: Create a group for the gitlab-runner service > + group: > + name: gitlab-runner > + > + - name: Create a user for the gitlab-runner service > + user: > + user: gitlab-runner > + group: gitlab-runner > + comment: GitLab Runner > + home: /home/gitlab-runner > + shell: /bin/bash > + > + - name: Remove the .bash_logout file when on Ubuntu systems > + file: > + path: /home/gitlab-runner/.bash_logout > + state: absent > + when: "ansible_facts['distribution'] == 'Ubuntu'" > + > + - name: Downloads the matching gitlab-runner > + get_url: > + dest: /usr/local/bin/gitlab-runner > + url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}" > + owner: gitlab-runner > + group: gitlab-runner > + mode: u=rwx,g=rwx,o=rx > + > + - name: Register the gitlab-runner > + command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" > + > + - name: Install the gitlab-runner service using its own functionality > + command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner > + register: gitlab_runner_install_service_result > + failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr" > + when: "ansible_facts['os_family'] != 'FreeBSD'" > + > + - name: Enable the gitlab-runner service > + service: > + name: gitlab-runner > + state: started > + enabled: yes > + when: "ansible_facts['os_family'] != 'FreeBSD'" > diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template > new file mode 100644 > index 0000000000..621435d030 > --- /dev/null > +++ b/scripts/ci/setup/vars.yml.template > @@ -0,0 +1,13 @@ > +# The version of the gitlab-runner to use > +gitlab_runner_version: 13.1.1 > +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH > +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner- > +# The URL of the gitlab server to use, usually https://gitlab.com unless you're > +# using a private GitLab instance > +gitlab_runner_server_url: https://gitlab.com > +# Defaults to linux, checks can be used to change this > +gitlab_runner_os: linux > +# Defaults to amd64 (x86_64), checks can be used to change this > +gitlab_runner_arch: amd64 > +# A unique token made available by GitLab to your project for registering runners > +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN > -- > 2.25.4 >
On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote: > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote: > > To have the jobs dispatched to custom runners, gitlab-runner must > > be installed, active as a service and properly configured. The > > variables file and playbook introduced here should help with those > > steps. > > > > The playbook introduced here covers a number of different Linux > > distributions and FreeBSD, and are intended to provide a reproducible > > environment. > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> > > --- > > In general, there's been put quite some effort into the playbooks - sorry I'm > late to the game - is there a plan to introduce QEMU as a project to lcitool? I think it's becoming quite clear that having so much duplication (in the dockerfiles, tests/vm, this playbook, etc) is costly and error prone. I don't know if anyone has invested time in a PoC to consolidate those (with lcitool), but I can certainly see the upside to that. BTW, are you volunteering (wink wink)? :) > We've taken care of most of the bits in the playbooks that are being introduced > and for the remaining ones I think it would be that big of an overhaul to do > the adjustments. One major re-factor though would IMO be to break the > dependency lcitool has on the machine naming, kind of restricting it to a > limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which > makes it inconvenient to prepare physical hosts. > Right... I wasn't aware of that depedency. And, this may be a nice project to make sure that lcitool doesn't have any other libvirt specificities. > More comments inline... > > > docs/devel/ci.rst | 63 ++++++++++++++++++++++++++ > > scripts/ci/setup/.gitignore | 1 + > > scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++ > > scripts/ci/setup/vars.yml.template | 13 ++++++ > > 4 files changed, 149 insertions(+) > > create mode 100644 scripts/ci/setup/.gitignore > > create mode 100644 scripts/ci/setup/gitlab-runner.yml > > create mode 100644 scripts/ci/setup/vars.yml.template > > > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst > > index 208b5e399b..a234a5e24c 100644 > > --- a/docs/devel/ci.rst > > +++ b/docs/devel/ci.rst > > @@ -84,3 +84,66 @@ To run the playbook, execute:: > > > > cd scripts/ci/setup > > ansible-playbook -i inventory build-environment.yml > > + > > +gitlab-runner setup and registration > > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > + > > +The gitlab-runner agent needs to be installed on each machine that > > +will run jobs. The association between a machine and a GitLab project > > +happens with a registration token. To find the registration token for > > +your repository/project, navigate on GitLab's web UI to: > > + > > + * Settings (the gears like icon), then > > + * CI/CD, then > > + * Runners, and click on the "Expand" button, then > > + * Under "Set up a specific Runner manually", look for the value under > > + "Use the following registration token during setup" > > + > > +Copy the ``scripts/ci/setup/vars.yml.template`` file to > > +``scripts/ci/setup/vars.yml``. Then, set the > > +``gitlab_runner_registration_token`` variable to the value obtained > > +earlier. > > + > > +.. note:: gitlab-runner is not available from the standard location > > + for all OS and architectures combinations. For some systems, > > + a custom build may be necessary. Some builds are avaiable > > + at https://cleber.fedorapeople.org/gitlab-runner/ and this > > + URI may be used as a value on ``vars.yml`` > > Yes, this can be suboptimal...Would it make sense to fall back to build the > binary of a given version from git as a fallback during this playbook if the > necessary arch version isn't provided the official way? Just an idea, I'd like > to avoid the need for you to become the maintainer of the binaries and keep up > with the releases. > Well, building them during the playbook would be a lot more complex... You can have your own repo with your own builds, and just tweak your vars.yml. > > + > > +To run the playbook, execute:: > > + > > + cd scripts/ci/setup > > + ansible-playbook -i inventory gitlab-runner.yml > > + > > +.. note:: there are currently limitations to gitlab-runner itself when > > + setting up a service under FreeBSD systems. You will need to > > + perform steps 4 to 10 manually, as described at > > Which one of them is considered an automation problem? In lcitool we made > gitlab-runner completely automated on all distros, including FreeBSD: > It looks like lcitool went the more practical route. I was hoping to not have to treat gitlab-runner in such a special way in any "supported" OS. What I mean is, I'd rather write the code within gitlab-runner (or reespective libraries). Of course, I did not get to it, so that's why I just documented the steps here. I'll take a look at lcitool's playbook to see if I can easily canibalize some of that. But, at this time, we don't runners for FreeBSD anyway, so I guess this is not *that* urgent. > 4) log file permissions - you're creating the user, you can as well create the > file with correct permissions > > 5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml > it will pull a bunch of dependencies which ensure the dir exists > > 6) gitlab_runner service script should IMO provided by this automation as a > template and install to the correct location > > 7) ensure the service script is executable - template module can do that > > 8) register the runner - you're doing that > > 9) enable the service - Ansible's service module is generic and supports > init/OpenRC > > 10) I don't see a step 10 > This was either a mistake or the installation steps got updated. I'll adjust that. > IOW I think there should be as little manual intervention as possible and in > this case I don't think any manual steps are needed by the user. > Agreed. I was not super happy with the current state of this playbook wrt FreeBSD, but I decided to pick other battles to fight. Also, newer gitlab-runner versions *may* be using the updated service management lib, which may reduce/remove the need for custom handling. Anyway, I'll look at what can be improved here, considering the cost. Thanks! - Cleber.
On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote: > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote: > > + > > + - name: Checks the availability of official gitlab-runner builds in the archive > > + uri: > > + url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-linux-386 > > + method: HEAD > > + status_code: > > + - 200 > > + - 403 > > Why is 403 an acceptable success status code? > I missed this question in my last reply, sorry. s3 will throw a 403 if the URI doesn't exist... and we don't want to fail the playbook because of that, given that we'll attempt the fallback repo defined in vars.yml{,.template}. > > + register: gitlab_runner_available_archive > > + > > + - name: Update base url > > + set_fact: > > + gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner- > > + when: gitlab_runner_available_archive.status == 200 > > + - debug: > > + msg: Base gitlab-runner url is {{ gitlab_runner_base_url }} > > + > > + - name: Set OS name (FreeBSD) > > + set_fact: > > + gitlab_runner_os: freebsd > > + when: "ansible_facts['system'] == 'FreeBSD'" > > + > > + - name: Create a group for the gitlab-runner service > > + group: > > + name: gitlab-runner > > + > > + - name: Create a user for the gitlab-runner service > > + user: > > + user: gitlab-runner > > + group: gitlab-runner > > + comment: GitLab Runner > > + home: /home/gitlab-runner > > + shell: /bin/bash > > + > > + - name: Remove the .bash_logout file when on Ubuntu systems > > + file: > > + path: /home/gitlab-runner/.bash_logout > > + state: absent > > + when: "ansible_facts['distribution'] == 'Ubuntu'" > > + > > + - name: Downloads the matching gitlab-runner > > + get_url: > > + dest: /usr/local/bin/gitlab-runner > > + url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}" > > + owner: gitlab-runner > > + group: gitlab-runner > > + mode: u=rwx,g=rwx,o=rx > > + Here, the actual download is performed, and no 40x is considered OK. I hope that explains it. Cheers, - Cleber.
On Mon, Oct 19, 2020 at 04:41:38PM -0400, Cleber Rosa wrote: > On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote: > > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote: > > > To have the jobs dispatched to custom runners, gitlab-runner must > > > be installed, active as a service and properly configured. The > > > variables file and playbook introduced here should help with those > > > steps. > > > > > > The playbook introduced here covers a number of different Linux > > > distributions and FreeBSD, and are intended to provide a reproducible > > > environment. > > > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> > > > --- > > > > In general, there's been put quite some effort into the playbooks - sorry I'm > > late to the game - is there a plan to introduce QEMU as a project to lcitool? > > I think it's becoming quite clear that having so much duplication (in > the dockerfiles, tests/vm, this playbook, etc) is costly and error > prone. I don't know if anyone has invested time in a PoC to > consolidate those (with lcitool), but I can certainly see the upside > to that. BTW, are you volunteering (wink wink)? :) I don't think I was trying to :), but sure, I can dedicate some time to it. I'll need a bit of guidance in the QEMU world though for sure. > > > We've taken care of most of the bits in the playbooks that are being introduced > > and for the remaining ones I think it would be that big of an overhaul to do > > the adjustments. One major re-factor though would IMO be to break the > > dependency lcitool has on the machine naming, kind of restricting it to a > > limited set of hosts and corresponding names (e.g. libvirt-fedora-32) which > > makes it inconvenient to prepare physical hosts. > > > > Right... I wasn't aware of that depedency. And, this may be a nice > project to make sure that lcitool doesn't have any other libvirt > specificities. Yeah, that one will be a major change. Thinking about this some more, while having a QEMU project inside lcitool sounds definitely useful, it would probably be a wasted effort wrt task priority right now, because without expanding lcitool's range of applicable hosts I don't see how QEMU would utilize the tool without using quirks. > > > More comments inline... > > > > > docs/devel/ci.rst | 63 ++++++++++++++++++++++++++ > > > scripts/ci/setup/.gitignore | 1 + > > > scripts/ci/setup/gitlab-runner.yml | 72 ++++++++++++++++++++++++++++++ > > > scripts/ci/setup/vars.yml.template | 13 ++++++ > > > 4 files changed, 149 insertions(+) > > > create mode 100644 scripts/ci/setup/.gitignore > > > create mode 100644 scripts/ci/setup/gitlab-runner.yml > > > create mode 100644 scripts/ci/setup/vars.yml.template > > > > > > diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst > > > index 208b5e399b..a234a5e24c 100644 > > > --- a/docs/devel/ci.rst > > > +++ b/docs/devel/ci.rst > > > @@ -84,3 +84,66 @@ To run the playbook, execute:: > > > > > > cd scripts/ci/setup > > > ansible-playbook -i inventory build-environment.yml > > > + > > > +gitlab-runner setup and registration > > > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > + > > > +The gitlab-runner agent needs to be installed on each machine that > > > +will run jobs. The association between a machine and a GitLab project > > > +happens with a registration token. To find the registration token for > > > +your repository/project, navigate on GitLab's web UI to: > > > + > > > + * Settings (the gears like icon), then > > > + * CI/CD, then > > > + * Runners, and click on the "Expand" button, then > > > + * Under "Set up a specific Runner manually", look for the value under > > > + "Use the following registration token during setup" > > > + > > > +Copy the ``scripts/ci/setup/vars.yml.template`` file to > > > +``scripts/ci/setup/vars.yml``. Then, set the > > > +``gitlab_runner_registration_token`` variable to the value obtained > > > +earlier. > > > + > > > +.. note:: gitlab-runner is not available from the standard location > > > + for all OS and architectures combinations. For some systems, > > > + a custom build may be necessary. Some builds are avaiable > > > + at https://cleber.fedorapeople.org/gitlab-runner/ and this > > > + URI may be used as a value on ``vars.yml`` > > > > Yes, this can be suboptimal...Would it make sense to fall back to build the > > binary of a given version from git as a fallback during this playbook if the > > necessary arch version isn't provided the official way? Just an idea, I'd like > > to avoid the need for you to become the maintainer of the binaries and keep up > > with the releases. > > > > Well, building them during the playbook would be a lot more > complex... You can have your own repo with your own builds, and just > tweak your vars.yml. I didn't want to suggest this as a replacement to vars.yml. What if people wanting to run the playbook don't have their own repo and you (I mean you personally) are not providing a fresh build of gitlab-runner for the given architecture? The automation machinery would surely be more complex, but at the same time, I can image a dedicated playbook that would handle gitlab-runner building and installation of the binary to the correct place, you'd then simply include that playbook if vars.yml don't specify an alternative repo and download from AWS failed. Another thing to consider - you currently host the missing arch builds under your personal namespace on fedorapeople, what if you decide one day you're not interested in being involved in QEMU CI that much and suddenly you stop building the binaries, someone has to take over. On the other hand, if you automate this whole process, having exposed the builds would definitely not slow down machine provisions, but would also not cause them to fail if nobody exposes them. > > > > + > > > +To run the playbook, execute:: > > > + > > > + cd scripts/ci/setup > > > + ansible-playbook -i inventory gitlab-runner.yml > > > + > > > +.. note:: there are currently limitations to gitlab-runner itself when > > > + setting up a service under FreeBSD systems. You will need to > > > + perform steps 4 to 10 manually, as described at > > > > Which one of them is considered an automation problem? In lcitool we made > > gitlab-runner completely automated on all distros, including FreeBSD: > > > > It looks like lcitool went the more practical route. I was hoping to > not have to treat gitlab-runner in such a special way in any > "supported" OS. What I mean is, I'd rather write the code within > gitlab-runner (or reespective libraries). Of course, I did not get to > it, so that's why I just documented the steps here. > > I'll take a look at lcitool's playbook to see if I can easily > canibalize some of that. But, at this time, we don't runners for > FreeBSD anyway, so I guess this is not *that* urgent. Okay, I was simply curious. If it's not something you don't need at the moment, then let's not focus on it. > > > 4) log file permissions - you're creating the user, you can as well create the > > file with correct permissions > > > > 5) ensure /usr/local/etc/rc.d exists - once you execute build-environment.yml > > it will pull a bunch of dependencies which ensure the dir exists > > > > 6) gitlab_runner service script should IMO provided by this automation as a > > template and install to the correct location > > > > 7) ensure the service script is executable - template module can do that > > > > 8) register the runner - you're doing that > > > > 9) enable the service - Ansible's service module is generic and supports > > init/OpenRC > > > > 10) I don't see a step 10 > > > > This was either a mistake or the installation steps got updated. I'll > adjust that. > > > IOW I think there should be as little manual intervention as possible and in > > this case I don't think any manual steps are needed by the user. > > > > Agreed. I was not super happy with the current state of this playbook > wrt FreeBSD, but I decided to pick other battles to fight. Also, > newer gitlab-runner versions *may* be using the updated service > management lib, which may reduce/remove the need for custom handling. What do you mean by "service management lib"? Regards, Erik
On Mon, Oct 19, 2020 at 04:54:50PM -0400, Cleber Rosa wrote: > On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote: > > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote: > > > + > > > + - name: Checks the availability of official gitlab-runner builds in the archive > > > + uri: > > > + url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-linux-386 > > > + method: HEAD > > > + status_code: > > > + - 200 > > > + - 403 > > > > Why is 403 an acceptable success status code? > > > > I missed this question in my last reply, sorry. > > s3 will throw a 403 if the URI doesn't exist... and we don't want to fail > the playbook because of that, given that we'll attempt the fallback > repo defined in vars.yml{,.template}. Right, forgot about the repo in vars.yml fallback, thanks for explaining. Regards, Erik
On Tue, Oct 20, 2020 at 08:58:39AM +0200, Erik Skultety wrote: > On Mon, Oct 19, 2020 at 04:41:38PM -0400, Cleber Rosa wrote: > > On Mon, Oct 19, 2020 at 12:26:10PM +0200, Erik Skultety wrote: > > > On Sun, Oct 18, 2020 at 09:50:02PM -0400, Cleber Rosa wrote: > > > > To have the jobs dispatched to custom runners, gitlab-runner must > > > > be installed, active as a service and properly configured. The > > > > variables file and playbook introduced here should help with those > > > > steps. > > > > > > > > The playbook introduced here covers a number of different Linux > > > > distributions and FreeBSD, and are intended to provide a reproducible > > > > environment. > > > > > > > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > > > > Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> > > > > --- > > > > > > In general, there's been put quite some effort into the playbooks - sorry I'm > > > late to the game - is there a plan to introduce QEMU as a project to lcitool? > > > > I think it's becoming quite clear that having so much duplication (in > > the dockerfiles, tests/vm, this playbook, etc) is costly and error > > prone. I don't know if anyone has invested time in a PoC to > > consolidate those (with lcitool), but I can certainly see the upside > > to that. BTW, are you volunteering (wink wink)? :) > > I don't think I was trying to :), but sure, I can dedicate some time to it. > I'll need a bit of guidance in the QEMU world though for sure. I think the obvious and easy place is start using lcitool is for the tests/docker/dockerfiles/*. All that's required is to add mappings to lcitool for the various deps that QEMU has which libvirt does not already have. Then we should be able to start auto-generating the dockerfiles without too much difficulty. This will be a significant step forward because it will help us keep te package lists in sync across all the dockerfiles which is a major fail in QEMU right now. Dealing with tests/vm replacement or these ansible recipes is likely to be a significantly more challenging proposition. Perhaps we can again start by just automating creation of the package lists that the tests/vm and ansibile recipes need, as again those are all inconsistent. Regards, Daniel
On Tue, 2020-10-20 at 09:29 +0100, Daniel P. Berrangé wrote: > I think the obvious and easy place is start using lcitool is for the > tests/docker/dockerfiles/*. All that's required is to add mappings > to lcitool for the various deps that QEMU has which libvirt does not > already have. Then we should be able to start auto-generating the > dockerfiles without too much difficulty. This will be a significant > step forward because it will help us keep te package lists in sync > across all the dockerfiles which is a major fail in QEMU right now. I took a quick look at the contents of those dockerfiles and while I agree that it's the obvious place to start, I'm not sure I would say that generating them using lcitool is going to be easy :) Basically there seem to be a number of assumptions both in lcitool and in the QEMU dockerfiles, and getting the two to agree on those will require quite some work from what I can tell. But again, I agree it's something that should be worked on. > Dealing with tests/vm replacement or these ansible recipes is likely > to be a significantly more challenging proposition. Perhaps we can > again start by just automating creation of the package lists that > the tests/vm and ansibile recipes need, as again those are all > inconsistent. Perhaps 'lcitool variables' could be of use here.
diff --git a/docs/devel/ci.rst b/docs/devel/ci.rst index 208b5e399b..a234a5e24c 100644 --- a/docs/devel/ci.rst +++ b/docs/devel/ci.rst @@ -84,3 +84,66 @@ To run the playbook, execute:: cd scripts/ci/setup ansible-playbook -i inventory build-environment.yml + +gitlab-runner setup and registration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The gitlab-runner agent needs to be installed on each machine that +will run jobs. The association between a machine and a GitLab project +happens with a registration token. To find the registration token for +your repository/project, navigate on GitLab's web UI to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * Under "Set up a specific Runner manually", look for the value under + "Use the following registration token during setup" + +Copy the ``scripts/ci/setup/vars.yml.template`` file to +``scripts/ci/setup/vars.yml``. Then, set the +``gitlab_runner_registration_token`` variable to the value obtained +earlier. + +.. note:: gitlab-runner is not available from the standard location + for all OS and architectures combinations. For some systems, + a custom build may be necessary. Some builds are avaiable + at https://cleber.fedorapeople.org/gitlab-runner/ and this + URI may be used as a value on ``vars.yml`` + +To run the playbook, execute:: + + cd scripts/ci/setup + ansible-playbook -i inventory gitlab-runner.yml + +.. note:: there are currently limitations to gitlab-runner itself when + setting up a service under FreeBSD systems. You will need to + perform steps 4 to 10 manually, as described at + https://docs.gitlab.com/runner/install/freebsd.html + +Following the registration, it's necessary to configure the runner tags, +and optionally other configurations on the GitLab UI. Navigate to: + + * Settings (the gears like icon), then + * CI/CD, then + * Runners, and click on the "Expand" button, then + * "Runners activated for this project", then + * Click on the "Edit" icon (next to the "Lock" Icon) + +Under tags, add values matching the jobs a runner should run. For a +FreeBSD 12.1 x86_64 system, the tags should be set as:: + + freebsd12.1,x86_64 + +Because the job definition at ``.gitlab-ci.d/custom-runners.yml`` +would contain:: + + freebsd-12.1-x86_64-all: + tags: + - freebsd_12.1 + - x86_64 + +It's also recommended to: + + * increase the "Maximum job timeout" to something like ``2h`` + * uncheck the "Run untagged jobs" check box + * give it a better Description diff --git a/scripts/ci/setup/.gitignore b/scripts/ci/setup/.gitignore new file mode 100644 index 0000000000..f112d05dd0 --- /dev/null +++ b/scripts/ci/setup/.gitignore @@ -0,0 +1 @@ +vars.yml \ No newline at end of file diff --git a/scripts/ci/setup/gitlab-runner.yml b/scripts/ci/setup/gitlab-runner.yml new file mode 100644 index 0000000000..c2f52dad10 --- /dev/null +++ b/scripts/ci/setup/gitlab-runner.yml @@ -0,0 +1,72 @@ +--- +- name: Installation of gitlab-runner + hosts: all + vars_files: + - vars.yml + tasks: + - debug: + msg: 'Checking for a valid GitLab registration token' + failed_when: "gitlab_runner_registration_token == 'PLEASE_PROVIDE_A_VALID_TOKEN'" + + - name: Checks the availability of official gitlab-runner builds in the archive + uri: + url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner-linux-386 + method: HEAD + status_code: + - 200 + - 403 + register: gitlab_runner_available_archive + + - name: Update base url + set_fact: + gitlab_runner_base_url: https://s3.amazonaws.com/gitlab-runner-downloads/v{{ gitlab_runner_version }}/binaries/gitlab-runner- + when: gitlab_runner_available_archive.status == 200 + - debug: + msg: Base gitlab-runner url is {{ gitlab_runner_base_url }} + + - name: Set OS name (FreeBSD) + set_fact: + gitlab_runner_os: freebsd + when: "ansible_facts['system'] == 'FreeBSD'" + + - name: Create a group for the gitlab-runner service + group: + name: gitlab-runner + + - name: Create a user for the gitlab-runner service + user: + user: gitlab-runner + group: gitlab-runner + comment: GitLab Runner + home: /home/gitlab-runner + shell: /bin/bash + + - name: Remove the .bash_logout file when on Ubuntu systems + file: + path: /home/gitlab-runner/.bash_logout + state: absent + when: "ansible_facts['distribution'] == 'Ubuntu'" + + - name: Downloads the matching gitlab-runner + get_url: + dest: /usr/local/bin/gitlab-runner + url: "{{ gitlab_runner_base_url }}{{ gitlab_runner_os }}-{{ gitlab_runner_arch }}" + owner: gitlab-runner + group: gitlab-runner + mode: u=rwx,g=rwx,o=rx + + - name: Register the gitlab-runner + command: "/usr/local/bin/gitlab-runner register --non-interactive --url {{ gitlab_runner_server_url }} --registration-token {{ gitlab_runner_registration_token }} --executor shell --description '{{ ansible_facts[\"distribution\"] }} {{ ansible_facts[\"distribution_version\"] }} {{ ansible_facts[\"architecture\"] }} ({{ ansible_facts[\"os_family\"] }})'" + + - name: Install the gitlab-runner service using its own functionality + command: /usr/local/bin/gitlab-runner install --user gitlab-runner --working-directory /home/gitlab-runner + register: gitlab_runner_install_service_result + failed_when: "gitlab_runner_install_service_result.rc != 0 and \"already exists\" not in gitlab_runner_install_service_result.stderr" + when: "ansible_facts['os_family'] != 'FreeBSD'" + + - name: Enable the gitlab-runner service + service: + name: gitlab-runner + state: started + enabled: yes + when: "ansible_facts['os_family'] != 'FreeBSD'" diff --git a/scripts/ci/setup/vars.yml.template b/scripts/ci/setup/vars.yml.template new file mode 100644 index 0000000000..621435d030 --- /dev/null +++ b/scripts/ci/setup/vars.yml.template @@ -0,0 +1,13 @@ +# The version of the gitlab-runner to use +gitlab_runner_version: 13.1.1 +# The base location of gitlab-runner binaries, this will be suffixed by $OS-$ARCH +gitlab_runner_base_url: https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner- +# The URL of the gitlab server to use, usually https://gitlab.com unless you're +# using a private GitLab instance +gitlab_runner_server_url: https://gitlab.com +# Defaults to linux, checks can be used to change this +gitlab_runner_os: linux +# Defaults to amd64 (x86_64), checks can be used to change this +gitlab_runner_arch: amd64 +# A unique token made available by GitLab to your project for registering runners +gitlab_runner_registration_token: PLEASE_PROVIDE_A_VALID_TOKEN