Page MenuHomePhabricator
Create Task
Maniphest T295481

Setup GitLab Runner in trusted environment
Closed, ResolvedPublic
Actions

Description

This task is for tracking the setup of GitLab Runners in a trusted environment.

In T286958 we discussed the long term requirements for GitLab Runners. One class of Runners should run in production environments (eqiad, codfw) and execute jobs which handle sensitive credentials and produce artifacts running in production. See also https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner#Specific_GitLab_Runners.

I would like to reuse the existing puppet code for the Shared Runners in WMCS. I think we could start with VMs on Ganeti and later order dedicated machines and/or migrate to some Kubernetes platform.
The Runners must not be used by arbitrary jobs but only by certain projects and branches. So this runners will be setup as Specific Runners, probably executing only jobs for protected branches.

Roughly the needed steps are:

  • setup dedicated ganeti VMs in codfw and eqiad (gitlab-runner1001 and gitlab-runner2001)
  • adjust puppet code and install role on new ganeti VMs
  • register new runners as specific runners and run test job
  • add monitoring (Prometheus metrics and some Grafana dashboards)
  • validate GitLab application permission concept ⌛
    • runners should execute protected branches only
    • runners should be available only to allowed projects
    • non-privileged project pipelines (feature branches) can not escalate privileges by altering gitlab-ci.yml file
    • make sure trusted runners must use protected branches
    • document permission and security concept
  • validate host security concept
    • runner jobs shouldn't be able to connect to other WMF service or hosts (except explicitly permitted, like docker-registry, apt repos, chart museum)
      • harden firewall rules so no other services can be reached from GitLab Runner Docker containers
    • runners shouldn't be able to execute code with root privileges/escalate to root privileges
      • prevent privileged containers (privileged = false)
      • run gitlab-runner as non-root user
      • evaluate if containers can be executed as non-root See T320411
      • evaluate if certain capabilities can be dropped See T320411
  • create automation for managing and requesting access to Trusted Runners /repos/releng/gitlab-trusted-runner

Details

SubjectRepoBranchLines +/-
gitlab_runner: add option to drop Docker capabilitiesoperations/puppetproduction+11 -1
gitlab_runner: restrict all internal traffic, not only TCPoperations/puppetproduction+1 -1
gitlab_runner: add webproxy to allowed_servicesoperations/puppetproduction+12 -0
gitlab_runner: allow Trusted Runners to access wikimedia docker-registryoperations/puppetproduction+6 -0
sre.gitlab.reboot-runner: fix pre_scripts calloperations/cookbooksmaster+2 -6
sre.gitlab.reboot-runner: add cookbook to restart gitlab-runnersoperations/cookbooksmaster+49 -0
gitlab_runner: move metrics listen_address to global sectionoperations/puppetproduction+1 -0
gitlab_runner: execute gitlab-runner as non-rootoperations/puppetproduction+24 -14
gitlab_runner: override ExecStart in service unit for non-rootoperations/puppetproduction+6 -2
gitlab_runner: override User of default service unit fileoperations/puppetproduction+3 -17
gitlab_runner: overwrite default service unit fileoperations/puppetproduction+1 -0
gitlab_runner: remove duplicate ferm rule for AAAAoperations/puppetproduction+1 -1
gitlab_runner: restrict docker traffic with additional ferm rulesoperations/puppetproduction+81 -26
gitlab_runner: add missing hiera entry for WMCSoperations/puppetproduction+2 -0
gitlab_runner: add missing hiera entry for WMCSoperations/puppetproduction+2 -0
gitlab_runner: cleanup service unit fileoperations/puppetproduction+2 -3
gitlab_runner: add dedicated service unit fileoperations/puppetproduction+27 -13
systemd::sysuser: create option to add additional groups to useroperations/puppetproduction+28 -11
gitlab_runner: add gitlab-runner to docker group, change folder permissionsoperations/puppetproduction+20 -4
gitlab_runner: fix service definitionoperations/puppetproduction+7 -1
install_server: update MAC address of gitlab-runner1001operations/puppetproduction+1 -1
P:prometheus::ops: add prometheus job and ferm rules for gitlab_runner metricsoperations/puppetproduction+33 -1
gitlab_runner: fix missing parameters in registration commandoperations/puppetproduction+2 -0
gitlab_runner: fix missing url in registration commandoperations/puppetproduction+1 -0
gitlab_runner: use config template for registering new runnersoperations/puppetproduction+47 -51
gitlab_runner: disable disk check for docker volumesoperations/puppetproduction+1 -0
gitlab_runner: rollback usage of dedicated module for runner configoperations/puppetproduction+20 -11
gitlab_runner: fix config.toml syntaxoperations/puppetproduction+8 -8
gitlab_runner: add --config parameter to register commandoperations/puppetproduction+1 -0
gitlab_runner: quote parameters in gitlab-runner configoperations/puppetproduction+4 -4
gitlab_runner: enable protected runnersoperations/puppetproduction+1 -1
gitlab_runner: create module for runner config and enable metricsoperations/puppetproduction+69 -20
profile::gitlab-runner add registration_token for protected GitLab Runnerslabs/privatemaster+1 -0
site: use gitlab_runner role on gitlab-runner2001operations/puppetproduction+1 -1
profile::gitlab-runner add hieradata for protected GitLab Runnersoperations/puppetproduction+19 -0
profile::gitlab_runner rename hiera file to gitlab_runneroperations/puppetproduction+0 -0
profile::gitlab_runner unregister runner gitlab-runner1001operations/puppetproduction+1 -1
site and install_server: add gitlab-runner2001operations/puppetproduction+10 -0
site: use gitlab_runner role on gitlab-runner1001operations/puppetproduction+1 -1
site and install_server: add gitlab-runner1001operations/puppetproduction+13 -1
gitlab: create role for prod gitlab-runners, adjust cumin aliasoperations/puppetproduction+15 -3
Show related patches Customize query in gerrit

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
sbassett mentioned this in T294050: Enable runners for projects under gitlab.wikimedia.org security group.Dec 8 2021, 6:14 PM
gerritbot added a comment.Dec 15 2021, 4:20 PM

Change 747539 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: use config template for registering new runners

https://gerrit.wikimedia.org/r/747539

gerritbot added a comment.Dec 17 2021, 12:41 PM

Change 748114 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: disable disk check for docker volumes

https://gerrit.wikimedia.org/r/748114

gerritbot added a comment.Dec 20 2021, 3:08 PM

Change 748114 merged by Jelto:

[operations/puppet@production] gitlab_runner: disable disk check for docker volumes

https://gerrit.wikimedia.org/r/748114

Jelto added a comment.Dec 21 2021, 1:51 PM

I've done some more testing around managing the GitLab Runner configuration file config.toml using puppet. The initial problem was that the gitlab-runner register command alters the config file during runtime (which conflicts with puppets workflow).

I see multiple options we could try to progress here:

  1. let puppet only create the config file but not alter it later (so that changes won't get overwritten by puppet)
  2. split the config and try to include some sub config somehow doesn't supported by toml specification
  3. use a config template managed by puppet and use the --template-config parameter during registering (implemented in /operations/puppet/+/747539).
  4. Don't register the runners with gitlab-runner register but with runner api and have a static, puppet managed config file
  5. Invent some more complex logic which merges multiple files during a puppet run

I implemented option 3. because for me it's the best compromise. Option 1. sounds misleading, because puppet code changes wouldn't have an effect on actual production infrastructure immediately. Option 5. sounds fragile and complex to implement. So option 3. explicitly creates a template, which is used during registration. It should be clear that the template has an effect on the registration workflow only and not during runtime. But I also think this is a major downside of this option. This would mean we have to un-register the runners (ensure absent) and re-register them (ensure present) for config changes to take effect.

I would favor option 4. and have all config files and tokens as static artifacts inside of puppet. But registering multiple runners and adding all of the tokens to private puppet would mean quite some work and we have to somehow automate this workflow.

I would like to discuss this issue with folks either here or in our weekly session.

Dzahn added a comment.Dec 22 2021, 12:07 PM

I agree that option 1 sounds misleading and not great and option 5 sounds overly complex / brittle. Fully on the same page with you here.

My immediate thought was that 4 is the best but that I probably don't understand yet why that isn't so easy. Maybe let's talk some more about the process to get tokens into private puppet in an efficient way.

gerritbot added a comment.Jan 4 2022, 3:03 PM

Change 751452 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] P:prometheus::ops: add prometheus job and ferm rules for gitlab_runner metrics

https://gerrit.wikimedia.org/r/751452

gerritbot added a comment.Jan 7 2022, 10:40 AM

Change 747539 merged by Jelto:

[operations/puppet@production] gitlab_runner: use config template for registering new runners

https://gerrit.wikimedia.org/r/747539

gerritbot added a comment.Jan 7 2022, 10:50 AM

Change 752137 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: fix missing url in registration command

https://gerrit.wikimedia.org/r/752137

gerritbot added a comment.Jan 7 2022, 10:56 AM

Change 752137 merged by Jelto:

[operations/puppet@production] gitlab_runner: fix missing url in registration command

https://gerrit.wikimedia.org/r/752137

gerritbot added a comment.Jan 7 2022, 11:03 AM

Change 752138 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: fix missing parameters in registration command

https://gerrit.wikimedia.org/r/752138

gerritbot added a comment.Jan 7 2022, 11:07 AM

Change 752138 merged by Jelto:

[operations/puppet@production] gitlab_runner: fix missing parameters in registration command

https://gerrit.wikimedia.org/r/752138

gerritbot added a comment.Jan 7 2022, 12:19 PM

Change 751452 merged by Jelto:

[operations/puppet@production] P:prometheus::ops: add prometheus job and ferm rules for gitlab_runner metrics

https://gerrit.wikimedia.org/r/751452

Jelto added a comment.Jan 10 2022, 9:43 AM

GitLab Runner in eqiad and codfw export Prometheus metrics now.
I created a GitLab CI overview dashboard in Grafana. This dashboard also links to a new GitLab Runner detail dashboard.

I couldn't find a good GitLab Runner/CI dashboard online which is not outdated or doesn't require an additional exporter.

Jelto updated the task description. (Show Details)Jan 19 2022, 2:56 PM
Jelto added a comment.Jan 24 2022, 3:36 PM

I've done some more evaluation and testing around the trusted GitLab Runners. For that I created a project under the /repos group to have access to the Shared Group runners in WMCS. I also allowed this project explicitly to use the Trusted Runners. I was able to execute un-reviewed changes on the Shared Runners in WMCS and reviewed changes on the Trusted Runners. I tried to document the current concept in GitLab/Gitlab_Runner#Access_and_permission_model (and also drew a overview diagram).

I also done some testing for special edge cases, like change the gitlab-ci.yml in a feature branch, forking a project or remove certain rules/tags from CI jobs. For now it seems a proper separation of unreviewed and reviewed code changes/jobs is possible.

However there are two big implications when allowing projects access to the Trusted Runners:

  • Main branch has to be protected
  • People with maintainer permissions for that project can execute code to some extend on WMF infrastructure. So access to project maintainer permission has to be controlled

For the first topic I also thought about some kind of check script, which automatically removes the trusted Runners from projects with unprotected main branches. There could also be a script to allow projects to the Trusted Runners, which could first protect the branch automatically.

I'm going to do some more testing and also try to get Security Team on board for a review once we came up with a finished concept around the Trusted Runners.

ops-monitoring-bot added a comment.Jan 25 2022, 1:30 PM

Icinga downtime set by jelto@cumin1001 for 6:00:00 1 host(s) and their services with reason: move gitlab-runner1001 to new ganeti row

gitlab-runner1001.eqiad.wmnet
ops-monitoring-bot added a comment.Jan 25 2022, 1:46 PM

cookbooks.sre.hosts.decommission executed by jelto@cumin1001 for hosts: gitlab-runner1001.eqiad.wmnet

  • gitlab-runner1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
gerritbot added a comment.Jan 26 2022, 7:55 AM

Change 757378 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] install_server: update MAC address of gitlab-runner1001

https://gerrit.wikimedia.org/r/757378

gerritbot added a comment.Jan 26 2022, 8:08 AM

Change 757378 merged by Jelto:

[operations/puppet@production] install_server: update MAC address of gitlab-runner1001

https://gerrit.wikimedia.org/r/757378

Jelto updated the task description. (Show Details)Feb 1 2022, 4:58 PM
Jelto updated the task description. (Show Details)Feb 2 2022, 12:11 PM
gerritbot added a comment.Feb 2 2022, 3:03 PM

Change 759254 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: execute gitlab-runner as non-root

https://gerrit.wikimedia.org/r/759254

gerritbot added a comment.Mar 4 2022, 1:16 PM

Change 759254 merged by Jelto:

[operations/puppet@production] gitlab_runner: execute gitlab-runner as non-root

https://gerrit.wikimedia.org/r/759254

gerritbot added a comment.Mar 4 2022, 1:27 PM

Change 768040 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: fix service definition

https://gerrit.wikimedia.org/r/768040

gerritbot added a comment.Mar 4 2022, 1:42 PM

Change 768040 merged by Jelto:

[operations/puppet@production] gitlab_runner: fix service definition

https://gerrit.wikimedia.org/r/768040

gerritbot added a comment.Mar 7 2022, 11:18 AM

Change 768683 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: add gitlab-runner to docker group, change folder permissions

https://gerrit.wikimedia.org/r/768683

gerritbot added a comment.Mar 7 2022, 3:39 PM

Change 768743 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] isystemd::sysuser: create option to add additional groups to user

https://gerrit.wikimedia.org/r/768743

gerritbot added a comment.Mar 8 2022, 1:22 PM

Change 768743 merged by Jelto:

[operations/puppet@production] systemd::sysuser: create option to add additional groups to user

https://gerrit.wikimedia.org/r/768743

gerritbot added a comment.Mar 8 2022, 1:22 PM

Change 768683 merged by Jelto:

[operations/puppet@production] gitlab_runner: add gitlab-runner to docker group, change folder permissions

https://gerrit.wikimedia.org/r/768683

gerritbot added a comment.Mar 8 2022, 3:45 PM

Change 769065 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: add dedicated service unit file

https://gerrit.wikimedia.org/r/769065

gerritbot added a comment.Mar 9 2022, 8:13 AM

Change 769065 merged by Jelto:

[operations/puppet@production] gitlab_runner: add dedicated service unit file

https://gerrit.wikimedia.org/r/769065

Jelto updated the task description. (Show Details)Mar 9 2022, 10:58 AM
Jelto updated the task description. (Show Details)Mar 9 2022, 2:19 PM
gerritbot added a comment.Mar 10 2022, 5:15 PM

Change 769737 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: cleanup service unit file

https://gerrit.wikimedia.org/r/769737

gerritbot added a comment.Mar 11 2022, 8:51 AM

Change 769737 merged by Jelto:

[operations/puppet@production] gitlab_runner: cleanup service unit file

https://gerrit.wikimedia.org/r/769737

gerritbot added a comment.Mar 11 2022, 1:19 PM

Change 769968 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: restrict docker traffic with additional ferm rules

https://gerrit.wikimedia.org/r/769968

gerritbot added a comment.Mar 15 2022, 10:11 AM

Change 769968 merged by Jelto:

[operations/puppet@production] gitlab_runner: restrict docker traffic with additional ferm rules

https://gerrit.wikimedia.org/r/769968

gerritbot added a comment.Mar 15 2022, 11:05 AM

Change 770891 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: add missing hiera entry for WMCS

https://gerrit.wikimedia.org/r/770891

gerritbot added a comment.Mar 15 2022, 11:10 AM

Change 770891 merged by Jelto:

[operations/puppet@production] gitlab_runner: add missing hiera entry for WMCS

https://gerrit.wikimedia.org/r/770891

gerritbot added a comment.Mar 15 2022, 11:31 AM

Change 770893 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: add missing hiera entry for WMCS

https://gerrit.wikimedia.org/r/770893

gerritbot added a comment.Mar 15 2022, 11:36 AM

Change 770893 merged by Jelto:

[operations/puppet@production] gitlab_runner: add missing hiera entry for WMCS

https://gerrit.wikimedia.org/r/770893

Jelto updated the task description. (Show Details)Mar 15 2022, 12:58 PM
gerritbot added a comment.Mar 17 2022, 4:04 PM

Change 771633 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: remove duplicate ferm rule for AAAA

https://gerrit.wikimedia.org/r/771633

gerritbot added a comment.Mar 18 2022, 9:41 AM

Change 771633 merged by Jelto:

[operations/puppet@production] gitlab_runner: remove duplicate ferm rule for AAAA

https://gerrit.wikimedia.org/r/771633

Jelto mentioned this in T304491: Standardize Debian package builds on GitLab CI.Mar 23 2022, 10:16 AM
Jelto updated the task description. (Show Details)
Jelto added a comment.Mar 23 2022, 10:25 AM

I mirrored wmf-sre-laptop to GitLab and created a very basic proof-of-concept CI to build the Debian package on Trusted Runners. The current implementation has limitations and is not complete. I created T304491 to further discuss the whole topic of Debian package builds on GitLab CI, as this is a bit out of scope for this task.

I'll proceed with polishing the documentation about the Trusted Runner setup and try to wrap my head around Docker image builds on Trusted Runners.

Jelto mentioned this in T304514: Security Readiness Reviews of Trusted GitLab Runners.Mar 23 2022, 2:26 PM
Jelto updated the task description. (Show Details)Mar 24 2022, 3:34 PM
gerritbot added a comment.Mar 25 2022, 9:31 AM

Change 773746 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: add option to drop Docker capabilities

https://gerrit.wikimedia.org/r/773746

gerritbot added a comment.Mar 31 2022, 9:21 AM

Change 775808 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: overwrite default service unit file

https://gerrit.wikimedia.org/r/775808

gerritbot added a comment.Mar 31 2022, 9:30 AM

Change 775808 merged by Jelto:

[operations/puppet@production] gitlab_runner: overwrite default service unit file

https://gerrit.wikimedia.org/r/775808

gerritbot added a comment.Mar 31 2022, 9:47 AM

Change 775815 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: override User of default service unit file

https://gerrit.wikimedia.org/r/775815

gerritbot added a comment.Mar 31 2022, 10:06 AM

Change 775815 merged by Jelto:

[operations/puppet@production] gitlab_runner: override User of default service unit file

https://gerrit.wikimedia.org/r/775815

gerritbot added a comment.Mar 31 2022, 10:20 AM

Change 775821 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: override ExecStart of default service unit file

https://gerrit.wikimedia.org/r/775821

gerritbot added a comment.Apr 5 2022, 2:58 PM

Change 775821 merged by Jelto:

[operations/puppet@production] gitlab_runner: override ExecStart in service unit for non-root

https://gerrit.wikimedia.org/r/775821

Jelto mentioned this in T297659: upgrade gitlab-runners to bullseye.Apr 19 2022, 8:19 AM
Jelto updated the task description. (Show Details)Apr 19 2022, 8:31 AM
Jelto mentioned this in T307541: Establish image trust system for GitLab/Blubber.May 4 2022, 1:32 PM
gerritbot added a comment.May 9 2022, 1:18 PM

Change 790369 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: move metrics listen_address to global section

https://gerrit.wikimedia.org/r/790369

gerritbot added a comment.May 9 2022, 5:35 PM

Change 790369 merged by Dzahn:

[operations/puppet@production] gitlab_runner: move metrics listen_address to global section

https://gerrit.wikimedia.org/r/790369

Jelto added a comment.May 17 2022, 12:04 PM
This comment was removed by Jelto.
Jelto added a comment.EditedMay 17 2022, 12:14 PM

Trusted Runner automation and access request

I added a first version on how to get and manage access to Trusted Runners. I created /repos/releng/gitlab-trusted-runner.

This repo is used as a central project were Trusted Runners will get registered (they were registered on a private project test project of mine before, I'll change the registration_token in private puppet soon). Furthermore the repo also contains a configuration file and a script to automate access to the Trusted Runners.

projects.json contains a list of authorized projects:

{
  "182": {
    "name": "gitlab-runner-test",
    "reason": "To verify and test Trusted GitLab Runners"
  },
  "339": {
    "name": "gitlab-trusted-runner",
    "reason": "Root/registration project for Trusted Runners"
  }
}

If the script is executed, you can see a (color!) diff with the new configuration:

add-project.py diff

gitlab-runner2001.codfw.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

gitlab-runner1001.eqiad.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

gitlab-runner2002.codfw.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

gitlab-runner2003.codfw.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

gitlab-runner2004.codfw.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

gitlab-runner1002.eqiad.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

gitlab-runner1003.eqiad.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

gitlab-runner1004.eqiad.wmnet:
+project with id 339 repos / releng / Gitlab Trusted Runner
+project with id 182 repos / releng / Gitlab Runner Test

If that looks good, you can execute add-project.py apply to authorize the project.

I'm planning to also create a CI, so the complete review and authorization workflow is inside of GitLab.

Further work

I'm planning to create non-blocking subtasks for

  • evaluate if containers can be executed as non-root
  • evaluate if certain capabilities can be dropped

This would be a useful addition to Trusted Runners but not strictly needed. But then we can move forward with making the Trusted Runners available for first projects.

Jelto updated the task description. (Show Details)May 17 2022, 12:16 PM
hashar unsubscribed.Jun 1 2022, 7:06 AM
Jelto added a comment.Jul 24 2022, 6:14 PM

I created CI jobs to assign authorized projects to Trusted Runners. This CI job uses the same script mentioned above. See /releng/gitlab-trusted-runner/gitlab-ci.yml.

The CI jobs need a global/personal access token, exposed in the variable $GITLAB_TOKEN. I tested the script with a project access token, but this token doesn't has enough permissions. I tested the Job successfully with a temporary personal access token, which is revoked again. I'm not sure about what degree of automation we aim for. We can create a access token with global read/write permission and store it in a protected CI variable.

Another approach could be to not store the access token in GitLab. Then the token has to be pasted into GitLab every time we run the job. This can be done in the Run pipeline button. With this approach we wouldn't be able to use scheduled ci jobs.

@brennen @Dzahn @Arnoldokoth : I would like to get some feedback what you think about putting GitLab admin credentials into automated CI jobs.

sbassett subscribed.Jul 25 2022, 5:04 PM

@Jelto - From a security perspective, as long as $GITLAB_TOKEN's value is never disclosed in any public, CI-related output and is only configurable by trusted Gitlab users, using it in this manner should be low risk. Inputting the value every time a CI job is triggered or run would likely only be marginally more secure, though would also be more prone to human error and obviously doesn't work for merge requests and pushes (but it sounds like the latter concern isn't that big of a deal in this case).

Jelto added a project: collaboration-services.Aug 1 2022, 2:53 PM
gerritbot added a comment.Aug 29 2022, 8:59 AM

Change 827456 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/cookbooks@master] sre.gitlab.reboot-runner: add cookbook to restart gitlab-runners

https://gerrit.wikimedia.org/r/827456

gerritbot added a comment.Sep 6 2022, 3:12 PM

Change 827456 merged by jenkins-bot:

[operations/cookbooks@master] sre.gitlab.reboot-runner: add cookbook to restart gitlab-runners

https://gerrit.wikimedia.org/r/827456

gerritbot added a comment.Sep 6 2022, 3:40 PM

Change 830189 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/cookbooks@master] sre.gitlab.reboot-runner: fix pre_scripts call

https://gerrit.wikimedia.org/r/830189

gerritbot added a comment.Sep 6 2022, 4:00 PM

Change 830189 merged by jenkins-bot:

[operations/cookbooks@master] sre.gitlab.reboot-runner: fix pre_scripts call

https://gerrit.wikimedia.org/r/830189

gerritbot added a comment.Sep 12 2022, 8:02 AM

Change 831481 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: allow Trusted Runners to access wikimedia docker-registry

https://gerrit.wikimedia.org/r/831481

gerritbot added a comment.Sep 12 2022, 8:29 AM

Change 831481 merged by Jelto:

[operations/puppet@production] gitlab_runner: allow Trusted Runners to access wikimedia docker-registry

https://gerrit.wikimedia.org/r/831481

Jelto mentioned this in T308271: Deploy buildkitd to trusted GitLab runners.Sep 14 2022, 8:38 PM
LSobanski moved this task from Incoming to Work in Progress on the collaboration-services board.Sep 20 2022, 3:41 PM
jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:20 PM
Jelto mentioned this in T320411: Evaluate if Docker can be executed as non-root on Trusted Runners.Oct 10 2022, 12:53 PM
Jelto updated the task description. (Show Details)
Jelto added a comment.Oct 10 2022, 12:58 PM

Some more explanation to the above edit:

Further security hardening of Docker daemon got a dedicated task T320411. Running Docker as non-root could enhance security even more but we should evaluate if that is needed at the moment.

I unchecked the firewall configuration because the currently configured firewall rules don't have the desired effect/are not working. This has to be fixed first until we can proceed with opening the Trusted Runners even more.

I also added buildkit in the list as an option to build docker images, because we are quite close of doing that (one test images was build and published already).

gerritbot added a comment.Oct 12 2022, 12:55 PM

Change 841910 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: restrict all internal traffic, not only TCP

https://gerrit.wikimedia.org/r/841910

gerritbot added a comment.Oct 12 2022, 1:05 PM

Change 841912 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab_runner: add webproxy to allowed_services

https://gerrit.wikimedia.org/r/841912

Jelto added a comment.Oct 12 2022, 1:21 PM

While doing more debugging around the firewall rules for Trusted Runners I found out the firewall mostly work, as defined in puppet.

The Trusted Runners use a more restrictive firewall setting, which configures the additional ferm rule docker-default-reject. This rule consists of daddr 10.0.0.0/8 proto tcp REJECT;.

However most of my test scripts used ICMP/ping instead of a TCP protocol. So it seemed the firewall has not the desired effect of blocking access to the private network (10.0.0.0/8). TCP access to internal services is blocked properly with the existing rule (for example curling a svc.eqiad.wmnet service).

In the above change I extended the firewall rule to all protocols instead of TCP only: daddr 10.0.0.0/8 REJECT;. This should make verifying this rule easier and prevent any kind of ICMP/UDP port scans and exploration attacks.

gerritbot added a comment.Oct 12 2022, 5:31 PM

Change 841910 merged by Dzahn:

[operations/puppet@production] gitlab_runner: restrict all internal traffic, not only TCP

https://gerrit.wikimedia.org/r/841910

gerritbot added a comment.Oct 12 2022, 5:52 PM

Change 841912 merged by Dzahn:

[operations/puppet@production] gitlab_runner: add webproxy to allowed_services

https://gerrit.wikimedia.org/r/841912

Jelto updated the task description. (Show Details)Oct 13 2022, 8:03 AM
Jelto mentioned this in T320730: Define access to external resources for GitLab CI Runners.Oct 13 2022, 2:04 PM
Jelto updated the task description. (Show Details)Oct 17 2022, 9:46 AM
Jelto closed this task as Resolved.Oct 17 2022, 10:00 AM

Firewall issues are solved with the patch above. So the Trusted Runners are functional now, including buildkitd support (T308271). A proof of concept Debian build and Docker Image build was successful.

There is a related tasks for further hardening of the Docker daemon T320411. Also some work is needed around more standardization like T320730, T304491 and T286958. But that's outside of this specific task. So I'm going to close this one.

Jelto mentioned this in rCCKB0ab5cf9ab108: sre.gitlab.reboot-runner: add cookbook to restart gitlab-runners.Dec 14 2022, 3:31 PM
Jelto mentioned this in rCCKB90cb78f12652: sre.gitlab.reboot-runner: fix pre_scripts call.
gerritbot added a comment.Feb 15 2023, 1:46 PM

Change 773746 abandoned by Jelto:

[operations/puppet@production] gitlab_runner: add option to drop Docker capabilities

Reason:

not needed at the moment

https://gerrit.wikimedia.org/r/773746

Maintenance_bot removed a project: Patch-For-Review.Feb 15 2023, 2:11 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits