Basic Information Section
This task tracks a Security Review of a subset of GitLab CI workers. This CI workers are inside WMF infrastructure and are called "Trusted Runners".
The Security Review will happen on the test instance running in WMCS: https://gitlab.devtools.wmcloud.org. The test instance has a mockup of shared and trusted Runners, similiar to the production instance. However all Runners available for the test instance run in WMCS too.
The project /repos/runner-test-project should be used for testing and evaluation of the Runner Security and configuration.
Brief description
GitLab Runner offer CI capabilities, so arbitrary code can be submitted as a CI job. This code gets executed on GitLab Runners. This jobs have multiple tiers of trust and security. Some jobs will only do linting or testing, whereas other jobs produce builds and artifacts running in production. This concept was mirrored in multiple tiers of Runners.
A set of Shared Runners was created for less critical CI jobs. This Shared Runners should not be part of this Security Review.
Jobs with more critical CI jobs will run on the Trusted Runners. To increase the trust and reliability of certain CI jobs, two GitLab Runner were created inside WMF infrastructure. This also means that a this Trusted Runners could reach a wide range of WMF production infrastructure, if getting compromised. So access to this Runners was restricted and additional security measures were implemented. Most of this can be found at security related documentation of Trusted Runners.
Do you have a project/product/program plan or documentation?
GitLab Runner overview documentation
Trusted Runner documentation
Security related documentation of Trusted Runner
Task for setting up the Trusted Runners: T295481
Documentations will change the next few weeks as additional features get implemented.
Primary Contacts
What Security Team services do you anticipate needing?
Security Readiness Reviews
What is the 'go live' date for deployment of this project
2-3 months
Privacy Information Section
Will any sensitive data to be collected, stored or exposed?
Certain CI jobs will need credentials to access infrastructure. So sensitive data like the following can be expected:
- tokens/certificates to access Kubernetes
- keys/tokens to access other WMF machines (like apt repo, helm chart museum, docker registry)
- tokens to access other infrastructure (WMCS, public clouds)
- passwords for technical users/logins
- keys to sign packages
Technical Information Section
Do related discussions exist in Phab, on wiki, or in an RFC'?
Task for setting up the Trusted Runners: T295481
Technology Stack
The current Runner cluster consist of two WMCS VMS:
gitlab-runner-1002.devtools.eqiad1.wikimedia.cloud (shared, non-trusted Runner) gitlab-runner-1003.devtools.eqiad1.wikimedia.cloud (trusted Runner)
Dedicated bare metal hosts are expected be deployed in Q4 FY2021/22 (pending hardware delivery)
Trusted Runners use puppet code for (role(gitlab_runner)).
The Runners use the gitlab-runner (see) executable which is written in golang. The Runners have a Docker environment to execute all CI jobs inside a separated Docker container. Runners also have Prometheus metrics exporter enabled.
Security Readiness Review Section
- Below is only relevant if this Project has reached maturity and requires a Readiness review.
- You can fill this in later if you are still in the Preview or other early phases :)
Code
Puppet configuration:
GitLab Runner modules: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/gitlab_runner/
GitLab Runner profiles: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/gitlab/runner.pp
GitLab Runner role: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/role/manifests/gitlab_runner.pp
GitLab Runner hiera data: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/gitlab_runner.yaml
Dependencies:
https://gitlab.devtools.wmcloud.org
Post-deployment
- Name of team responsible for project support post-deployment and primary contact(s).
ServiceOps
Working test environment
There is a GitLab test instance at https://gitlab.devtools.wmcloud.org/explore however this instance has no Runners. It would be possible to create additional test Runners, but the setup is a bit different because the test instance is running in WMCS/VPS.
It would be possible to block one Trusted Runner for security team during the review. It would also be possible to assign this Runner to the GitLab replica if certain tests could compromise the production instance. The replica can be found here https://gitlab-replica.wikimedia.org/ and is restored every 24 hours.