This task is for tracking the setup of GitLab Runners in a trusted environment.
In T286958 we discussed the long term requirements for GitLab Runners. One class of Runners should run in production environments (eqiad, codfw) and execute jobs which handle sensitive credentials and produce artifacts running in production. See also https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner#Specific_GitLab_Runners.
I would like to reuse the existing puppet code for the Shared Runners in WMCS. I think we could start with VMs on Ganeti and later order dedicated machines and/or migrate to //some// Kubernetes platform.
The Runners must not be used by arbitrary jobs but only by certain projects and branches. So this runners will be setup as Specific Runners, probably executing only jobs for protected branches.
Roughly the needed steps are:
[x] setup dedicated ganeti VMs in codfw and eqiad (`gitlab-runner1001` and `gitlab-runner2001`)
[x] adjust puppet code and install role on new ganeti VMs
[x] register new runners as specific runners and run test job
[x] add monitoring (Prometheus metrics and some Grafana dashboards)
[x] validate GitLab application permission concept ⌛
[x] runners should execute protected branches only
[x] runners should be available only to allowed projects
[x] non-privileged project pipelines (feature branches) can not escalate privileges by altering `gitlab-ci.yml` file
[x] make sure trusted runners must use protected branches
[x] document [permission and security concept](https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Security_Evaluation)
[] validate host security concept
[] runner jobs shouldn't be able to connect to other WMF service or hosts (except explicitly permitted, like docker-registry, apt repos, chart museum)
[] harden firewall rules so no other services can be reached from GitLab Runner Docker containers
[x] runners shouldn't be able to execute code with root privileges/escalate to root privileges
[x] prevent privileged containers (`privileged = false`)
[x] run `gitlab-runner` as non-root user
[] ~~evaluate if containers can be executed as non-root~~ See T320411
[] ~~evaluate if certain capabilities [can be dropped](https://docs.gitlab.com/runner/security/#usage-of-docker-executor)~~ See T320411
[x] create automation for managing and requesting access to Trusted Runners [/repos/releng/gitlab-trusted-runner](https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner)
[] migrate one prof-of-concept Docker Image pipeline to GitLab trusted runner
[] usage of buildkitd T308271
[] maybe T307536?
[x] migrate one prof-of-concept Debian package pipeline to GitLab trusted runner ([wmf-sre-laptop](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop)) further work in T304491
[x] open task for security review of Trusted Runners - T304491