Release Engineering's "GitLab-a-thon" sprint for May 10th-24th (roughly) focused on the mechanics of migrating a Wikimedia service to GitLab, setting up a CI pipeline, building container images from that service, and publishing images to the Wikimedia registry. We selected the Blubber project as a good candidate for experimentation:
- Workboard: Release-Engineering-Team (GitLab-a-thon 🦊)
- T301168: Migrate Blubber project to GitLab
- T307536: Build and Run Blubber test variant on GitLab untrusted runners
- T307599: Investigate alternatives to docker-in-docker for container image creation in GitLab
- T308213: Investigate Kaniko as an option to build CI images
- T307810: Investigate buildkitd instances as image builders for GitLab
We ultimately landed on BuildKit as the least constraining for future options, and the most in line with features we'd like to offer.
We explored a range of options for building and publishing, including variations on:
- Building on runners provisioned on a DigitalOcean Kubernetes cluster and importing to the production registry from some trusted location (contint, for example) by way of a shim.
- Building on trusted runners and publishing to the GitLab Container Registry, then importing to the production registry by way of a shim.
- Building on trusted runners and publishing directly from there to the prod registry, authenticated against GitLab by way of JWT.
We eventually landed on this latter, and work is well underway on implementation: T308501: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token
Other work included implementing CI for Blubber on GitLab (T307534), improvements to user-facing documentation (T307535, T307538), enforcing the allowlist for container images in GitLab CI (T291978), experimentation with the GitLab Container Registry (T307537), and extensive discussions with ServiceOps on GitLab infrastructure.