Even though RelEng already decided in T307810 on using BuildKit as the way to build images in GitLab , I'm writing down the outcome of this story for documentation purposes.

A proof of concept can be found here. The Makefile and the manifests can be used to build a K8s cluster with a GitLab runner that generates images using Kaniko. A couple of branches in that repo demonstrate and compare the generation of images with and without remote caching.

When compared to the BuildKit solution:

• Benefits:
  • Rootless/daemonless. No special privileges required and no possibility of building containers interfering with each other
  • Entire build process contained within the executor container. No need to maintain a separate build environment and no need to secure access to it
  • Management of remote cache layers via TTLs
• Downsides:
  • No support for Dockerfile frontends
  • No local caching (except for base images)
  • No multi-architecture support. Although in our case we probably don't need this
  • Re-implementation of low-level features. In the best-case scenario, some features already in BuildKit will take longer to be implemented
  • (Potentially) Access to credentials to publish production images is required. But this wouldn't be an issue for non-prod images or when using a proxy registry