Change Details

This task tracks the future of Debian package builds on GitLab CI. This task came up when setting up trusted GitLab Runners (T295481) and creating a proof-of-concept Debian package build in CI for [/repos/sre/wmf-sre-laptop](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop). Properly building Debian packages in CI (and Docker) goes beyond the proof-of-concept done in [/repos/sre/wmf-sre-laptop](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop). With the introduction of GitLab CI and Trusted Runners we gain additional CI capabilities. So we should evaluate if the want to use this CI capabilities for building Debian packages in the future. ## Building .debs in CI Most documentation recommend to use privileged containers for building. Due to serious security concerns privileged containers are disabled for Trusted GitLab Runners (see [Security_Evaluation#Security_measures](https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Security_Evaluation#Security_measures)). So we have to create a alternative solution on how we want to build Debian packages in CI. I see multiple routes which could be taken: * Create a a deneb-like Docker build image which can be used without additional `chroot`/`pbuilder`. Builds can be executed with `dpkg-buildpackage` on Trusted Runners as standard Docker CI jobs then. I'm not sure if one build image is needed or every Debian package needs its own build image. * Create a workaround to use `chroot`/`pbuilder` inside Docker without privileged containers * Use [shell executor](https://docs.gitlab.com/runner/executors/shell.html) (or privileged containers) to run builds on a dedicated machine similar to package build host `deneb`. With this approach, no ephemeral builds are possible and shell executor is less secure than Docker executor. However `chroot`/`pbuilder` could be used on the host. * evaluate [whalebuilder](https://gitlab.com/uhoreg/whalebuilder)? * Don't build Debian packages in CI and keep using package builder host ## Releasing .debs GitLab CI creates job artifacts ([see](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop/-/jobs/10766/artifacts/browse/build/) for an example.) This artifacts could be used to publish them on apt hosts/repositories. It would be possible to push artifacts to apt machines or to pull them from apt machines over GitLab API. The process could be automated or half-automated (like some manual steps on apt would be needed as a additional review step). The exact workflow on how to transfer artifacts to apt hosts is to be discussed. It would also be possible to leverage GitLabs [releases](https://gitlab.wikimedia.org/help/user/project/releases/index) feature. ## Additional conciderations I found some efforts around scap builds on Docker in T265501.

This task tracks the future of Debian package builds on GitLab CI. This task came up when setting up trusted GitLab Runners (T295481) and creating a proof-of-concept Debian package build in CI for [/repos/sre/wmf-sre-laptop](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop). Properly building Debian packages in CI (and Docker) goes beyond the proof-of-concept done in [/repos/sre/wmf-sre-laptop](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop). With the introduction of GitLab CI and Trusted Runners we gain additional CI capabilities. So we should evaluate if the want to use this CI capabilities for building Debian packages in the future. ## Building .debs in CI Most documentation recommend to use privileged containers for building. Due to serious security concerns privileged containers are disabled for Trusted GitLab Runners (see [Security_Evaluation#Security_measures](https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Security_Evaluation#Security_measures)). So we have to create a alternative solution on how we want to build Debian packages in CI. I see multiple routes which could be taken: * Create a deneb-like Docker build image which can be used without additional `chroot`/`pbuilder`. Builds can be executed with `dpkg-buildpackage` on Trusted Runners as standard Docker CI jobs then. I'm not sure if one build image is needed or every Debian package needs its own build image. * Create a workaround to use `chroot`/`pbuilder` inside Docker without privileged containers * Use [shell executor](https://docs.gitlab.com/runner/executors/shell.html) (or privileged containers) to run builds on a dedicated machine similar to package build host `deneb`. With this approach, no ephemeral builds are possible and shell executor is less secure than Docker executor. However `chroot`/`pbuilder` could be used on the host. * evaluate [whalebuilder](https://gitlab.com/uhoreg/whalebuilder)? * Don't build Debian packages in CI and keep using package builder host ## Releasing .debs GitLab CI creates job artifacts ([see](https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop/-/jobs/10766/artifacts/browse/build/) for an example.) This artifacts could be used to publish them on apt hosts/repositories. It would be possible to push artifacts to apt machines or to pull them from apt machines over GitLab API. The process could be automated or half-automated (like some manual steps on apt would be needed as a additional review step). The exact workflow on how to transfer artifacts to apt hosts is to be discussed. It would also be possible to leverage GitLabs [releases](https://gitlab.wikimedia.org/help/user/project/releases/index) feature. ## Additional conciderations I found some efforts around scap builds on Docker in T265501.