- Title of session: Using wmf-debci to build Debian packages on GitLab
- Session description:
In this session we will present the tool wmf-debci. wmf-debci standardizes GitLab CI pipelines for Debian package building by providing a CI-template which can be included in other projects. wmf-debci is an effort from the Site Reliability Engineering team because a lot of software we use is distributed using Debian packages. But any software which has to be packaged and installed on a Debian system can benefit from this standardization.
We will cover the basics of Debian package building, the usage of wmf-debci, different use cases and some examples projects.
- Username for contact: @Jelto & @brennen
- Session duration (25 or 50 min): 25 minutes
- Session type (presentation, workshop, discussion, etc.): presentation
- Language of session (English, Arabic, etc.): English
- Prerequisites (some Python, etc.): Debian package building
- Any other details to share?:
- Interested? Add your username below:
Notes from session:
Using wmf-debci to build Debian packages on GitLab
Date and time: 2024-05-03 @ 16:00
Relevant links
- Phabricator task: https://phabricator.wikimedia.org/T361922
- Session slides: https://docs.google.com/presentation/d/1cQoPSzKWZw-JlS7VNiCkGDCa1iaN52DvMpfzzisHeVw/edit?usp=sharing
- https://gitlab.wikimedia.org/repos/sre/wmf-debci
Presenter
@[[phab:p/Jelto/|Jelto]]
Participants
Notes
- wmf-debci
- A standardized way of building Debian packages
- Some influence from Kokkuri, which is for building for container images
- Mostly the same, but for Debian packages
- No abstraction layer like Blubber
- Basics of Debian packages
- Can be quite easy, but can be quite a deep rabbithole
- Overview of Debian package use cases
- System level stuff, applications, etc.
- Distributed via repositories
- We have an apt.wikimedia.org
- A bit like an app store. Unified interface for installing, upgrading, removing software.
- Handle dependencies automatically
- Two kinds of packages:
- Source packages - tarball of upstream source code, Debian-specific changes, checksums, can be used to build the binary package
- Binary package: The built artifact
- We mostly don't care about source packages - we track source code in git (GitLab or Gerrit)
- Demo: The nano editor's source package
- <code>apt source nano</code>
- We mostly use degit to download source code and automatically create a repository
- Package building:
- debian/ folder contains a bunch of metadata
- debian/control, debian/copyright, debian/changelog, debian/rules
- There are tools that handle these files, they can be created with dh_make.
- Binary packages (.deb):
- Contains the compiled executable(s), config files, man pages, metadata
- Can be distributed by apt.wikimedia.org
- For example, <code>apt-get download nano</code>
- Gives you an archive, which you can extract with <code>ar</code>, in turn contains control data, files to be installed, etc.
- Files to be installed are in a corresponding layout to the root filesystem of the machine
- This is all pretty complicated. Tried to simplify it so that you just have source and some GitLab CI and you are mostly done.
- wmf-debci
- Reusable .gitlab-ci.yml templates
- Implements all the build magic
- Standardizes build process for all the Debian packages we build on GitLab
- Works in a container
- Some builds use chroot, doesn't work properly in a container
- Usage:
- Can just override the CI/CD Configuration file to use builddbs.yml@repos/sre/debci
- Can use the Kokkuri-style inclusion of the CI file and extend
- Can automatically track upstream code - dgit/<suite> branch. Useful for automatically getting for example security updates. Can be tricky if the package is changed a lot. Stable Debian packages are a good candidate for this.
- Putting it all together
- Similar to Kokkuri
- If we push to a specific branch (one for the Debian release - buster, bullseye, bookworm, etc.), then packages build.
- If you want you can track upstream.
- Work in progress: Automatic syncing to apt hosts, to cut out the manual GitLab download-and-upload cycle
- Examples
- pcre2 - overwrites GitLab CI file to use the wmf-debci one
- swift
- wmf-sre-laptop - does an include