Page MenuHomePhabricator

[Session] Using wmf-debci to build Debian packages on GitLab
Closed, ResolvedPublic


  • Title of session: Using wmf-debci to build Debian packages on GitLab
  • Session description:

In this session we will present the tool wmf-debci. wmf-debci standardizes GitLab CI pipelines for Debian package building by providing a CI-template which can be included in other projects. wmf-debci is an effort from the Site Reliability Engineering team because a lot of software we use is distributed using Debian packages. But any software which has to be packaged and installed on a Debian system can benefit from this standardization.
We will cover the basics of Debian package building, the usage of wmf-debci, different use cases and some examples projects.

  • Username for contact: @Jelto & @brennen
  • Session duration (25 or 50 min): 25 minutes
  • Session type (presentation, workshop, discussion, etc.): presentation
  • Language of session (English, Arabic, etc.): English
  • Prerequisites (some Python, etc.): Debian package building
  • Any other details to share?:
  • Interested? Add your username below:

Notes from session:

Using wmf-debci to build Debian packages on GitLab

Date and time: 2024-05-03 @ 16:00

Relevant links





  • wmf-debci
  • A standardized way of building Debian packages
  • Some influence from Kokkuri, which is for building for container images
  • Mostly the same, but for Debian packages
  • No abstraction layer like Blubber
  • Basics of Debian packages
    • Can be quite easy, but can be quite a deep rabbithole
    • Overview of Debian package use cases
      • System level stuff, applications, etc.
      • Distributed via repositories
      • We have an
    • A bit like an app store. Unified interface for installing, upgrading, removing software.
    • Handle dependencies automatically
    • Two kinds of packages:
      • Source packages - tarball of upstream source code, Debian-specific changes, checksums, can be used to build the binary package
      • Binary package: The built artifact
    • We mostly don't care about source packages - we track source code in git (GitLab or Gerrit)
    • Demo: The nano editor's source package
      • <code>apt source nano</code>
    • We mostly use degit to download source code and automatically create a repository
  • Package building:
    • debian/ folder contains a bunch of metadata
    • debian/control, debian/copyright, debian/changelog, debian/rules
    • There are tools that handle these files, they can be created with dh_make.
  • Binary packages (.deb):
    • Contains the compiled executable(s), config files, man pages, metadata
    • Can be distributed by
    • For example, <code>apt-get download nano</code>
      • Gives you an archive, which you can extract with <code>ar</code>, in turn contains control data, files to be installed, etc.
      • Files to be installed are in a corresponding layout to the root filesystem of the machine
  • This is all pretty complicated. Tried to simplify it so that you just have source and some GitLab CI and you are mostly done.
  • wmf-debci
    • Reusable .gitlab-ci.yml templates
    • Implements all the build magic
    • Standardizes build process for all the Debian packages we build on GitLab
    • Works in a container
      • Some builds use chroot, doesn't work properly in a container
    • Usage:
      • Can just override the CI/CD Configuration file to use builddbs.yml@repos/sre/debci
      • Can use the Kokkuri-style inclusion of the CI file and extend
      • Can automatically track upstream code - dgit/<suite> branch. Useful for automatically getting for example security updates. Can be tricky if the package is changed a lot. Stable Debian packages are a good candidate for this.
  • Putting it all together
    • Similar to Kokkuri
    • If we push to a specific branch (one for the Debian release - buster, bullseye, bookworm, etc.), then packages build.
    • If you want you can track upstream.
    • Work in progress: Automatic syncing to apt hosts, to cut out the manual GitLab download-and-upload cycle
  • Examples
    • pcre2 - overwrites GitLab CI file to use the wmf-debci one
    • swift
    • wmf-sre-laptop - does an include





Other Assignee

Event Timeline

Hello! 👋 The 2024 Hackathon Program is now open for scheduling! If you are still interested in organizing a session, you can claim a slot on a first-come, first-serve basis by adding your session to the daily program, following these instructions. We look forward to hearing your presentation!

debt triaged this task as Medium priority.Apr 17 2024, 7:07 PM
debt updated the task description. (Show Details)
Jelto updated the task description. (Show Details)

I this can be closed, thanks for taking and copying the notes here @debt !