You have plenty of options there already; presumably another would be to spin up a VM to do a build (then discard it afterwards)?

This is what we're currently using for Toolforge packages: https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/tree/main/debian-builder-bullseye

I'm looking at this as a KR for this quarter (for some packages we want to deploy), and I'm expecting to use a dgit-based approach (since that avoids a lot of hassle round source package formats, and makes sure what you build matches the tip of the relevant branch).

An example is https://gitlab.wikimedia.org/repos/data_persistence/swift/-/blob/dgit/bullseye-wikimedia/debian/.wmf-gitlab-ci.yml - this approach will work with any dgit-based tree (i.e. you should be able to dgit clone any Debian package regardless of packaging workflow / vcs use / quilt / etc ; work on it, and build it thus) - example run output and resulting artifacts.

It should also be possible to e.g. arrange to run dgit fetch to update the dgit/bullseye branch and then merge that to dgit/bullseye-wikimedia and rebuild the result semi-automatically.

IWBNI the resulting artifacts could be uploaded to apt.wm.org too :)

[this is still a bit work-in-progress]

Picking up the discussion from last weeks meeting and your mail @MatthewVernon :

I was able to include your wmf-debci pipeline abstraction in my test project wmf-sre-laptop. I used the following pipeline:

include:
  - project: 'repos/data_persistence/wmf-debci'
    file: 'builddebs.yml'

stages:  
  - build

test-build-job:
  extends: .build_wmf_deb
  stage: build
  variables:
    SUITE: "bullseye"

The job created a deb package: https://gitlab.wikimedia.org/repos/sre/wmf-sre-laptop/-/jobs/133891/artifacts/browse/WMF_BUILD_DIR/. Thanks for the work, I think that's a good step in the right direction!

I was not able to use builddebs.yml@repos/data_persistence/wmf-debci as my CI configuration file. When I configure this ci file I get:

Pipeline cannot be run.
Pipeline will not run for the selected trigger. The rules configuration prevented any jobs from being added to the pipeline.

Do I miss something when using a different CI configuration file from a external project?

Speaking about includes/extends and changing the CI configuration file. I think the option of changing the CI configuration file is quite convenient and may work for a lot of projects out of the box. But the image building abstraction kokkuri uses includes and extends. Maybe it makes sense to use the same type in both tools to make the user experience more homogeneous? However wmf-debci supports both, so that should be good.

[I'm in the process of tweaking the branch-naming convention for that - I'm instead moving to extracting the build suite from the changelog, and using branch name to work out which Debian suite to track - see wip.yml for some idea of how that's going to look, but maybe don't rely on it just yet :)

This is necessary because it turns out to be Hard to track upstream suites automatically, so moving to a convention of "call your branch SUITE-wikimedia[-foo]" to have it automatically track Debian SUITE solves that problem (and is reasonably easy to do since it naturally follows from the dgit clone operation), whereas build-suite is easy to determine from debian/changelog so we don't need to use branch name for that.
]

I think the failure you're seeing is because builddebs.yml uses branch name to determine what build to run, so only runs on branches with a name like "suite-wikimedia" - you don't have any such branches, so the CI will never run. This would be an advantage of my moving towards extracting build suite from changelog, we could potentially relax the branch-naming requirements for build jobs (...but we presumably then want some convention for "don't try and build this branch", so maybe still having "call your branch [something like this] to have it built automatically" is sensible?)

OK, I have enough progress to warrant an update here, I think.

Short version: you can now arrange for CI to check Debian for updates and attempt to merge those into your packaging branches and build the results. As an example, pcre2 in unstable was updated yesterday, following which CI updated the local dgit/sid branch and merged changes into the two tracking branches, which in turn triggered a build pipeline in the two tracking branches, resulting in two successful builds (for bullseye and for bookworm). For reference, an example of what the changelog entries look like.

I had to change the branch naming convention - name a branch foo-wikimedia (or foo-wikimedia-*) to have it track upstream release foo via the dgit branch dgit/foo, and instead the CI extracts the suite name to build for from debian/changelog, which is more correct anyhow.

There's some brief docs in comments in builddebs.yml itself, and the wmf-debci repo README; and rather fuller documentation on wikitech.

Noting here that we've moved the wmf-debci repo to https://gitlab.wikimedia.org/repos/sre/wmf-debci - so if you want to use it, the CI/CD configuration file setting should now be builddebs.yml@repos/sre/wmf-debci.

...now with added build tagging too :)

Change 963052 had a related patch set uploaded (by MVernon; author: MVernon):

[operations/puppet@production] aptrepo: install zip on aptrepo servers

https://gerrit.wikimedia.org/r/963052

Change 963052 merged by MVernon:

[operations/puppet@production] aptrepo: install zip on aptrepo servers

https://gerrit.wikimedia.org/r/963052

Documentation has now been updated in the main namespace - https://wikitech.wikimedia.org/wiki/Debian_packaging_with_dgit_and_CI and https://wikitech.wikimedia.org/wiki/Debian_packaging ; I've also put together a more WMF-specific tutorial on making new packages - https://wikitech.wikimedia.org/wiki/User:MVernon/Packaging_Tutorial

In T304491#9242773, @MatthewVernon wrote:

Documentation has now been updated in the main namespace - https://wikitech.wikimedia.org/wiki/Debian_packaging_with_dgit_and_CI and https://wikitech.wikimedia.org/wiki/Debian_packaging ; I've also put together a more WMF-specific tutorial on making new packages - https://wikitech.wikimedia.org/wiki/User:MVernon/Packaging_Tutorial

Thanks for creating and updating the docs! The steps described in https://wikitech.wikimedia.org/wiki/User:MVernon/Packaging_Tutorial are straight forward and focus on the most important topics (as Debian package building can get quite complex).

I was able to follow and do a (noop) upstream update of my test mirror of wmf-sre-laptop. It also worked with setting the ci file to builddebs.yml@repos/sre/wmf-debci. It's a bit tricky that you have to use the branch name .*wikimedia.*, otherwise the CI jobs won't pickup the job. But as soon as I read the doc properly and used the branch name packaging-wikimedia, the CI job worked fine.

Cool, I'm glad it's working for you :)

I'm not wedded to the .*-wikimedia.* naming restriction, but I think some restriction on which branches get auto-built is necessary (e.g. you want to be able to keep upstream's main or master in gitlab without the CI trying to build it), and given the use of e.g. bookworm-wikimedia.* for branches tracking Debian updates to bookworm, it seemed to me reasonable to start with .*-wikimedia.* (since it allows you to use e.g. packaging-wikimedia for packaging branches that aren't tracking upstream, and I've also reviewed some packaging done by a colleague that went with suite-wikimedia-bookworm and suite-wikimedia-bullseye for their two branches).

Tutorial now moved to main namespace Debian_packaging/Tutorial.
And support for using the -backports suite added (T350658).

I just found this task while looking for ways to build .deb packages from Gitlab. My understanding is that https://gitlab.wikimedia.org/repos/sre/wmf-debci is currently the state of the art, and it does address all the requirements initially drafted by @Jelto in the description of this task.

Is anything missing before this task can be resolved? And does anybody have other short or long-term plans for improving this setup?

Two features I would like to add (but should be tracked in separate Phab tasks):

• Building on all branches, not just on the main branch, e.g. when testing a MR I want to know if the package build is successful, even if I don't want to publish the .deb to a repository
• Reproducing the build on my laptop, running exactly the same commands. This will likely require moving the pipeline steps to a script that can be run inside the container.

I'm very much interested in sending some MRs, but I would like to double check if somebody else is already working on it, or if there are plans to go for entirely different approaches.

Side note: WMCS currently maintains a similar setup for publishing .deb packages to the Toolforge Apt repo. Ideally the two setups could share the same build script.

A couple of thoughts on your desiderata:

Building on all branches would require, I think, a separate workflow file (e.g. like wmfdeb.yml), because builddebs.yml makes assumptions about branches (e.g. that dgit/* is a tracking branch and so shouldn't be built against), but you could ensure all your branches were built by always naming them suitably (so that the regex /.*-wikimedia.*$/ matches).

I think you needn't have the full setup for local building - it's really just "install build-dependencies (mk-build-deps -i debian/control) then run dpkg-buildpackage -uc -b".

We also make assumptions about the state of the repository (that it should be a patches-applied tree); I don't know if these are suitable for how you build things.

The aim with wmf-debci was to make a general-purpose tool, so I'd hope we could make it work for WMCS.

@MatthewVernon thanks for your quick reply!

Building on all branches would require, I think, a separate workflow file (e.g. like wmfdeb.yml)

Yes, I should have specified that the use case I was thinking of is publishing custom packages with wmfdeb.yml, where the repo has no upstream and no special dgit branches.

I think you needn't have the full setup for local building - it's really just "install build-dependencies (mk-build-deps -i debian/control) then run dpkg-buildpackage -uc -b".

Yes, and in most cases this is fine. But sometimes it's useful to run the build exactly in the same docker container the CI uses, in order to remove any difference from your local environment and the CI, which might cause things to work locally but not in the CI or vice versa.

A container-based workflow is also very useful if you're on a different Linux distro, or on macOS like myself.

The aim with wmf-debci was to make a general-purpose tool, so I'd hope we could make it work for WMCS.

I think it's definitely worth trying, our requirements are not very special. :)