Page MenuHomePhabricator

Revamp the build process for debian packages in Toolforge
Closed, ResolvedPublic

Description

With a little work, the tools-package-builder-02 host in Toolforge could be set up with a simple and more modern workflow for building Debian packages.

Since the role is now role::wmcs::toolforge::package_builder, it should be possible to move to an sbuild based procedure with everything in place for users to build for the needed distros.

This will also include updating the docs to include release procedures, etc.

Event Timeline

Bstorm triaged this task as Low priority.Apr 9 2020, 4:05 PM
Bstorm created this task.
aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.

Will take a look at this soon!

I'm reviewing the current steps we have in our workflow:

image.png (556×1 px, 100 KB)

(from https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Bd808's_method_(pdebuild))

Those are a lot of steps, but most of them are related to publishing and managing the apt repository and not building the package itself.
Building the package is a single step (step 5). If I introduce sbuild here we would still have a long workflow anyway.

I will work on replacing pdebuild with sbuild, but perhaps it makes sense to start a discussion on the whole workflow rather than the concrete build tool we use (which is just a little step in a long workflow).

What would be our ideal workflow here? Perhaps something like adding a tag in the git repo and then we get the deb package in the apt repo directly?

Change 587991 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] sbuild: introduce module and use it in toolforge package builder

https://gerrit.wikimedia.org/r/587991

What would be our ideal workflow here? Perhaps something like adding a tag in the git repo and then we get the deb package in the apt repo directly?

I think something like that would definitely be the ideal. I don't know how well-suited to that the pipeline config in production is. It may end up waiting on things like T249946: Basic evaluation of gitea as a repo setup? I could be wrong about that, though. It's just very focused around deploying to production k8s at this time vs something like committing into our repos.

Our own repo mirrors would definitely seem capable on the other hand. That's why I'm looking at the mirroring ability of things like that.

We could script some of the steps for distributing on the services nodes at least. That might not be a bad intermediate step.

Change 587991 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] sbuild: introduce module and use it in toolforge package builder

https://gerrit.wikimedia.org/r/587991

Change 588987 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] sbuild: chroot: update it using the root user

https://gerrit.wikimedia.org/r/588987

Change 588987 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] sbuild: chroot: update it using the root user

https://gerrit.wikimedia.org/r/588987

Change 591026 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: add wmcs-package-build.py script

https://gerrit.wikimedia.org/r/591026

Change 591026 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: add wmcs-package-build.py script

https://gerrit.wikimedia.org/r/591026

I introduced the wmcs-package-build.py script and created https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Packaging

comments and changes welcome :-)

Change 591318 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] toolforge: wmcs-package-build: don't use the $INSTANCEPROJECT env var

https://gerrit.wikimedia.org/r/591318

Change 591318 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] toolforge: wmcs-package-build: don't use the $INSTANCEPROJECT env var

https://gerrit.wikimedia.org/r/591318

Summary of what was done here:

  • sbuild was introduced in our package builder VM
  • the wmcs-package-build script was created to automate most of our current workflow for building & distributing a package
  • some docs were created in wikitech

I think I'll stop working on this for now, until we need more iterations.

I may have some ideas on facilitating even more the testing process etc by extending the script. For example, being able to handle the case in which we need a quick package release just for testing purposes (i.e, per-change package build so we are able to tests individual patches). This might be as easy as generating a temporal (snapshot) d/changelog entry, building the package and publishing a "testing" repo (or just toolsbe.

Please @Bstorm reopen if required.