Page MenuHomePhabricator

Deprecate use of bullseye-backports
Closed, ResolvedPublic

Description

backports only exists for Debian stable and Debian oldstable, but not for Debian LTS suites. Since Bullseye is in LTS stage it no longer gets updated and will be archived at some point.

All uses of bullseye-backports in Puppet and container images need to be revisited, if there's an ongoing need for a package from bullseye-backports it should be moved to a dedicated repository component.


On 2025-07-22, bullseye-backports was archived, which uncovered that our bullseye base images still referenced it in sources.list. This breaks operations such as apt-get update in dependent image builds.

The work to identify packages that need moved to dedicated components or images that need updated to bookworm had already happened, and the fact that the source was left in the base image was just an oversight.

Current status:

As of ~ 02:00 UTC on 2025-07-23:

  • bullseye-backports has been removed (r/1171716) from the bullseye base image
  • the bullseye base image has been rebuilt (published as docker-registry.discovery.wmnet/bullseye:20250723, tagged latest)
  • the php8.1 production images have been rebuilt (r/1171747) in order to expedite fixing MediaWiki image builds during scap deployments and a test deployment has been completed

Follow-up actions:

  • A full production-images rebuild should be started in order to update dependent images other than php8.1. -- Decision: Unless anything urgent comes up, we'll just wait for the weekly rebuild.
  • Affected CI images (e.g., in integration/config) should be rebuilt. -- Done by @dancy and @Jdforrester-WMF (r/1172093, r/1172111). Note that CI jobs that rely directly on the bullseye:latest image should already be fixed.
  • r/1131630 should be revived to remove these now-broken-but-unused images. -- Done by @elukey (together with r/1171985).

Note: additional follow-up actions might be needed per history on T362518: Deprecate buster-backports.

Additional points of note:

  • golang1.19 users: If you use the golang1.19 image as a builder in a multi-stage build, where the resulting binaries are copied into a bullseye-based image for production use, note that is has been updated from bullseye to bookworm (r/1131631). In order to avoid glibc errors in the resulting image due to mismatch between Debian versions, you will likely need to migrate your later-stage image to bookworm.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
integration/configmaster+110 -110
integration/configmaster+377 -0
integration/configmaster+377 -0
operations/docker-images/production-imagesmaster+0 -1 K
operations/docker-images/production-imagesmaster+18 -0
operations/puppetproduction+1 -1
operations/puppetproduction+3 -1
operations/docker-images/production-imagesmaster+8 -9
operations/puppetproduction+1 -1
operations/puppetproduction+1 -19
operations/puppetproduction+0 -10
operations/puppetproduction+4 -2
operations/puppetproduction+1 -0
operations/puppetproduction+0 -6
operations/puppetproduction+3 -2
operations/puppetproduction+1 -0
operations/puppetproduction+1 -15
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1125413 merged by Muehlenhoff:

[operations/puppet@production] keepalived: Install keepalived from the "main" component

https://gerrit.wikimedia.org/r/1125413

Mentioned in SAL (#wikimedia-operations) [2025-03-10T16:00:40Z] <moritzm> imported keepalived 1:2.2.7-1~bpo11+1 to main component of bullseye-wikimedia T383557

Mentioned in SAL (#wikimedia-operations) [2025-03-21T09:27:55Z] <moritzm> imported python3-flask-sqlalchemy 2.1-4 to main component of wikimedia-bullseye (imported from bullseye-backports which will be archived soon) T383557

Change #1130078 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] dynamicproxy::api: Install python3-flask-sqlalchemy from "main" component

https://gerrit.wikimedia.org/r/1130078

Change #1130082 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Stop including bullseye-backports on Bullseye hosts

https://gerrit.wikimedia.org/r/1130082

Change #1130084 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] apt::package_from_bpo: Fail if used on Bullseye

https://gerrit.wikimedia.org/r/1130084

Change #1130078 merged by Muehlenhoff:

[operations/puppet@production] dynamicproxy::api: Install python3-flask-sqlalchemy from "main" component

https://gerrit.wikimedia.org/r/1130078

Change #1130082 merged by Muehlenhoff:

[operations/puppet@production] Stop including bullseye-backports on Bullseye hosts

https://gerrit.wikimedia.org/r/1130082

All uses of bullseye-backports in Puppet have been remove and I've merged a patch so that bullseye-backports is no longer added to the apt config.

Change #1131630 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/docker-images/production-images@master] Remove golang-1.17 and golang-1.18 images

https://gerrit.wikimedia.org/r/1131630

Change #1131631 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/docker-images/production-images@master] Update the 1.19 image to be based on Bookworm, not bullseye-backports

https://gerrit.wikimedia.org/r/1131631

Change #1131631 merged by Muehlenhoff:

[operations/docker-images/production-images@master] Update the 1.19 image to be based on Bookworm, not bullseye-backports

https://gerrit.wikimedia.org/r/1131631

Change #1130084 abandoned by Muehlenhoff:

[operations/puppet@production] apt::package_from_bpo: Fail if used on Bullseye

https://gerrit.wikimedia.org/r/1130084

MoritzMuehlenhoff claimed this task.

All uses of bullseye-backports have been removed and bullseye-backports is no longer included in our bullseye hosts.

Mentioned in SAL (#wikimedia-operations) [2025-04-29T12:05:52Z] <moritzm> imported python3-flask-sqlalchemy 2.1-4 to main component of wikimedia-bullseye (imported from bullseye-backports which will be archived soon) and removed the previously, erronously import into bookworm-wikimedia T383557

dancy subscribed.

The docker-registry.wikimedia.org/bullseye:latest@sha256:c9fff9943e3e3d42774f94b6ef07c1c53c417fc9fa964400d769b21a4f3ae28f image (which is the current latest at this time of writing) still references bullseye-backports. This is resulting in CI errors for scap. For example https://gitlab.wikimedia.org/repos/releng/scap/-/jobs/568451

I just ran into this as well. It appears that we're still configuring bullseye base image builds to include bullseye-backports in the sources list.

I suspect that means we're going to need to extend the conditional here, rebuild the base images, and rebuild dependent production images.

Edit: This will likely look a lot like what was done in T362518: Deprecate buster-backports, except here it seems as if the parts surrounding creation of components for backported packages has already happened.

Change #1171716 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] docker: remove bullseye-backports from sources.list

https://gerrit.wikimedia.org/r/1171716

I'm no longer seeing any references to bullseye-backports in puppet, so I believe Moritz took care of all of those.

Once https://gerrit.wikimedia.org/r/c/operations/puppet/+/1171716 is merged, we'll need to rebuild the bullseye base image, which I think uses the procedure described here, e.g.,

sudo -i
set_proxy
DISTRIBUTIONS="bullseye" build-base-images

Once that's done, I believe we'll need to kick off production-images builds for all (transitively) bullseye-based images. We should prioritize rebuilding the php8.1 images, in order to get deployments out of their current risky state.

In parallel, any CI images that depend on the bullseye base image can also be rebuilt.

Separately, I see that https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1131630 attached to this task is still pending. It looks like that should be unblocked now per T390139.

Tagging serviceops as well, since this will require work on our end of things.

Change #1171716 merged by Scott French:

[operations/puppet@production] docker: remove bullseye-backports from sources.list

https://gerrit.wikimedia.org/r/1171716

Mentioned in SAL (#wikimedia-operations) [2025-07-23T00:33:09Z] <swfrench-wmf> ran DISTRIBUTIONS="bullseye" build-base-images on build2001 - T383557

Change #1171747 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/docker-images/production-images@master] php8.1: rebuild to pick up removal of bullseye-backports

https://gerrit.wikimedia.org/r/1171747

Change #1171747 merged by Scott French:

[operations/docker-images/production-images@master] php8.1: rebuild to pick up removal of bullseye-backports

https://gerrit.wikimedia.org/r/1171747

Mentioned in SAL (#wikimedia-operations) [2025-07-23T00:46:15Z] <swfrench-wmf> rebuilt php8.1 production images (8.1.33-1-s2) on build2001 - T383557

Alright, MediaWiki deployments should no longer be at risk: the php8.1 production images have been rebuilt on docker-registry.discovery.wmnet/bullseye:20250723 and no longer reference bullseye-backports.

There's still work to do as described in T383557#11026158, which I'll summarize in the task description, but will otherwise plan to hands-off for the evening.

Mentioned in SAL (#wikimedia-operations) [2025-07-23T01:32:21Z] <swfrench@deploy1003> Started scap sync-world: Test deployment to verify new php8.1 images - T383557

Mentioned in SAL (#wikimedia-operations) [2025-07-23T02:04:33Z] <swfrench@deploy1003> Finished scap sync-world: Test deployment to verify new php8.1 images - T383557 (duration: 34m 39s)

Change #1131630 merged by Elukey:

[operations/docker-images/production-images@master] Remove golang-1.17 and golang-1.18 images

https://gerrit.wikimedia.org/r/1131630

Should we perhaps use latest tag for Gitlab CI images? I suppose other things could break if the base image is silently upgraded between different pipeline runs, but at least this kind of thing wouldn't happen?

@Ottomata - So, image build workflows in CI that use the latest tag would still have been affected by this, but they would have been transparently fixed when the rebuild happened last night. Indeed, there's a tradeoff w.r.t. reproducibility in using latest, but for something like a minimal base image like this, latest seems like a fairly safe / stable bet.

FTR, went with versioned tag for repeatability.

I'm no longer seeing any references to bullseye-backports in puppet, so I believe Moritz took care of all of those.

Once https://gerrit.wikimedia.org/r/c/operations/puppet/+/1171716 is merged, we'll need to rebuild the bullseye base image, which I think uses the procedure described here, e.g.,

sudo -i
set_proxy
DISTRIBUTIONS="bullseye" build-base-images

Once that's done, I believe we'll need to kick off production-images builds for all (transitively) bullseye-based images. We should prioritize rebuilding the php8.1 images, in order to get deployments out of their current risky state.

In parallel, any CI images that depend on the bullseye base image can also be rebuilt.

We now can't build the CI Quibble images, as we indeed get The repository 'http://mirrors.wikimedia.org/debian bullseye-backports Release' does not have a Release file. as an error during the build. What image are you expecting us to re-build/change to make this work?

@Jdforrester-WMF - Basically, the rebuilds would need to start at the first image that depends on docker-registry.discovery.wmnet/bullseye.

Here, for example, I believe that would mean ci-bullseye -> quibble-bullseye -> [... dependents ...]. I've been chatting with @dancy about this out band, but let me follow up in #wikimedia-releng.

@Jdforrester-WMF I'll do the docker-pkg stuff and pass it by you for review.

Change #1172092 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Docker: [ci-bullseye] Re-build now that bullseye-backports is dead

https://gerrit.wikimedia.org/r/1172092

Change #1172093 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[integration/config@master] dockerfiles: Refresh ci-bullseye and descendants

https://gerrit.wikimedia.org/r/1172093

docker-registry.wikimedia.org/python3-devel:latest is another image that needs a rebuild.

Change #1172092 abandoned by Jforrester:

[integration/config@master] Docker: [ci-bullseye] Re-build now that bullseye-backports is dead

Reason:

Over to Dancy's version.

https://gerrit.wikimedia.org/r/1172092

Change #1172093 merged by jenkins-bot:

[integration/config@master] dockerfiles: Refresh ci-bullseye and descendants

https://gerrit.wikimedia.org/r/1172093

Mentioned in SAL (#wikimedia-releng) [2025-07-23T19:42:47Z] <dancy> Rebuilding ci-buster image and descendants (T383557)

Change #1172111 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[integration/config@master] Update all images references to the latest

https://gerrit.wikimedia.org/r/1172111

Change #1172111 merged by jenkins-bot:

[integration/config@master] jjb: Update all bullseye-based image references to the latest

https://gerrit.wikimedia.org/r/1172111

No further manual clean-up actions are currently planned, though there will be various spot fixes as teams update their build configuration where needed. Thus, I will re-resolve this. Thanks, all!

The image used for debci building (bullseye) is still affected by the bullseye-backports issue:

 ~ > podman run -it --rm docker-registry.wikimedia.org/wmf-debci-bullseye
root@65157c89343f:/# apt-get update
Get:1 http://security.debian.org/debian-security bullseye-security InRelease [27.2 kB]
Get:2 http://security.debian.org/debian-security bullseye-security/main amd64 Packages [387 kB]        
Get:3 http://apt.wikimedia.org/wikimedia bullseye-wikimedia InRelease [153 kB]                         
Get:4 http://mirrors.wikimedia.org/debian bullseye InRelease [116 kB]
Get:5 http://mirrors.wikimedia.org/debian bullseye-updates InRelease [44.0 kB]
Ign:6 http://mirrors.wikimedia.org/debian bullseye-backports InRelease
Get:7 http://apt.wikimedia.org/wikimedia bullseye-wikimedia/main amd64 Packages [74.6 kB]
Get:8 http://mirrors.wikimedia.org/debian bullseye/main amd64 Packages [8066 kB]
Get:9 http://mirrors.wikimedia.org/debian bullseye-updates/main amd64 Packages [18.8 kB]
Err:10 http://mirrors.wikimedia.org/debian bullseye-backports Release
  404  Not Found [IP: 208.80.154.139 80]
Reading package lists... Done                  
E: The repository 'http://mirrors.wikimedia.org/debian bullseye-backports Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
root@65157c89343f:/#

Thanks, @Fabfur - If this is urgent, a manual rebuild of the wmf-debci-bullseye image should resolve it. If it can wait, it should be fixed automatically by this weekend's rebuild.

Thanks a lot @Scott_French ! This weekend is fine, I'll retry on Monday!