Change #1295934 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[mediawiki/libs/Shellbox@master] Rebuild images following base image upgrade

https://gerrit.wikimedia.org/r/1295934

Alas, an attempt to deploy newly built Shellbox images (i.e., reflecting all code changes through v4.5.0) on Tuesday surfaced T428013. Unless that's resolved in the interim, we'll need to use a different approach to trigger image builds after the php8.3 production images switch to bookworm. For that, I've found the job-replay approach described in Shellbox#Deploying_a_new_version to work reasonably well.

The current live images (2026-05-20-192555) were produced using this approach, replaying the postmerge job from https://gerrit.wikimedia.org/r/c/mediawiki/libs/Shellbox/+/1264727 (actually the second round thereof, as you'll see in the PipelineBot comments on the patch).

Change #1295934 abandoned by Kamila Součková:

[mediawiki/libs/Shellbox@master] Rebuild images following base image upgrade

Reason:

we will use the replay pipeline approach instead of this

https://gerrit.wikimedia.org/r/1295934

Following up on one item we discussed today:

Starting from a local build of the bookworm-based php8.3 image stack in https://gerrit.wikimedia.org/r/1295044, I rebuilt all Shellbox variants and dumped the installed versions of debian packages explicitly installed in blubber.yaml (i.e., this does not include their dependencies), for comparison with the same packages installed on bullseye.

The idea is to give us a basic (if qualitative, without digging extensively into changelogs) sense of which variants may see the largest jumps in package versions, and thus potential compatibility issues.

The results appear below. There are a handful of what appear to be major version bumps (e.g., libfluidsynth2 -> libfluidsynth3) that may suggest higher compatibility risks, but there are others do not (e.g., I believe we've used ffpmeg 5 for videoscaling previously without issue, back when we were on buster and manually backporting).

In any case, I think this is useful as a prior for which variants we should pay particularly close attention to exec-channel errors / warnings (e.g.., score is perhaps more likely to encounter issues than syntaxhighlight or probably timeline).

Thank you Scott, that's super useful!

Below is a shortened version with only the ones I intend to pay extra attention to when doing the migration (extra extra attention in bold).

Variant	Package	Bullseye	Bookworm
media83	libtiff-tools	4.2.0-1+deb11u8	4.5.0-6+deb12u4
media83	poppler-utils	20.09.0-3.1+deb11u2	22.12.0-2+deb12u1
score83	fluidsynth	2.1.7-1.1	2.3.1-2
score83	ghostscript	9.53.3~dfsg-7+deb11u11	10.0.0~dfsg-11+deb12u8
score83	libfluidsynth2	2.1.7-1.1	-
score83	libfluidsynth3	-	2.3.1-2
score83	lilypond	2.22.0-10	2.24.1-2
syntaxhighlight83	python3-venv	3.9.2-3	3.11.2-1+b1
timeline83	librsvg2-bin	2.50.3+dfsg-1+deb11u1	2.54.7+dfsg-1~deb12u1
timeline83	perl-modules-5.32	5.32.1-4+deb11u5	-
timeline83	perl-modules-5.36	-	5.36.0-7+deb12u3
timeline83	perl	5.32.1-4+deb11u5	5.36.0-7+deb12u3
video83	ffmpeg	7:4.3.9-0+deb11u2	7:5.1.9-0+deb12u1
video83	fluidsynth	2.1.7-1.1	2.3.1-2
video83	libfluidsynth2	2.1.7-1.1	-
video83	libfluidsynth3	-	2.3.1-2

Change #1298729 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/deployment-charts@master] shellbox: switch to bookworm

https://gerrit.wikimedia.org/r/1298729

Change #1298729 merged by jenkins-bot:

[operations/deployment-charts@master] shellbox: switch to bookworm

https://gerrit.wikimedia.org/r/1298729

Seems to have caused T428484 regression of Score extension (no music scores are shown any more).

https://github.com/lilypond/lilypond/releases/tag/v2.23.12 is from 2022

Wrong task, never mind.

Do we consider the migration to the bookworm shellbox images, at least for Score, to be stable?

Not that we consider lilyponds safemode to be giving much security vs Shellbox...

Change #1299436 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/deployment-charts@master] shellbox: revert score image version to bullseye

https://gerrit.wikimedia.org/r/1299436

Reverting shellbox-score to bullseye to stop the bleeding, but we should fix this properly, as new images will be built against bookworm.

@Raine JFYI but T428484 was essentially resolved by setting $wgScoreSafeMode variable to false, so maybe a revert isn’t too necessary? In the config docs it says this about it:

Must be set to false for LilyPond versions 2.23.12 or newer (safe mode was removed in 2.23.12 as it could no longer be guaranteed safe). For older versions, set this to true to enable LilyPond's safe mode (default). NOTE: if set to false, all inputs should be sanitized and/or trusted, and the LilyPond binary must be appropriately sandboxed.

So maybe someone should just advise you on whether having safe mode is functionally different in Wikimedia setup from not having it (and whether it’s not just a result of leftover config from before Shellbox days).

Change #1299436 abandoned by Kamila Součková:

[operations/deployment-charts@master] shellbox: revert score image version to bullseye

Reason:

not needed, Reedy already fixed score

https://gerrit.wikimedia.org/r/1299436

In T427820#11998321, @stjn wrote:

@Raine JFYI but T428484 was essentially resolved by setting $wgScoreSafeMode variable to false, so maybe a revert isn’t too necessary? In the config docs it says this about it:

Must be set to false for LilyPond versions 2.23.12 or newer (safe mode was removed in 2.23.12 as it could no longer be guaranteed safe). For older versions, set this to true to enable LilyPond's safe mode (default). NOTE: if set to false, all inputs should be sanitized and/or trusted, and the LilyPond binary must be appropriately sandboxed.

So maybe someone should just advise you on whether having safe mode is functionally different in Wikimedia setup from not having it (and whether it’s not just a result of leftover config from before Shellbox days).

Yes, it appears that the $wgScoreSafeMode indeed fixed it and was a leftover. Thanks for connecting the two!

Other than the hiccup in T428484, this appears to have gone smoothly (knocks on wood due to bad visibility into shellbox). Some remaining funkiness:

1. small performance regression in syntaxhighlight

Mean latency went up by a few ms, and there is a corresponding slight increase in busy workers percentage.

https://grafana.wikimedia.org/dashboard/snapshot/0C4KTP1B5xybUyqELhU85Nc2GrnOTAFc

I do not think this is significant enough to justify looking deeper into it at this point, so consider this just mentioned for posterity.

2. substantial(?) performance regression in score

After deploying the image change around 17:15Z, shellbox-score was silently broken due to a change in Lilypond. Then at 19:52, a MW config change to remedy this was rolled out. The latency and number of requests substantially increased after this, with latency's p95 flat at 2.5s.

[edited with new info]
The flat p95 and the increased number of requests hints to me at a performance regression severe enough that we started hitting a lot of timeouts. Therefore, this part needs further investigation (or at least increasing the timeout).

Correction: The 2.5s is the max histogram bucket. I will track this work in T428904.