A new upstream version of Pygments is now available: 2.18.0.
This release includes updates for the wikitext lexer by @Diskdance:
Previously: T351744: New upstream release for Pygments (2.17.2)
Next: T383018: New upstream release for Pygments (2.19.2)
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Feature | bd808 | T383018 New upstream release for Pygments (2.19.2) | ||
| Resolved | Feature | bd808 | T364249 New upstream release for Pygments (2.18.0) | ||
| Resolved | Scott_French | T377038 Migrate production Shellbox variants to PHP 8.1 | |||
| Resolved | Scott_French | T375243 Turn up PHP 8.1 Shellbox deployments | |||
| Resolved | Scott_French | T374502 Prepare PHP 8.1 service images for Shellbox | |||
| Resolved | Scott_French | T388260 Converge / update image version across shellbox service instances (cleanup) |
Change #1027245 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Update Pygments to 2.18.0
Change #1027251 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[mediawiki/libs/Shellbox@master] syntaxhighlight: Update Pygments to 2.18.0
Pygments v2.18.0 requires Python 3.8+ as a runtime. The current test and default local development environments for MediaWiki are using Debian Buster containers which provide the EOL Python 3.7.3. Until these bump to a newer Debian distro with a newer Python version things are stuck. Debian Bookworm provides Python 3.11.2; Debian Bullseye provides Python 3.9.2.
I am not exactly sure what to mark this as blocked on. Maybe T319432: Migrate WMF production from PHP 7.4 to PHP 8.1? I think that fundamentally this blocked until we bump all the things past Buster to Bullseye or newer. That PHP 8.1 task looks like the same fundamental block.
Change #1032606 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Pygmentize: report stderr when exit code != 0 and stdout is empty
Change #1032606 merged by jenkins-bot:
[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Pygmentize: report stderr when exit code != 0 and stdout is empty
I guess it could be T291916: Tracking task for Bullseye migrations in production, or (if Pygments runs in the MediaWiki image) specifically T356293: Migrate MW appservers' base images to bullseye?
T377038: Migrate production Shellbox variants to PHP 8.1 is our proper blocker:
[16:57] < bd808> Is there a task anywhere in Phabricator for upgrading the base image for the Shellbox containers to bullseye or bookworm? Context is upgrading Pygments for Extension:SyntaxHighlight. We need Python 3.8+ for the T364249 and now T383018 updates. [16:59] < rzl> bd808: I think swfrench-wmf might be looking at that as an aspect of https://phabricator.wikimedia.org/T377038 but I'm not positive [17:00] < rzl> (and we're all heading into a meeting so responses might be slowish) [17:02] <swfrench-wmf> bd808: yes, I'll be bumping them to bullseye (so 3.9) as part of the 8.1 update, and the syntaxhighlight variant is actually the first one I'll be doing (ideally starting this or next week) [17:02] < claime> the php 8.1 version of the shellbox image is bullseye [17:02] < claime> heh 4 seconds too slow
I talked to @Scott_French today and learned that the shellbox deployment for syntaxhighlight is now running on a bullseye base image. This means that I can start working on moving this upgrade forward again.
I need to double check, but I think we still need to make some changes in CI to start that off. I probably should also decide between doing this version bump and T383018: New upstream release for Pygments (2.19.2) in series or combining them to make one jump from 2.17.2 to 2.19.1 in the Wikimedia deployment.
I looked at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/SyntaxHighlight_GeSHi/+/1027245 to remind myself about how this was actually blocked. The issue is that the quibble-vendor-mysql-php74-noselenium test suite fails because it is using the quibble-buster-php74 container image. This would need to move to using the quibble-bullseye-php74 container image to inherit the necessary Python version for running the newer version of pygments.
[16:53] < bd808> hasharAway, James_F: would it be at all reasonable to switch the quibble-vendor-mysql-php74-noselenium tests to use the quibble-bullseye-php74 container instead of the quibble-buster-php74 container? Context is unblocking a pygments upgrade for syntaxhighlight -- https://phabricator.wikimedia.org/T364249#10485277 [16:54] < James_F> bd808: Prod on PHP 7.4 is using the buster container, so moving CI will give you false confidence about deployability. [16:55] < James_F> bd808: We're hoping SRE will finish the 8.1 migration PDQ so we can just burn 7.4 out of reality. [16:55] < bd808> I heard "6 weeks" earlier today [16:56] < James_F> That soon? That'd be awesome. [16:56] < James_F> But that's also a long time to carry the Pygments issue. [16:56] < bd808> The particular code that is being blocked in the syntaxhighlight case is actually running on the needed bullseye base now in prod. It is jsut twisted up in how we test things. [16:56] < James_F> Should we just skip the tests in 7.4? [16:57] < James_F> `if PHP_VERSION < 8000000 … ->markTestSkipped()` etc. [16:57] < bd808> The pygments upgrade has already been blocked since May 2024 so in the big picture another 6 weeks isn't much.
As much as I would like to move faster, I don't think we have any issues that are addressed by the pygments upgrades that would rise to the level of mucking around with the test suite just to get past CI. I think this means we are really still stalled until the PHP 8.1 rollout is complete enough that CI changes to newer base containers.
Change #1027245 merged by jenkins-bot:
[mediawiki/extensions/SyntaxHighlight_GeSHi@master] Update Pygments to 2.18.0
Things are unblocked in the CI pipeline!
[16:09] < James_F> !log Zuul: Drop all PHP 7.4 testing for MediaWiki things, for T328921 and T328922 [16:09] < stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:09] < stashbot> T328921: Drop PHP 7.4 and PHP 8.0 support from MediaWiki - https://phabricator.wikimedia.org/T328921 [16:09] < stashbot> T328922: Drop PHP 8.0 support from MediaWiki - https://phabricator.wikimedia.org/T328922 [16:10] < James_F> It is done!
Change #1027251 merged by jenkins-bot:
[mediawiki/libs/Shellbox@master] syntaxhighlight: Update Pygments to 2.18.0
Change #1154132 had a related patch set uploaded (by BryanDavis; author: Bryan Davis):
[operations/deployment-charts@master] shellbox: Bump to 2025-06-05-215815
Change #1154132 merged by jenkins-bot:
[operations/deployment-charts@master] shellbox-syntaxhighlight: Bump to 2025-06-05-215815
[20:00] < wikibugs> (Merged) jenkins-bot: shellbox-syntaxhighlight: Bump to 2025-06-05-215815 [deployment-charts] - https://gerrit.wikimedia.org/r/1154132 (https://phabricator.wikimedia.org/T364249) (owner: BryanDavis) [20:01] <logmsgbot> !log bd808@deploy1003 helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply [20:02] <logmsgbot> !log bd808@deploy1003 helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [20:09] <logmsgbot> !log bd808@deploy1003 helmfile [codfw] START helmfile.d/services/shellbox-syntaxhighlight: apply [20:11] <logmsgbot> !log bd808@deploy1003 helmfile [codfw] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [20:13] <logmsgbot> !log bd808@deploy1003 helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply [20:14] <logmsgbot> !log bd808@deploy1003 helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply
The version number cache for Pygments on Special:Version should expire in the next hour or so.
Change #1172019 had a related patch set uploaded (by Reedy; author: Bryan Davis):
[mediawiki/extensions/SyntaxHighlight_GeSHi@REL1_43] Update Pygments to 2.18.0
Change #1172020 had a related patch set uploaded (by Reedy; author: Bryan Davis):
[mediawiki/extensions/SyntaxHighlight_GeSHi@REL1_44] Update Pygments to 2.18.0
Change #1172020 merged by jenkins-bot:
[mediawiki/extensions/SyntaxHighlight_GeSHi@REL1_44] Update Pygments to 2.18.0
Change #1172019 merged by jenkins-bot:
[mediawiki/extensions/SyntaxHighlight_GeSHi@REL1_43] Update Pygments to 2.18.0