T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Is there any particular technical reasons for picking PHP 8.3, instead of PHP 8.2 which is available in Debian stable, or PHP 8.4 which will be available in next Debian stable?

In T360995#11070703, @valerio.bozzolan wrote:

Is there any particular technical reasons for picking PHP 8.3, instead of PHP 8.2 which is available in Debian stable, or PHP 8.4 which will be available in next Debian stable?

The plan is to make PHP upgrades smooth and more frequent so we can keep up with active support in the future and ideally we will always pickup the most recent version when we are ready for the upgrade. The operational infrastructure we have built will allow us to move forward yearly in sync with the November/December MediaWiki Release.

The effort to prepare PHP 8.4, due to the ecosystem complexity, doesn't make it a candidate for the current upgrade cycle and PHP 8.3 was chosen to reduce the risks of running on a version (8.2) which would become EOL very close to our next upgrade cycle.

In T360995#11070285, @Jdforrester-WMF wrote:

T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Yep, that's exactly what this is. It is post-rollout of the PHP upgrade, not post-release.

From https://wikitech.wikimedia.org/wiki/PHP_upgrade_process_for_MediaWiki#Post-rollout

Post-rollout

• before the next MW release branch cut, but after the rollout
  • Raise minimum PHP version
  • Upgrade vendor dependencies
• after the rollout, and after the MW release branch cut
  • …

The current symfony versions in vendor do work on PHP 8.1 and 8.3, and are thus not a hard blocker for rollout at WMF, given that MediaWiki indeed already officially supports PHP 8.1-8.3 https://www.mediawiki.org/wiki/Compatibility#PHP.

Any item can usually happen earlier as well. And for this particular sub-item, it will indeed have to happen before the rollout even (so likely months before the release), because the current webauthn-lib version won't work cleanly on PHP 8.3, and that in turn blocks the symfony update.

However, if nothing else, we want the MW release to have clean and updated dependencies, so that's where the generic checkpoint is in the checklist.

In T360995#11070703, @valerio.bozzolan wrote:

Is there any particular technical reasons for picking PHP 8.3, instead of PHP 8.2 which is available in Debian stable, or PHP 8.4 which will be available in next Debian stable?

Note that MediaWiki itself already supports PHP 8.1, 8.2, 8.3, and soon 8.4. What you run is up to you. And we indeed look at Debian and other LTS programs to anchor our support milestones (mw:Support_policy_for_PHP and mw:Support_policy_for_PHP/Tables).

This task is specifically about WMF upgrading from PHP 8.1. At WMF we maintain our own apt component with packages, so it doesn't matter too much what version we pick. Newer is preferred I think, as it deploys perf improvements sooner, and allows better collab with upstream php.net, as they're not yet in the maintenance/security-only phase so we can actually report stuff and have things fixed quickly!

Mentioned in SAL (#wikimedia-releng) [2025-09-04T14:26:30Z] <Krinkle> Cherry-pick https://gerrit.wikimedia.org/r/1184101/ to Beta Cluster puppetserver, to make PHP 8.3 available. ref T360995

Mentioned in SAL (#wikimedia-releng) [2025-09-04T14:28:08Z] <Krinkle> Install php8.3 on deployment-mwmaint in Beta Cluster. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/14f83cf52c91c64976b13a5e7664b572156163b0 ref T360995

Mentioned in SAL (#wikimedia-releng) [2025-09-04T14:29:16Z] <Krinkle> Install php8.3 on deployment-jobrunner05 in Beta Cluster. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/35c3149ae0fa78cd97a1ab3c45227b8b36232af7 ref T360995

Mentioned in SAL (#wikimedia-releng) [2025-09-04T16:46:24Z] <Krinkle> Install php8.3 on deployment-mediawiki13 and deployment-mediawiki14 in Beta Cluster. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ec8a68dc394279c574c10a78a4eff75cfcdcefbc%5E%21/#F0 ref T360995

Nothing new in channel:exception or channel:error on "mwmaint" and "jobrunner" hosts in Beta Cluster Logstash after running PHP 8.3 for two hours.

I've also confirmed that Scap is still able to sync and restart to both php81 and php83 hosts side by side, by doing a dummy scap sync-world from the "deploy04" host, which finished clean.

With that, I've rolled forward to the "mediawiki13" and "mediawiki14" hosts. PHP 8.3 is now live at:

https://en.wikipedia.beta.wmcloud.org/wiki/Special:Version

In T360995#11070981, @Krinkle wrote:

In T360995#11070285, @Jdforrester-WMF wrote:

T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Yep, that's exactly what this is. It is post-rollout of the PHP upgrade, not post-release.

No, you have mis-understood me. I'm saying it's a blocker to production release, not third-party release, as it will break 2FA workflows in production, and must be done before you start rolling out PHP 8.3 to logged-in users in production.

From https://wikitech.wikimedia.org/wiki/PHP_upgrade_process_for_MediaWiki#Post-rollout

Post-rollout

• before the next MW release branch cut, but after the rollout
  • Raise minimum PHP version
  • Upgrade vendor dependencies
• after the rollout, and after the MW release branch cut
  • …

The current symfony versions in vendor do work on PHP 8.1 and 8.3, and are thus not a hard blocker for rollout at WMF, given that MediaWiki indeed already officially supports PHP 8.1-8.3 https://www.mediawiki.org/wiki/Compatibility#PHP.

MediaWiki does except for this code, which does not pass our tests and I am told does not work in practice either on PHP 8.3

In T360995#11152911, @Jdforrester-WMF wrote:

In T360995#11070981, @Krinkle wrote:

In T360995#11070285, @Jdforrester-WMF wrote:

T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Yep, that's exactly what this is. It is post-rollout of the PHP upgrade, not post-release.

No, you have mis-understood me. I'm saying it's a blocker to production release, […], as it will break 2FA workflows […]

I see. You're referring to the WebAuthn upgrade (T363639). The WebAuthn upgrade is indeed a blocker for PHP 8.3 rollout to mw-web at WMF, and is prioritized as such. I was referring to the symfony upgrade (T396296).

In T360995#11157715, @Krinkle wrote:

In T360995#11152911, @Jdforrester-WMF wrote:

In T360995#11070981, @Krinkle wrote:

In T360995#11070285, @Jdforrester-WMF wrote:

T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Yep, that's exactly what this is. It is post-rollout of the PHP upgrade, not post-release.

No, you have mis-understood me. I'm saying it's a blocker to production release, […], as it will break 2FA workflows […]

I see. You're referring to the WebAuthn upgrade (T363639). The WebAuthn upgrade is indeed a blocker for PHP 8.3 rollout to mw-web at WMF, and is prioritized as such. I was referring to the symfony upgrade (T396296).

Oh, right, yes, the symfony upgrade is not a blocker to production, just a nice-to-have. My point was that marking:

4. MW: Make CI for MediaWiki core and WMF extensions pass on PHP 8.3. T353362, Jun 2024

… as complete is wrong.

Mentioned in SAL (#wikimedia-releng) [2025-09-09T16:24:12Z] <Krinkle> Remove unused deployment-snapshot- prefix Hiera file. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/0c6a99c4e12e9d1a55ce353e302d7dc891ae1177 - Follows-up decom of deployment-snapshot05 host. ref T398438, T360995

Progress report for 18 Aug - 5 Sep (three weeks) on WE6.4.5 Unblock PHP 8.3 upgrade (m:FY2025-2026#Q1).

Scott (SRE ServiceOps) published a "next" container to WikimediaDebug, where MediaWiki runs on PHP 8.3, and is now publicly available for testing in your browser.

We carried out exploratory testing in the "next" environment for CentralAuth, ResourceLoader, REST API, Revision deletion, Parsoid, Kartographer, and other components owned by MediaWiki Platform, MediaWiki Interfaces, and Content Transform teams (T402597, T402809, T402810).

Scott (SRE ServiceOps) and Timo decided on how to upgrade the Beta Cluster servers, and the parsoidtest production bare-metal server (T403772). This required a separate mini-plan because both of these servers do not run on Kubernetes yet.

Scott implemented the plan through a Puppet profile suitable for both bare metal and Beta Clusters outside Kubernetes. Timo deployed the patch to the Beta Cluster, and upgraded the various Beta Cluster servers over a 24-hour period. The Beta rollout helped us find and fix an issue with the php-json installation.

Monitoring on the Beta Cluster yielded no PHP 8.3 issues so far. From a monitoring perspective, we're ready for the production rollout to begin. Note: We have one known blocker remaining outside the scope of monitoring (T363639).

The value and confidence of testing on the Beta Cluster is generally reduced at the moment. While not specific to the PHP 8.3 upgrade, this means the value of monitoring the PHP 8.3 rollout is also impacted. Pre-existing issues mean that certain features are failing early, or execute differently than they otherwise would, and thus don't cover key business logic.

These pre-existing issues include:

• MainStash DB fatal error (T401227), which I decided to fix myself.
• AutoModerator config ignored (T403756), which hadn't been reported to Phabricator. Once I did, the ModTools team promptly fixed this. Thank you!
• PoolCounter down (T380881), and remains down.
• PCS lacks rest-gateway routing (T404387), which I've started porting to Beta based on rest-gateway in prod (work in progress).

Progress report for 8 Sep - 19 Sep (two weeks) on WE6.4.5 Unblock PHP 8.3 upgrade (m:FY2025-2026#Q1).

Gergo (MediaWiki Platform) along with Bartosz (MediaWiki Platform) and Sam Reed (Security) are upgrading webauthn-lib from v3 to v4. This package powers MediaWiki's support for logging in with YubiKey devices, through the WebAuthn standard. This upgrade changes key encoding formats and adds conflicting dependency requirements that we untangled this week. This in turn is a prerequisite for other PHP package upgrades (psr/http, and brick/math) before the MediaWiki 1.45 release branch cuts on 1 November 2025 (T363639).

No update on monitoring, as there is no traffic yet.

Meanwhile, following the previous status update, I've made general Beta Cluster improvements to make testing more valuable (for PHP 8.3, M-dot, and anything else at WMF):

• Implement the missing rest-gateway router for the Beta Cluster (T404387). This allows PCS and other services to work without RESTBase, a necessary part of RESTBase sunsetting.
• PoolCounter down for 12 months, and remains down (T380881).

Progress report for 13 Oct - 24 Oct (two weeks) on WE6.4.8 Support PHP 8.3 upgrade (m:FY2025-2026#Q2).

Scott (SRE ServiceOps) started the PHP 8.3 rollout in production on Tue 14 Oct, by enabling it for 1% of traffic on the mw-web and mw-api-ext server groups. This was rolled back on Wednesday due to a new error from the GrowthExperiments extension, when unserialising cached DatePeriod objects across different versions (T407403).

The bug was triaged and patched on Fri 17 Oct by the Growth Team, code reviewed and backported on Thu 23 Oct by Timo. Scott re-rolled PHP 8.3 to 1% later that same day (T405955#11262893).

Meanwhile, following the previous status update, Timo created a new PoolCounter server in Beta Cluster for MediaWiki. This had been down for 12 months (T380881). Finishing this improves Beta Cluster test coverage, increases value of all other testing efforts, and reduces log noise to ease triage and discovery of regressions.

Change #1204908 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Zuul: Switch wmf/* PHP testing from PHP 8.1 to PHP 8.3

https://gerrit.wikimedia.org/r/1204908

Change #1204908 merged by jenkins-bot:

[integration/config@master] Zuul: Switch wmf/* PHP testing from PHP 8.1 to PHP 8.3

https://gerrit.wikimedia.org/r/1204908

Mentioned in SAL (#wikimedia-releng) [2025-11-13T18:32:07Z] <James_F> Zuul: Switch wmf/* PHP testing from PHP 8.1 to PHP 8.3, for T360995

Progress report for 27 Oct - 24 Nov (four weeks) on WE6.4.8 Support PHP 8.3 upgrade (m:FY2025-2026#Q2).

Following the first 1% rollout of PHP 8.3 on Oct 14, Timo started monitoring and addressing log warnings. After two weeks, we set up a new rotation across the three MediaWiki Engineering teams to delegate this to (T401855).

Roll out by Scott French and monitoring by MwEng progressed in lockstep over the course of November without any blockers. On Nov 3, we reached 100% of cookie traffic on PHP 8.3 for the main mw-web and mw-api-ext server groups. MediaWiki JobQueue was migrated on Nov 4. Remaining cookieless traffic reached 100% on PHP 8.3 by Nov 12, and misc servers were migrated by Nov 25. (For a detailed breakdown, see task description of T360995 and T405955.)

After the rollout completed on Nov 25, work started on the post-rollout steps by James Forrester, Sam Reed, Zabe, and Timo (tracked under sub task T358666). This includes:

• CI: Change PHP test matrix jobs for MediaWiki to use PHP 8.3 by default instead of PHP 8.1.
• CI: Migrate MediaWiki jobs from PHP 8.1 to PHP 8.3 (PHPUnit coverage check, Phan static analysis, Fresnel performance check, etc).
• CI: Remove PHP 8.1 jobs from MediaWiki CI gate-and-submit pipeline for master and REL1_45 branch.
• MW: Raise runtime requirement in MediaWiki core from PHP 8.1 to PHP 8.3.
• MW: Backport new runtime requirement to MediaWiki 1.45 release candidate (because we missed the branch cut).

While the checklist did account for some environments outside production, such as the Beta Cluster's MediaWiki servers, and CI, and MediaWiki local dev environments; it did not account for Catalyst/Patchdemo or certain Scap commands run in the Beta Cluster. Those broke for a few hours (T411277, T411235). Big thanks to Jaime Nuche and Bryan Davis for fixing those on a Friday!

Once the project is done, Timo and Scott will revise the checklist to reflect what we did in practice.