Mentioned in SAL (#wikimedia-releng) [2025-09-04T14:26:30Z] <Krinkle> Cherry-pick https://gerrit.wikimedia.org/r/1184101/ to Beta Cluster puppetserver, to make PHP 8.3 available. ref T360995

Mentioned in SAL (#wikimedia-releng) [2025-09-04T14:28:08Z] <Krinkle> Install php8.3 on deployment-mwmaint in Beta Cluster. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/14f83cf52c91c64976b13a5e7664b572156163b0 ref T360995

Mentioned in SAL (#wikimedia-releng) [2025-09-04T14:29:16Z] <Krinkle> Install php8.3 on deployment-jobrunner05 in Beta Cluster. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/35c3149ae0fa78cd97a1ab3c45227b8b36232af7 ref T360995

Mentioned in SAL (#wikimedia-releng) [2025-09-04T16:46:24Z] <Krinkle> Install php8.3 on deployment-mediawiki13 and deployment-mediawiki14 in Beta Cluster. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/ec8a68dc394279c574c10a78a4eff75cfcdcefbc%5E%21/#F0 ref T360995

Nothing new in channel:exception or channel:error on "mwmaint" and "jobrunner" hosts in Beta Cluster Logstash after running PHP 8.3 for two hours.

I've also confirmed that Scap is still able to sync and restart to both php81 and php83 hosts side by side, by doing a dummy scap sync-world from the "deploy04" host, which finished clean.

With that, I've rolled forward to the "mediawiki13" and "mediawiki14" hosts. PHP 8.3 is now live at:

https://en.wikipedia.beta.wmcloud.org/wiki/Special:Version

In T360995#11070981, @Krinkle wrote:

In T360995#11070285, @Jdforrester-WMF wrote:

T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Yep, that's exactly what this is. It is post-rollout of the PHP upgrade, not post-release.

No, you have mis-understood me. I'm saying it's a blocker to production release, not third-party release, as it will break 2FA workflows in production, and must be done before you start rolling out PHP 8.3 to logged-in users in production.

From https://wikitech.wikimedia.org/wiki/PHP_upgrade_process_for_MediaWiki#Post-rollout

Post-rollout

• before the next MW release branch cut, but after the rollout
  • Raise minimum PHP version
  • Upgrade vendor dependencies
• after the rollout, and after the MW release branch cut
  • …

The current symfony versions in vendor do work on PHP 8.1 and 8.3, and are thus not a hard blocker for rollout at WMF, given that MediaWiki indeed already officially supports PHP 8.1-8.3 https://www.mediawiki.org/wiki/Compatibility#PHP.

MediaWiki does except for this code, which does not pass our tests and I am told does not work in practice either on PHP 8.3

In T360995#11152911, @Jdforrester-WMF wrote:

In T360995#11070981, @Krinkle wrote:

In T360995#11070285, @Jdforrester-WMF wrote:

T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Yep, that's exactly what this is. It is post-rollout of the PHP upgrade, not post-release.

No, you have mis-understood me. I'm saying it's a blocker to production release, […], as it will break 2FA workflows […]

I see. You're referring to the WebAuthn upgrade (T363639). The WebAuthn upgrade is indeed a blocker for PHP 8.3 rollout to mw-web at WMF, and is prioritized as such. I was referring to the symfony upgrade (T396296).

In T360995#11157715, @Krinkle wrote:

In T360995#11152911, @Jdforrester-WMF wrote:

In T360995#11070981, @Krinkle wrote:

In T360995#11070285, @Jdforrester-WMF wrote:

T396296: Upgrade symfony/* to PHP 8.1 versions blocked on T363639

This is a blocker to release, it's not a post-release clean-up.

Yep, that's exactly what this is. It is post-rollout of the PHP upgrade, not post-release.

No, you have mis-understood me. I'm saying it's a blocker to production release, […], as it will break 2FA workflows […]

I see. You're referring to the WebAuthn upgrade (T363639). The WebAuthn upgrade is indeed a blocker for PHP 8.3 rollout to mw-web at WMF, and is prioritized as such. I was referring to the symfony upgrade (T396296).

Oh, right, yes, the symfony upgrade is not a blocker to production, just a nice-to-have. My point was that marking:

4. MW: Make CI for MediaWiki core and WMF extensions pass on PHP 8.3. T353362, Jun 2024

… as complete is wrong.

Mentioned in SAL (#wikimedia-releng) [2025-09-09T16:24:12Z] <Krinkle> Remove unused deployment-snapshot- prefix Hiera file. https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/0c6a99c4e12e9d1a55ce353e302d7dc891ae1177 - Follows-up decom of deployment-snapshot05 host. ref T398438, T360995

Progress report for 18 Aug - 5 Sep (three weeks) on WE6.4.5 Unblock PHP 8.3 upgrade (m:FY2025-2026#Q1).

Scott (SRE ServiceOps) published a "next" container to WikimediaDebug, where MediaWiki runs on PHP 8.3, and is now publicly available for testing in your browser.

We carried out exploratory testing in the "next" environment for CentralAuth, ResourceLoader, REST API, Revision deletion, Parsoid, Kartographer, and other components owned by MediaWiki Platform, MediaWiki Interfaces, and Content Transform teams (T402597, T402809, T402810).

Scott (SRE ServiceOps) and Timo decided on how to upgrade the Beta Cluster servers, and the parsoidtest production bare-metal server (T403772). This required a separate mini-plan because both of these servers do not run on Kubernetes yet.

Scott implemented the plan through a Puppet profile suitable for both bare metal and Beta Clusters outside Kubernetes. Timo deployed the patch to the Beta Cluster, and upgraded the various Beta Cluster servers over a 24-hour period. The Beta rollout helped us find and fix an issue with the php-json installation.

Monitoring on the Beta Cluster yielded no PHP 8.3 issues so far. From a monitoring perspective, we're ready for the production rollout to begin. Note: We have one known blocker remaining outside the scope of monitoring (T363639).

The value and confidence of testing on the Beta Cluster is generally reduced at the moment. While not specific to the PHP 8.3 upgrade, this means the value of monitoring the PHP 8.3 rollout is also impacted. Pre-existing issues mean that certain features are failing early, or execute differently than they otherwise would, and thus don't cover key business logic.

These pre-existing issues include:

• MainStash DB fatal error (T401227), which I decided to fix myself.
• AutoModerator config ignored (T403756), which hadn't been reported to Phabricator. Once I did, the ModTools team promptly fixed this. Thank you!
• PoolCounter down (T380881), and remains down.
• PCS lacks rest-gateway routing (T404387), which I've started porting to Beta based on rest-gateway in prod (work in progress).

Progress report for 8 Sep - 19 Sep (two weeks) on WE6.4.5 Unblock PHP 8.3 upgrade (m:FY2025-2026#Q1).

Gergo (MediaWiki Platform) along with Bartosz (MediaWiki Platform) and Sam Reed (Security) are upgrading webauthn-lib from v3 to v4. This package powers MediaWiki's support for logging in with YubiKey devices, through the WebAuthn standard. This upgrade changes key encoding formats and adds conflicting dependency requirements that we untangled this week. This in turn is a prerequisite for other PHP package upgrades (psr/http, and brick/math) before the MediaWiki 1.45 release branch cuts on 1 November 2025 (T363639).

No update on monitoring, as there is no traffic yet.

Meanwhile, following the previous status update, I've made general Beta Cluster improvements to make testing more valuable (for PHP 8.3, M-dot, and anything else at WMF):

• Implement the missing rest-gateway router for the Beta Cluster (T404387). This allows PCS and other services to work without RESTBase, a necessary part of RESTBase sunsetting.
• PoolCounter down for 12 months, and remains down (T380881).

Progress report for 13 Oct - 24 Oct (two weeks) on WE6.4.8 Support PHP 8.3 upgrade (m:FY2025-2026#Q2).

Scott (SRE ServiceOps) started the PHP 8.3 rollout in production on Tue 14 Oct, by enabling it for 1% of traffic on the mw-web and mw-api-ext server groups. This was rolled back on Wednesday due to a new error from the GrowthExperiments extension, when unserialising cached DatePeriod objects across different versions (T407403).

The bug was triaged and patched on Fri 17 Oct by the Growth Team, code reviewed and backported on Thu 23 Oct by Timo. Scott re-rolled PHP 8.3 to 1% later that same day (T405955#11262893).

Meanwhile, following the previous status update, Timo created a new PoolCounter server in Beta Cluster for MediaWiki. This had been down for 12 months (T380881). Finishing this improves Beta Cluster test coverage, increases value of all other testing efforts, and reduces log noise to ease triage and discovery of regressions.

Change #1204908 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Zuul: Switch wmf/* PHP testing from PHP 8.1 to PHP 8.3

https://gerrit.wikimedia.org/r/1204908

Change #1204908 merged by jenkins-bot:

[integration/config@master] Zuul: Switch wmf/* PHP testing from PHP 8.1 to PHP 8.3

https://gerrit.wikimedia.org/r/1204908

Mentioned in SAL (#wikimedia-releng) [2025-11-13T18:32:07Z] <James_F> Zuul: Switch wmf/* PHP testing from PHP 8.1 to PHP 8.3, for T360995

Progress report for 27 Oct - 24 Nov (four weeks) on WE6.4.8 Support PHP 8.3 upgrade (m:FY2025-2026#Q2).

Following the first 1% rollout of PHP 8.3 on Oct 14, Timo started monitoring and addressing log warnings. After two weeks, we set up a new rotation across the three MediaWiki Engineering teams to delegate this to (T401855).

Roll out by Scott French and monitoring by MwEng progressed in lockstep over the course of November without any blockers. On Nov 3, we reached 100% of cookie traffic on PHP 8.3 for the main mw-web and mw-api-ext server groups. MediaWiki JobQueue was migrated on Nov 4. Remaining cookieless traffic reached 100% on PHP 8.3 by Nov 12, and misc servers were migrated by Nov 25. (For a detailed breakdown, see task description of T360995 and T405955.)

After the rollout completed on Nov 25, work started on the post-rollout steps by James Forrester, Sam Reed, Zabe, and Timo (tracked under sub task T358666). This includes:

• CI: Change PHP test matrix jobs for MediaWiki to use PHP 8.3 by default instead of PHP 8.1.
• CI: Migrate MediaWiki jobs from PHP 8.1 to PHP 8.3 (PHPUnit coverage check, Phan static analysis, Fresnel performance check, etc).
• CI: Remove PHP 8.1 jobs from MediaWiki CI gate-and-submit pipeline for master and REL1_45 branch.
• MW: Raise runtime requirement in MediaWiki core from PHP 8.1 to PHP 8.3.
• MW: Backport new runtime requirement to MediaWiki 1.45 release candidate (because we missed the branch cut).

While the checklist did account for some environments outside production, such as the Beta Cluster's MediaWiki servers, and CI, and MediaWiki local dev environments; it did not account for Catalyst/Patchdemo or certain Scap commands run in the Beta Cluster. Those broke for a few hours (T411277, T411235). Big thanks to Jaime Nuche and Bryan Davis for fixing those on a Friday!

Once the project is done, Timo and Scott will revise the checklist to reflect what we did in practice.

Proposed changes for the checklist so far:

• (diff) In the Test phase, swap "Validation and end-to-end testing" and "Monitoring and fixing".
  • This recognises the hidden work people were doing before we officially started, because once the next PHP version is on WikimediaDebug and people start doing end-to-end-testing, it is inevitable that production errors will be filed, and we respond to those.
• (diff) Move "Beta Cluster switch" from Rollout to Preparation phase, and add "Create production images for PHP X.Y and add new flavour to MediaWiki deployments" to Preparation phase.
  • This reflects what we actually did.
  • It was essential for the Test phase to have Beta Cluster upgraded first (not just WikimediaDebug "next") in order to test MediaWiki JobQueue (jobrunner), MediaWiki Maintenance scripts (mw-cron), and Scap itself.
  • By moving Beta Cluster to the prep, the prep phase includes both k8s prep and baremetal prep (because Beta Cluster acts as baremetal), so that rollout phase can (in theory) be just rolling out, and not involve unplanned work to write Puppet provisioning for baremetal hosts. To that end, I've also made a separate bullet point for creating production images for mw-on-k8s. This was previously implied somewhere between "Deploy title-case mapping" and "Switch mw-debug/next", but seems worth making its own step.
• (diff) In the Rollout phase, make inclusion of Dumps 1.0 (mw-on-k8s servergroup "mediawiki-dumps-legacy") and deployment server explicit.
  • This reflects what we actually did.

Any thoughts yet on what version of PHP we should target next? 8.4 or 8.5? Picking this would let us create the next ticket in the series, since this one is close to wrapping up.

In T360995#11437108, @Krinkle wrote:

Proposed changes for the checklist so far:

• (diff) In the Test phase, swap "Validation and end-to-end testing" and "Monitoring and fixing". […]
• (diff) Move "Beta Cluster switch" from Rollout to Preparation phase, and add "Create production images for PHP X.Y and add new flavour to MediaWiki deployments" to Preparation phase. […]
• (diff) In the Rollout phase, make inclusion of Dumps 1.0 (mw-on-k8s servergroup "mediawiki-dumps-legacy") and deployment server explicit. […]

Two more revisions:

• (diff) Add step for switching Scap to use PHP X.Y images for mwscript, and upgrade Scap in Beta Cluster and production. This avoids Beta Cluster breaking like it did in T411235.
• (diff) Add step for switching Patchdemo to PHP X.Y to the prep phase. This would be right after upgrading Beta Cluster, and avoids Patchdemo breaking again, ref T411277.

In T360995#11467234, @Novem_Linguae wrote:

Any thoughts yet on what version of PHP we should target next? 8.4 or 8.5? Picking this would let us create the next ticket in the series, since this one is close to wrapping up.

Not yet. I've created T413223: Migrate Wikimedia production from PHP 8.3 to PHP 8.5 so we can discuss this there.

Final report for WE6.4.8 Support PHP 8.3 upgrade (m:FY2025-2026#Q2).

Hypothesis (task: T360995)

If the MediaWiki Engineering Team monitors and fixes issues in MediaWiki related to the PHP upgrade, this will enable the SRE team to complete the PHP 8.3 upgrade by November 2025.

Was this hypothesis supported or contradicted?

Supported. SRE completed the PHP 8.3 rollout on 25 November, and closed their WE6.4.6 hypothesis on 5 December.

Briefly describe what was accomplished.

• Monitoring to find and report production errors post-upgrade relating to PHP 8.3.
• Investigate and fix unowned MediaWiki components' production errors.
• Help other teams with understanding and fixing PHP 8.3 issues.
• Provision and upgrade gaps in Beta Cluster infrastructure.
• Migrate dozens of MediaWiki CI jobs from PHP 8.1 to PHP 8.3 including Quibble, PHPUnit coverage, Phan, and Fresnel (credit to James F for most of this).
• Drop PHP 8.1 support from MediaWiki and upgrade outdated vendor packages ahead of the MediaWiki 1.45 release. Then re-do it for the MW 1.45 branch, because the PHP upgrade completed several weeks after the MW 1.45 branch cut (credit to Sam Reed for much of this).
• Revise checklist to incorporate what we learned (details).

What are the key lessons from this hypothesis?

• The checklist on Wikitech is effective in conveying to everyone what each task is for, who should do it, and giving visibility in what work is left.
• The monitoring work is something most engineers in MediaWiki Engineering can do. We experimented with this after the 10th week in the T401855 rotation.
• Alignment between PHP upgrade and MediaWiki release was a success. We are now caught up such that early in the year MediaWiki releases in May can comfortably add support for the next PHP version early (MW 1.46 in May 2026 adds support PHP 8.4 and 8.5), and late in the year WMF can upgrade to that next version while MW 1.47 in November 2026 potentially drops PHP 8.2 without being blocked by the upgrade (because it would go from PHP 8.3 to PHP 8.5, not involving the version being dropped).
• We can complete an upgrade like this in two quarters with ~1 SRE and ~1 MwEng person. However, Scott and Timo were both reliant on unplanned time from peers for code review, and any PTO or other demands for their time halted progress.

I've made a handful of (mostly minor) additional revisions to the SRE-driven items in the checklist:

• (diff) Preparation for WMF: Extend SRE tasks with 8.3 migration examples
• (diff) Preparation for WMF: Add note about coordination with MW for PHP extension versions
  • This makes explicit that although SRE drives the package build process, there is some amount of coordination with MediaWiki Engineering to review the selected PHP extension versions (and more rarely, assist with migrating to appropriate alternatives when an extension is no longer supported, as in the case of tideways this time around).
• (diff) Preparation for WMF: Explicitly refer to maintenance scripts to avoid possible confusion with mwscript
• (diff) Rollout: Add context on rollout sequence, references to T405955 and hypothetical schedule sheet
  • This adds a bit more detail on the high-level structure of the production rollout and links example artifacts from the 8.3 migration (e.g., the rollout task that reflects "what happened" and a sheet containing a hypothetical progressive rollout schedule).
• (diff) Post-rollout: Add previous examples for SRE items and note that title-case mapping cleanup may skipped if none was necessary
• (diff) Preparation for WMF: Add note that title-case mapping may not be needed in some cases

Many thanks for @Krinkle for creating an excellent checklist.