Page MenuHomePhabricator

Migrate WMF production from PHP 7.4 to PHP 8.1
Open, Stalled, MediumPublic

Assigned To
Authored By
Krinkle
Oct 5 2022, 3:28 PM
Referenced Files
F53712686: 2023_php_EOL_Composer.png
May 18 2024, 3:17 PM
F53712005: 2023_php_EOL_wmf.png
May 18 2024, 3:17 PM
Tokens
"Love" token, awarded by Michael."Love" token, awarded by hashar."Yellow Medal" token, awarded by kostajh.

Description

T271736: Migrate WMF production from PHP 7.2 to PHP 7.4 | T360995: Migrate Wikimedia production from PHP 8.1 to PHP 8.3

The next target for WMF production after PHP 7.4, is PHP 8.1. This will incorporate:

  • Assurance that this should work
    • CI configured to require PHP 8.1 to pass for all production code
      • Making CI passing and enforced ("voting") for MediaWiki core on PHP 8.0 and PHP 8.1.
      • Making CI passing wmf-deployed MediaWiki extensions and skins on PHP 8.0 and PHP 8.1.
    • Other tooling changes as needed
  • Preparation of Wikimedia production
    • Provision PHP 8.1 in production via puppet and docker (including any ports or changes needed for php extension packages such as php-apcu, php-luasandbox, php-excimer, etc.)
    • Configure MW with ICU emulation for PHP 8.1 that matches PHP 7.4 (see also: T263437, T292552).
  • Pre-launch testing
    • Benchmarks for different workloads and different clusters.
    • ? Add an option to WikimediaDebug to select which version is used
    • Add ability to do consistent client-side ramp-up via a WikimediaEvents cookie.
  • Per-cluster ramp-up:
    • App servers
      • N% of all cookie-based traffic to Appservers and API appservers
      • Parsoid
      • Job runners
      • Appservers
      • API appservers
    • Snapshot hosts ("dumps")
    • Deployment hosts
    • Maintenance hosts
    • Wikitech host
  • Decommission support for PHP 7.4 from production
  • Drop PHP 7.4 and 8.0 testing from CI, if not otherwise needed for release branches

Details

TitleReferenceAuthorSource BranchDest Branch
Add PHP 8.1 imagesrepos/releng/dev-images!23jforresterphp81main
Customize query in GitLab

Related Objects

StatusSubtypeAssignedTask
StalledNone
OpenNone
OpenNone
OpenNone
StalledNone
OpenNone
StalledNone
StalledFeatureNone
StalledKrinkle
Resolvedtstarling
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
Resolvedtstarling
ResolvedReedy
ResolvedBUG REPORTtstarling
Resolvedtstarling
ResolvedDaimona
ResolvedDaimona
ResolvedNone
ResolvedJdforrester-WMF
ResolvedBUG REPORTNone
Resolvedtstarling
ResolvedJdforrester-WMF
Resolved ssastry
Resolvedkostajh
Resolvedkostajh
Resolvedthiemowmde
Resolvedtstarling
Resolvedtstarling
ResolvedBUG REPORTLucas_Werkmeister_WMDE
Resolvedhoo
Resolvedhoo
ResolvedJdforrester-WMF
Resolvedthiemowmde
Resolvedkostajh
ResolvedUmherirrender
ResolvedPRODUCTION ERROR brooke
ResolvedTheresNoTime
Resolvedtstarling
OpenJdforrester-WMF
Resolvedlarissagaulia
ResolvedJMeybohm
Resolved MoritzMuehlenhoff
In ProgressNone
Resolvedjijiki
Resolvedaaron
Openjijiki
In ProgressClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
DuplicateClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
OpenNone
OpenNone
ResolvedJoe
Resolvedkamila
Resolvedhnowlan
OpenNone
OpenNone
ResolvedJoe
Resolvedcolewhite
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
InvalidClement_Goubert
ResolvedJoe
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedJoe
ResolvedJoe
ResolvedJoe
ResolvedJMeybohm
ResolvedJoe
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
DeclinedClement_Goubert
ResolvedClement_Goubert
Resolvedelukey
InvalidKrinkle
Resolvedjijiki
ResolvedJoe
ResolvedJoe
ResolvedClement_Goubert
ResolvedBUG REPORTClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedJoe
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedJclark-ctr
ResolvedJMeybohm
ResolvedJoe
ResolvedJoe
ResolvedNone
Resolvedjijiki
Resolvedjijiki
Resolved dancy
Resolved dancy
ResolvedJoe
ResolvedJoe
Resolvedjeena
ResolvedJoe
ResolvedJoe
Resolved dancy
ResolvedJoe
Resolved dpifke
Resolved dancy
ResolvedJoe
ResolvedClement_Goubert
Resolvedcolewhite
Resolvedjijiki
Resolved dpifke
ResolvedLegoktm
ResolvedClement_Goubert
ResolvedJMeybohm
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
Resolvedhnowlan
Resolvedakosiaris
Openhnowlan
ResolvedClement_Goubert
ResolvedNone
ResolvedDreamy_Jazz
ResolvedPRODUCTION ERRORDreamy_Jazz
Resolvedkostajh
Resolvedjijiki
OpenNone
Resolvedkamila
ResolvedJhancock.wm
ResolvedJclark-ctr
ResolvedJhancock.wm
ResolvedVRiley-WMF
ResolvedJhancock.wm
ResolvedClement_Goubert
ResolvedClement_Goubert
Resolvedakosiaris
OpenNone
Resolvedakosiaris
Resolved dancy
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedClement_Goubert
ResolvedCDanis
Openjijiki
ResolvedJoe
In Progressjijiki
Resolvedjijiki
OpenNone
ResolvedJdforrester-WMF
Resolvedjijiki
In ProgressClement_Goubert
ResolvedClement_Goubert

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

8.1 not 8.0 or 8.2?

The next target for production is PHP 8.1. In practical terms, this was decided by perf and SRE as the ones proposing and pushing for the objective and resourcing it as-needed. (I recognise of course that yourself and @Reedy have already been doing a lot of work in addressing deprecation warnings as part of making MediaWiki master pass CI, for releases and local dev on PHP 8.x, and thus in prep for production PHP 8.x).

We choose PHP 8.1, based on PHP 8.2 not being ready yet, and possibly too big a jump from PHP 7.4 in one go to ease vendor management. We can skip PHP 8.0 and doing so reduces the number of migrations we have to carry out. If there are benefits to individual teams and maintainers if we stop at PHP 8.0 first, though, this is certainly something we can consider. I'm not aware of a major reason against PHP 8.0 in prod. But let us know soon in that case so that we can adjust plans, or alternatively by helping resolve the issue another way.

8.1 not 8.0 or 8.2?

The next target for production is PHP 8.1. In practical terms, this was decided by perf and SRE as the ones proposing and pushing for the objective and resourcing it as-needed. (I recognise of course that yourself and @Reedy have already been doing a lot of work in addressing deprecation warnings as part of making MediaWiki master pass CI, for releases and local dev on PHP 8.x, and thus in prep for production PHP 8.x).

We choose PH 8.1 based on PHP 8.2 not being ready yet, and possibly too big a jump from PHP 7.4 in one go to ease vendor management. We can skip PHP 8.0 and doing so reduces the number of migrations we have to carry out. If there are benefits to individual teams and maintainers if we stop at PHP 8.0 first, though, this is certainly something we can consider. I'm not aware of a major reason against PHP 8.0 in prod. But let us know soon in that case so that we can adjust plans, or alternatively by helping resolve the issue another way.

Happy the decision has been made; no particular pressure for 8.0 from me! Could the decision, who made it, and what went into it be documented somewhere (and ideally announced)? Thanks!

Change 855580 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/core@master] MediaWiki-Docker: Switch PHP images to PHP 8.1

https://gerrit.wikimedia.org/r/855580

Commit 9e7b5c19... pushed by jforrester:

[ repos/releng/dev-images @php81] Add PHP 8.1 images

Commit 9e7b5c19... pushed by brennen:

[ repos/releng/dev-images @main] Add PHP 8.1 images

Mentioned in SAL (#wikimedia-releng) [2022-12-07T22:59:44Z] <brennen> Updating development images on contint primary to release php 8.1 images for T319432

Change 855580 merged by jenkins-bot:

[mediawiki/core@master] MediaWiki-Docker: Switch PHP images to PHP 8.1

https://gerrit.wikimedia.org/r/855580

Thanks for this task.

Just to put this in writing, serviceops has planned to do the Per Cluster ramp-up in the April-Jun quarter (this quarter we are focusing on MediaWiki on Kubernetes, which should contribute to making future PHP migrations easier). The prep work (e.g. CI, some ICU work needed by SRE) can happen well before that of course.

Krinkle changed the task status from Open to Stalled.May 1 2023, 7:36 PM

We choose PHP 8.1, based on PHP 8.2 not being ready yet

In the meantime, 8.2 has been out for over nine months, and it’s the version included in Debian Bookworm. Are you sure it makes sense to stick with 8.1, despite it not being included in any Debian version? It would be important to support at least one PHP version Debian includes (i.e. 7.4 or 8.2) for the benefit of third-party wikis; while it doesn’t need to be the one used in WMF production, it’s easier to support a PHP version if we ourselves use it.

Configure MW with ICU emulation for PHP 8.1 that matches PHP 7.4 (see also: T263437: Allow easier ICU transitions in MediaWiki (change how sortkey collation is managed in the categorylinks table), T292552: Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11).

I'm not sure where to raise this, so I'll raise it here: the category collation transition should also be done for 'uppercase' collations, not just ICU collations, as they are also affected by upgrades. For example, see T323868 and T251698 where this problem was reported.

We choose PHP 8.1, based on PHP 8.2 not being ready yet

In the meantime, 8.2 has been out for over nine months, and it’s the version included in Debian Bookworm. Are you sure it makes sense to stick with 8.1, despite it not being included in any Debian version? It would be important to support at least one PHP version Debian includes (i.e. 7.4 or 8.2) for the benefit of third-party wikis; while it doesn’t need to be the one used in WMF production, it’s easier to support a PHP version if we ourselves use it.

https://www.mediawiki.org/wiki/Support_policy_for_PHP

Hi, Data Products owns dumps now, and so I think we're responsible for the snapshot hosts mentioned in the description here. Do we have a timeline for this upgrade?

@Krinkle this task is marked as stalled, is it blocked on something?

Also, to bump @Tacsipacsi's question above (T319432#9177477), should we consider 8.2 for WMF production, since the decision to go with 8.1 instead of 8.2 (T319432#8338918) was made ~1 year ago?

@Krinkle this task is marked as stalled, is it blocked on something?

The ICU migration needs to happen first: https://phabricator.wikimedia.org/T329491

Please set subtasks in tasks to express dependencies. T329491 and T345561 are not connected to anything so it's not obvious what needs to happen first.

In the meantime, 8.2 has been out for over nine months, and it’s the version included in Debian Bookworm. Are you sure it makes sense to stick with 8.1, despite it not being included in any Debian version? It would be important to support at least one PHP version Debian includes (i.e. 7.4 or 8.2) for the benefit of third-party wikis; while it doesn’t need to be the one used in WMF production, it’s easier to support a PHP version if we ourselves use it.

Given the amount of work that went into getting 8.1 running in CI and making everything pass, I would suggest upgrading to 8.1 within the next three months, and then starting a PHP 8.3 upgrade project immediately after that, with target completion in 2024.

We can start work on 8.3 CI whenever. I found out at php-src#11469 that Drupal are constantly running their CI against PHP master. My PR was merged on June 6 and they filed a bug about it on June 18 asking for a revert, which they immediately got. That's the kind of CI I'd like to have.

We can start work on 8.3 CI whenever.

Yup, T339350 is tracking that; right now we're blocked by a lack of releases, particularly of php-ast; I believe 8.3.0's release date will be as is traditional US Thanksgiving again, so I'll have some time around then to see if we can get it out then.

@Krinkle this task is marked as stalled, is it blocked on something?

The ICU migration needs to happen first: https://phabricator.wikimedia.org/T329491

Which was replaced by T345561: Upgrade the MediaWiki servers to ICU 67, which is resolved..

Yup, AIUI that for simplicity it's been decided that this is blocked by T290536: Serve production traffic via Kubernetes; should we mark that formally in Phab?

Yeah, otherwise it's unclear what/why this is stalled

(Disparate notes for ease of linking during an upcoming internal WMF presentation.)

Upstreams: PHP.net, Debian, Composer (lib dependencies)

We benefit from Debian Linux and their LTS packages for PHP, which maintain and provide security patches beyond the official PHP.net support at https://www.php.net/supported-versions. For example, while php.net ended "Active support" for PHP 7.4 in 2021, and ended Security support in 2022; We actually use Debian Linux 11 ("Bullseye") which provides LTS and security support for php7.4 until June 2027 per https://wiki.debian.org/LTS.

2023_php_EOL_wmf.png (526×1 px, 31 KB)

However, this is limited to the PHP engine itself. It does not help us with our runtime dependency on specific libraries, which we install from Packagist.org via Composer. Examples of libraries that we deploy in production to run Wikipedia on MediaWiki, include: Monolog, JWT, Symfony, PHPUnit, etc.

Libraries have much shorter support in practice. Plus, bugs in an old version rarely receive a published CVE that one could subscribe to. This just isn't a practice for most individually maintained libraries, since only the latest version at any given time is supported. Individual releases generally do not enjoy long-term support. In practice, "support" from a library translates to a patch to the latest version, and you have to upgrade to latest version to get it.

Over 30% of the 1000 most popular Composer packages require PHP 8.0 or later. This is not theoretical, this is today. From https://stitcher.io/blog/php-version-stats-july-2023:

2023_php_EOL_Composer.png (954×1 px, 18 KB)

This is already affecting MediaWiki dependencies. Below is a list of packages we have in production ("mediawiki-vendor") that have incompatible upgrades available upstream. We are unable to install these due to running an outdated PHP version.

f we look at upstream "main" branches, there are several more who discontinued PHP 7.4 in their main branch, but have not released it yet. This means any upcoming of security fix, bug fix, or other improvement will be part of a release we can't install.

Misc links

  • Example of six months period from June 2023 to Feb 2024, where our own performance tooling did not work in our official MediaWiki-Docker dev environment. This because we moved from PHP 7.4 to PHP 8.1, and thus by extent away from WMF packaging to a third-party (deb.sury.org) which happened to lack some of the required packages. https://github.com/oerdnj/deb.sury.org/issues/1952