Configure MW with ICU emulation for PHP 8.1 that matches PHP 7.4 (see also: T263437: Allow easier ICU transitions in MediaWiki (change how sortkey collation is managed in the categorylinks table), T292552: Rename articles and users to update our case mapping to PHP 7.4 and Unicode 11).

I'm not sure where to raise this, so I'll raise it here: the category collation transition should also be done for 'uppercase' collations, not just ICU collations, as they are also affected by upgrades. For example, see T323868 and T251698 where this problem was reported.

In T319432#9177477, @Tacsipacsi wrote:

In T319432#8338918, @Krinkle wrote:

We choose PHP 8.1, based on PHP 8.2 not being ready yet

In the meantime, 8.2 has been out for over nine months, and it’s the version included in Debian Bookworm. Are you sure it makes sense to stick with 8.1, despite it not being included in any Debian version? It would be important to support at least one PHP version Debian includes (i.e. 7.4 or 8.2) for the benefit of third-party wikis; while it doesn’t need to be the one used in WMF production, it’s easier to support a PHP version if we ourselves use it.

https://www.mediawiki.org/wiki/Support_policy_for_PHP

Hi, Experimentation Lab owns dumps now, and so I think we're responsible for the snapshot hosts mentioned in the description here. Do we have a timeline for this upgrade?

@Krinkle this task is marked as stalled, is it blocked on something?

Also, to bump @Tacsipacsi's question above (T319432#9177477), should we consider 8.2 for WMF production, since the decision to go with 8.1 instead of 8.2 (T319432#8338918) was made ~1 year ago?

In T319432#9234692, @kostajh wrote:

@Krinkle this task is marked as stalled, is it blocked on something?

The ICU migration needs to happen first: https://phabricator.wikimedia.org/T329491

Please set subtasks in tasks to express dependencies. T329491 and T345561 are not connected to anything so it's not obvious what needs to happen first.

In T319432#9177477, @Tacsipacsi wrote:

In the meantime, 8.2 has been out for over nine months, and it’s the version included in Debian Bookworm. Are you sure it makes sense to stick with 8.1, despite it not being included in any Debian version? It would be important to support at least one PHP version Debian includes (i.e. 7.4 or 8.2) for the benefit of third-party wikis; while it doesn’t need to be the one used in WMF production, it’s easier to support a PHP version if we ourselves use it.

Given the amount of work that went into getting 8.1 running in CI and making everything pass, I would suggest upgrading to 8.1 within the next three months, and then starting a PHP 8.3 upgrade project immediately after that, with target completion in 2024.

We can start work on 8.3 CI whenever. I found out at php-src#11469 that Drupal are constantly running their CI against PHP master. My PR was merged on June 6 and they filed a bug about it on June 18 asking for a revert, which they immediately got. That's the kind of CI I'd like to have.

In T319432#9306384, @tstarling wrote:

We can start work on 8.3 CI whenever.

Yup, T339350 is tracking that; right now we're blocked by a lack of releases, particularly of php-ast; I believe 8.3.0's release date will be as is traditional US Thanksgiving again, so I'll have some time around then to see if we can get it out then.

In T319432#9234734, @MoritzMuehlenhoff wrote:

In T319432#9234692, @kostajh wrote:

@Krinkle this task is marked as stalled, is it blocked on something?

The ICU migration needs to happen first: https://phabricator.wikimedia.org/T329491

Which was replaced by T345561: Upgrade the MediaWiki servers to ICU 67, which is resolved..

Yup, AIUI that for simplicity it's been decided that this is blocked by T290536: Serve production traffic via Kubernetes; should we mark that formally in Phab?

Yeah, otherwise it's unclear what/why this is stalled

(Disparate notes for ease of linking during an upcoming internal WMF presentation.)

Upstreams: PHP.net, Debian, Composer (lib dependencies)

We benefit from Debian Linux and their LTS packages for PHP, which maintain and provide security patches beyond the official PHP.net support at https://www.php.net/supported-versions. For example, while php.net ended "Active support" for PHP 7.4 in 2021, and ended Security support in 2022; We actually use Debian Linux 11 ("Bullseye") which provides LTS and security support for php7.4 until June 2027 per https://wiki.debian.org/LTS.

However, this is limited to the PHP engine itself. It does not help us with our runtime dependency on specific libraries, which we install from Packagist.org via Composer. Examples of libraries that we deploy in production to run Wikipedia on MediaWiki, include: Monolog, JWT, Symfony, PHPUnit, etc.

Libraries have much shorter support in practice. Plus, bugs in an old version rarely receive a published CVE that one could subscribe to. This just isn't a practice for most individually maintained libraries, since only the latest version at any given time is supported. Individual releases generally do not enjoy long-term support. In practice, "support" from a library translates to a patch to the latest version, and you have to upgrade to latest version to get it.

Over 30% (as of July 2023), and 50% (as of 2024) of the 1000 most popular Composer packages require PHP 8.0 or later. This is not theoretical, this is today. https://stitcher.io/blog/php-version-stats-july-2024:

This is already affecting MediaWiki dependencies. Below is a list of packages we have in production ("mediawiki-vendor") that have incompatible upgrades available upstream. We are unable to install these due to running an outdated PHP version.

• brick/math requires PHP 8.0. https://packagist.org/packages/brick/math
• symfony/process, symfony/console and other Symfony packages require PHP 8.1. https://packagist.org/packages/symfony/process https://packagist.org/packages/symfony/console https://github.com/symfony/symfony/pull/45377
• endroid/qr-code requires PHP 8.1. https://packagist.org/packages/endroid/qr-code
• monolog requires PHP 8.1. https://packagist.org/packages/monolog/monolog https://github.com/Seldaek/monolog/commit/22c8b19358
• phpunit requires PHP 8.1. https://phpunit.de/supported-versions.html
• uuid requires PHP 8.0. https://packagist.org/packages/ramsey/uuid
• ramsey/collection requires PHP 8.1. https://packagist.org/packages/ramsey/collection

f we look at upstream "main" branches, there are several more who discontinued PHP 7.4 in their main branch, but have not released it yet. This means any upcoming of security fix, bug fix, or other improvement will be part of a release we can't install.

• GeoIP2. https://packagist.org/packages/geoip2/geoip2 https://github.com/maxmind/GeoIP2-php/commit/a02b5d5c9b
• ruflin/Elastica. https://packagist.org/packages/ruflin/elastica https://github.com/ruflin/Elastica/pull/2131

Misc links

• Example of six months period from June 2023 to Feb 2024, where our own performance tooling did not work in our official MediaWiki-Docker dev environment. This because we moved from PHP 7.4 to PHP 8.1, and thus by extent away from WMF packaging to a third-party (deb.sury.org) which happened to lack some of the required packages. https://github.com/oerdnj/deb.sury.org/issues/1952

PHP 8.4 was released yesterday, and PHP 8.1 is now receiving critical security fixes only until December 31, 2025, after which no support is provided:

After this two year period of active support, each branch is then supported for two additional years for critical security issues only. Releases during this period are made on an as-needed basis: there may be multiple point releases, or none, depending on the number of reports.

Perhaps the target for this task should be 8.2, 8.3 or 8.4, to avoid completing the upgrade to 8.1, only to then need to upgrade to a newer version that has some level of security support?

In T319432#10347278, @kostajh wrote:

PHP 8.4 was released yesterday, and PHP 8.1 is now receiving critical security fixes only until December 31, 2025, after which no support is provided:

After this two year period of active support, each branch is then supported for two additional years for critical security issues only. Releases during this period are made on an as-needed basis: there may be multiple point releases, or none, depending on the number of reports.

Perhaps the target for this task should be 8.2, 8.3 or 8.4, to avoid completing the upgrade to 8.1, only to then need to upgrade to a newer version that has some level of security support?

We do not rely on upstream security support since we backport security patches internally. Our main wiki workloads are on PHP 7.4 and all security patches applicable to 7.4 have been backported, see e.g. https://phabricator.wikimedia.org/T378173 for the previous PHP security releases.

In T319432#10347290, @MoritzMuehlenhoff wrote:

In T319432#10347278, @kostajh wrote:

PHP 8.4 was released yesterday, and PHP 8.1 is now receiving critical security fixes only until December 31, 2025, after which no support is provided:

After this two year period of active support, each branch is then supported for two additional years for critical security issues only. Releases during this period are made on an as-needed basis: there may be multiple point releases, or none, depending on the number of reports.

Perhaps the target for this task should be 8.2, 8.3 or 8.4, to avoid completing the upgrade to 8.1, only to then need to upgrade to a newer version that has some level of security support?

We do not rely on upstream security support since we backport security patches internally. Our main wiki workloads are on PHP 7.4 and all security patches applicable to 7.4 have been backported, see e.g. https://phabricator.wikimedia.org/T378173 for the previous PHP security releases.

I see, thanks for the explanation.

Noting also that getting the k8s stuff done was done before this in attempt to make future upgrades easier..

I’m not sure we’re immediately start going to go to 8.3 or similar when we’re done with the 8.1 upgrade, but it should result in a much shorter timeline going forward