Change #1062753 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] Add component/php81 for bullseye-wikimedia

https://gerrit.wikimedia.org/r/1062753

Change #1062754 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] Add pbuilder hook for component/php81

https://gerrit.wikimedia.org/r/1062754

Change #1062753 merged by Scott French:

[operations/puppet@production] Add component/php81 for bullseye-wikimedia

https://gerrit.wikimedia.org/r/1062753

Change #1062754 merged by Scott French:

[operations/puppet@production] Add pbuilder hook for component/php81

https://gerrit.wikimedia.org/r/1062754

Mentioned in SAL (#wikimedia-sre) [2024-08-20T20:31:12Z] <swfrench-wmf> imported php8.1_8.1.29-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-20T20:53:56Z] <swfrench-wmf> imported php-defaults_92+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-20T21:04:47Z] <swfrench-wmf> imported dh-php_5.4+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T16:40:43Z] <swfrench-wmf> imported php-apcu_5.1.23-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T16:41:49Z] <swfrench-wmf> imported php-excimer_1.2.2-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T16:42:45Z] <swfrench-wmf> imported php-imagick_3.7.0-6+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T16:44:16Z] <swfrench-wmf> imported php-luasandbox_4.1.2-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T16:45:13Z] <swfrench-wmf> imported php-msgpack_2.2.0-4+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T16:45:58Z] <swfrench-wmf> imported php-pcov_1.0.11-5+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T17:34:02Z] <swfrench-wmf> imported php-wmerrors_2.0.0-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T17:34:41Z] <swfrench-wmf> imported php-yaml_2.2.3-2+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T17:35:50Z] <swfrench-wmf> imported tideways_5.0.4-16+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T17:36:27Z] <swfrench-wmf> imported wikidiff2_1.14.1-2+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T17:37:01Z] <swfrench-wmf> imported xdebug_3.3.2-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T17:56:03Z] <swfrench-wmf> imported php-igbinary_3.2.15-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T18:13:42Z] <swfrench-wmf> imported php-redis_6.0.2-1+wmf11u1 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-21T18:21:50Z] <swfrench-wmf> imported php-memcached_3.2.0++-1+wmf11u1 into component/php81 - T372507

Verified that in a fresh docker-registry.wikimedia.org/bullseye:latest image, I can successfully:

echo 'deb http://apt.wikimedia.org/wikimedia bullseye-wikimedia component/php81' > /etc/apt/sources.list.d/php81.list
apt update && apt install -y php8.1-cli php8.1-bcmath php8.1-bz2 php8.1-curl php8.1-dba php8.1-gd php8.1-gmp php8.1-intl php8.1-mbstring php8.1-mysql php8.1-xml php8.1-excimer php8.1-apcu php8.1-igbinary php8.1-memcached php8.1-msgpack php8.1-redis php8.1-fpm php8.1-excimer php8.1-luasandbox php8.1-wikidiff2 php8.1-wmerrors php8.1-yaml

That covers the entire set of packages installed in the 7.4-flavored base images [0], minus php-geoip.

[0] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/production-images/+/refs/heads/master/images/php/7.4

While working through the production image definitions for T372602, I discovered that the three extension packages maintained by WMF (php-luasandbox, php-wmerrors, wikidiff2) build successfully, but with incomplete contents.

I suspect this is related to their use of "older" style dh-php integration vs. the more recent dh-php required to build other upstream extensions.

For example:

dpkg -c php8.1-luasandbox_4.1.2-1+wmf11u1_amd64.deb 
drwxr-xr-x root/root         0 2024-08-21 09:10 ./
drwxr-xr-x root/root         0 2024-08-21 09:10 ./etc/
drwxr-xr-x root/root         0 2024-08-21 09:10 ./etc/php/
drwxr-xr-x root/root         0 2024-08-21 09:10 ./etc/php/8.1/
drwxr-xr-x root/root         0 2024-08-21 09:10 ./etc/php/8.1/mods-available/
-rw-r--r-- root/root        24 2024-08-21 09:10 ./etc/php/8.1/mods-available/luasandbox.ini
drwxr-xr-x root/root         0 2024-08-21 09:10 ./usr/
drwxr-xr-x root/root         0 2024-08-21 09:10 ./usr/share/
drwxr-xr-x root/root         0 2024-08-21 09:10 ./usr/share/doc/
drwxr-xr-x root/root         0 2024-08-21 09:10 ./usr/share/doc/php8.1-luasandbox/
-rw-r--r-- root/root      1396 2024-08-21 09:10 ./usr/share/doc/php8.1-luasandbox/changelog.Debian.gz
-rw-r--r-- root/root      1437 2024-08-20 08:57 ./usr/share/doc/php8.1-luasandbox/copyright

i.e., it looks like the shared object never made it in there.

I'll take a closer look later today or tomorrow.

Alright, I think I see a way out of this: I'd overlooked that the debian/rules files for these packages set an explicit INSTALL_ROOT for make install in override_dh_auto_install, which did not include the version number (i.e., does not match the "manually made coinstallable" package name). Fixing that makes it so that the result in where dh-php expects it to be.

Mentioned in SAL (#wikimedia-operations) [2024-08-22T20:17:20Z] <swfrench-wmf> imported php-luasandbox_4.1.2-1+wmf11u2 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-22T20:18:28Z] <swfrench-wmf> imported php-wmerrors_2.0.0-1+wmf11u2 into component/php81 - T372507

Mentioned in SAL (#wikimedia-operations) [2024-08-22T20:19:15Z] <swfrench-wmf> imported wikidiff2_1.14.1-2+wmf11u2 into component/php81 - T372507

Shared objects are now present in all three packages, as well as a previously missing .ini file from wikidiff2. Verified that a local build of docker-registry.wikimedia.org/php8.1-fpm-multiversion-base no longer produces warnings about missing extensions.

Following up on the status of the php-geoip extension (h/t to @Krinkle for all the discussion out of band):

In Debian, there is no php-geoip package after bullseye (where it's built against 7.4). See also this commit [0] from 2020 in the packaging repo suggesting lack of 8.x support. At the same time, there have been more recent (unreleased?) changes in the upstream repo [1] that suggest 8.1 support efforts.

In any case, the main question is really whether we need this at all.

In puppet, this comment [2] would suggest it's only needed for fundraising use cases. That's a little surprising, as we wouldn't expect that to be used in a fundraising context (although I'm told there are cases where "fundraising" colloquially can refer to specific functionality in, e.g., CentralNotice).

While the presence of the php-geoip package can be traced much earlier, the oldest reference I can find that provides clear context on its presence is [3], again pointing solely to fundraising.

Ultimately, probably the best we can really do is trust code search and proceed with a progressive rollout to shake out surprises. For the former, the only reference to a geoip_ function call in a php source file is [4] in the DeviceMapLogCapture extension. This is an experimental extension that we do not use (and further, the call to geoip_country_code_by_name is conditioned on the function existing).

Note: We do use the newer pure-php GeoIp2 package provided by maxmind in a number of spots, and it's possible those have superseded earlier uses of the extension (e.g., in order to support the new DB format).

[0] https://salsa.debian.org/php-team/pecl/php-geoip/-/commit/c1a02ab265b65bcff21d03b15e86edfb91b22f6d

[1] https://github.com/php/pecl-networking-geoip

[2] https://gerrit.wikimedia.org/g/operations/puppet/+/678708e5d3a5fb8b4a37deb28a6c77e5f845f6f6/modules/profile/manifests/mediawiki/php.pp#13

[3] https://gerrit.wikimedia.org/r/c/operations/puppet/+/394977

[4] https://gerrit.wikimedia.org/g/mediawiki/extensions/DeviceMapLogCapture/+/b9e02723ce0d52905eaaecd98de687f4f6cddb31/includes/api/ApiDeviceMapLogCapture.php#32

In T372507#10088733, @Scott_French wrote:

In puppet, this comment [2] would suggest it's only needed for fundraising use cases. That's a little surprising, as we wouldn't expect that to be used in a fundraising context (although I'm told there are cases where "fundraising" colloquially can refer to specific functionality in, e.g., CentralNotice).

That comment dates from 2018. In 2019 I see this patch for the "DonationInterface" extension to migrate it to GeoIP2: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/469058

I really really doubt it's needed at this point :)

One last test and point-of-note:

The suite of packages in T372507#10082651 appear to be co-installable with their 7.4 equivalents from component/php74 as long as php-common (i.e., the version-independent common utilities package in php-defaults) from that component is installed, rather than the newer version from component/php81 (or, equivalently, 7.4 is installed first).

This appears to be an artifact of the way that php-common is specified to be incompatible with phpX.Y-common for all versions X.Y prior to the then-default version.

In any case, while this may(?) be something we can simplify with some package surgery, I'm not sure that's warranted here.

This is something we'll need to think about for any use cases that require co-installable php versions. That clearly does not include the production image use case, but may include, e.g., maintenance hosts (if the migration to mw-script on k8s has not yet completed).

In T372507#10093823, @Scott_French wrote:

In any case, while this may(?) be something we can simplify with some package surgery, I'm not sure that's warranted here.

This is something we'll need to think about for any use cases that require co-installable php versions. That clearly does not include the production image use case, but may include, e.g., maintenance hosts (if the migration to mw-script on k8s has not yet completed).

We haven't used/needed this so far and I don't think we'll need this in the future either. In the past we had some hosts/packages which pulled in the Debian default names (so e.g. when we were on PHP 7.2 for the main production loads), some packages or puppet manifests installed php-cli and thus pulled in the distro defaults (back then 7.0), but all these cases were cleaned out over time.

Adding @hashar and Continuous-Integration-Infrastructure. The packages and images that will be used to run production on php8.1 in the medium term are ready and can be used in CI.

Thank you for the early notice @akosiaris! I am subscribing @Jdforrester-WMF who is doing a lot of updates to the CI images, we might file a task dedicated to CI though.

The list of packages we use for MediaWiki tests is at https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/dockerfiles/quibble-bullseye-php81/Dockerfile.template

php8.1-apcu
php8.1-bcmath
php8.1-cli
php8.1-curl
php8.1-fpm
php8.1-gd
php8.1-gmp
php8.1-intl
php8.1-ldap
php8.1-mbstring
php8.1-memcached
php8.1-mysql
php8.1-pgsql
php8.1-sqlite3
php8.1-tidy
php8.1-xml
php8.1-zip

There are a few more extensions we use, but that get carried from sury.org rather than the php flavor used on WMF production

php8.1-ast
php8.1-excimer
php8.1-imagick
php8.1-pcov
php8.1-yaml

We would need xdebug as well for the dev images, but I see it got imported (apparently with whatever was the latest stable version).

While working through the production image definitions for T372602, I discovered that the three extension packages maintained by WMF (php-luasandbox, php-wmerrors, wikidiff2) build successfully, but with incomplete contents.

I suggest you get added to upstream team and possibly adjust the php extensions there? https://salsa.debian.org/mediawiki-team

Alright, I think I see a way out of this: I'd overlooked that the debian/rules files for these packages set an explicit INSTALL_ROOT for make install in override_dh_auto_install, which did not include the version number (i.e., does not match the "manually made coinstallable" package name). Fixing that makes it so that the result in where dh-php expects it to be.

Interestingly our Excimer extension has the same:

override_dh_auto_install:
	INSTALL_ROOT=$(CURDIR)/debian/php-excimer make install

Which is forked by the php-team at https://salsa.debian.org/php-team/pecl/php-excimer/-/blob/debian/main/debian/rules?ref_type=heads to have it aligned with the rest of the extensions they maintain:

#!/usr/bin/make -f
export REPORT_EXIT_STATUS:=0
include /usr/share/dh-php/pkg-pecl.mk

Regardless kudos on achieving to build those packages. I have been there some years ago, and once built, that grants the feeling of a great achievement.

If I can be a pain, can php-uuid (see T373752: Build php-uuid package, and add to WMF production and CI; not currently installed, and not explicitly needed for PHP 7.4 at least) be built going forward too please?

Thanks!

Change #1071941 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Docker: [php81] Re-build based on Wikimedia-provided binaries

https://gerrit.wikimedia.org/r/1071941

Thanks, @hashar!

Great, the core set of packages you mention in T372507#10135311 should all be available in component/php81 on bullseye, and it looks like all of the additional extension packages as well except php8.1-ast.

Also yes, exactly: the Excimer version supported by the php-team has the newer-style dh-php integration. I can take a look at what would be involved in updating the mediawiki-team extensions to adopt the same (properly, that is, rather than the way I did it here).

@Reedy - thanks for flagging. Yes, I can take a look adding php-uuid as well.

php8.1-ast

That one is for the Phan static analyzer and we currently get it from sury.org (which is usually pretty uptodate). If it easy to build, lets import it as well. Else we will keep borrowing it from Sury.org which has a version for Bullseye and hence php8.1 and hence is binary compatible.

Change #1071958 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Docker: [quibble-bullseye-php81] Re-build on Wikimedia-flavoured PHP

https://gerrit.wikimedia.org/r/1071958

I have a php8.1-uuid package (1.20-12) built and ready to go. There's one thing I'd like to check on first, but barring any surprises I'll import it into component/php81 tomorrow.

Mentioned in SAL (#wikimedia-operations) [2024-09-11T17:39:42Z] <swfrench-wmf> imported php-uuid_1.2.0-12+wmf11u1 into component/php81 - T372507

Alright, php8.1-uuid should now be available in component/php81. I'll add it to the 8.1 production images next (T372602).

Change #1071941 merged by jenkins-bot:

[integration/config@master] Docker: [php81] Re-build based on Wikimedia-provided binaries

https://gerrit.wikimedia.org/r/1071941

Mentioned in SAL (#wikimedia-releng) [2024-09-11T22:58:37Z] <James_F> Docker: [php81] Re-build based on Wikimedia-provided binaries, for T372507

Change #1072312 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] jjb: Switch non-Quibble PHP 8.1 jobs to ones with Wikimedia-provided PHP binaries

https://gerrit.wikimedia.org/r/1072312

Change #1072312 merged by jenkins-bot:

[integration/config@master] jjb: Switch non-Quibble PHP 8.1 jobs to ones with Wikimedia's PHP binaries

https://gerrit.wikimedia.org/r/1072312

Change #1071958 merged by jenkins-bot:

[integration/config@master] Docker: [quibble-bullseye-php81] Re-build on Wikimedia-flavoured PHP

https://gerrit.wikimedia.org/r/1071958

Mentioned in SAL (#wikimedia-releng) [2024-09-11T23:11:29Z] <James_F> Docker: [quibble-bullseye-php81] Re-build on Wikimedia-flavoured PHP, for T372507

Change #1072317 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] jb: Switch Quibble PHP 8.1 jobs to ones with Wikimedia's PHP binaries

https://gerrit.wikimedia.org/r/1072317

Change #1072317 merged by jenkins-bot:

[integration/config@master] jb: Switch Quibble PHP 8.1 jobs to ones with Wikimedia's PHP binaries

https://gerrit.wikimedia.org/r/1072317