Change #1165151 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] aptrepo: add php83 component and pcre2 updates

https://gerrit.wikimedia.org/r/1165151

Change #1165152 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] package_builder: add pbuilder hook for php83

https://gerrit.wikimedia.org/r/1165152

Change #1165151 merged by Scott French:

[operations/puppet@production] aptrepo: add php83 component and pcre2 updates

https://gerrit.wikimedia.org/r/1165151

Change #1165152 merged by Scott French:

[operations/puppet@production] package_builder: add pbuilder hook for component/php83

https://gerrit.wikimedia.org/r/1165152

Once I'm able to pull the PCRE2 backport builds into component/php83, I should be able to start on the build process.

As before, my intention is to pick up the latest available package versions from https://salsa.debian.org/php-team, which is largely equivalent to what one would get from deb.sury.org as of now.

Given that, these are the package versions I expect we'll pick up (but cannot guarantee, until I've actually tried to build them and dealt with any issues that arise):

Note: The only package with a major version bump is php-msgpack (3.0.0). That sounds potentially risky in terms of inter-operation with 2.2.0-serialized objects, though in practice, I suspect we don't actually use msgpack in production - e.g., php-memcached supports using it, but we use the built-in php serializer there. From a quick look at the project's changelog, this may be related to the addition of PHP 8.1 enum serialization.

If either:

1. there are specific extensions where a different version is desired, or
2. more generally, there is a preference for maintaining parity with what we have been using on 8.1 (i.e., minimize the number of things changing)

... then now would be a good time to call that out.

Change #1165606 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] aptrepo: add pcre2-php83-bullseye to Update list

https://gerrit.wikimedia.org/r/1165606

Change #1165606 merged by Scott French:

[operations/puppet@production] aptrepo: add pcre2-php83-bullseye to Update list

https://gerrit.wikimedia.org/r/1165606

Mentioned in SAL (#wikimedia-operations) [2025-07-02T17:53:45Z] <swfrench-wmf> reprepro update component/php83 with pcre2 10.42-1~wmf11+1 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-02T18:42:15Z] <swfrench-wmf> reprepro include php8.3_8.3.22-1+wmf11u1 in component/php83 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-02T20:29:04Z] <swfrench-wmf> reprepro include php-defaults_94+wmf11u1 in component/php83 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-02T20:34:18Z] <swfrench-wmf> reprepro include dh-php_5.5+wmf11u1 in component/php83 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-03T00:03:34Z] <swfrench-wmf> reprepro include php-apcu_5.1.24-1+wmf11u1 in component/php83 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-03T00:08:41Z] <swfrench-wmf> reprepro include php-igbinary_3.2.16-4+wmf11u1 in component/php83 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-03T00:09:40Z] <swfrench-wmf> reprepro include php-msgpack_3.0.0-1+wmf11u1 in component/php83 - T398245

Status update before I disappear for a bit:

I was able to build nearly the full set of necessary packages today as a "pilot" run of sorts, including extensions listed in T398245#10965108 at the indicated version.

Only those packages necessary to satisfy build dependencies have been included in component/php83, as I would like to do a bit more validation first (they are, however, present on apt1002 and ready for inclusion once that's done).

Thus, the component is not yet ready for use. I will pick this work back up when I return.

Open questions:

The one package I'm holding off on for the moment is tideways, of which I am unclear of the current status. The Debian php-team repository appears to be untouched since the prep for bookworm (8.2), and the package appears to not be slated for inclusion in trixie as far as I can tell.

If folks could please chime in on compatibility / suitability for use with PHP 8.3, that would be greatly appreciated. I know that I can modify the package to make it build under 8.3 (and indeed I've tested this), but the question is more whether I should be doing so (i.e., the extension is expected to work if indeed it compiles).

More generally, if folks have feedback on the specific versions listed in T398245#10965108, now would be a good time to surface it.

Since it wasn't mentioned on task; do the packages include the backported mail patches Jesse made?

Thanks for the heads-up, @MoritzMuehlenhoff - I'd not seen those!

I need to rebuild PHP core anyway upon my return due to today's release of 8.3.23, so when I do so, I'll also add patches for Jesse's mail fixes.

Reference: T360995#10942948

Alright, I was able to build 8.3.23 with patches for Jesse's error handling / logging improvements:

• https://github.com/php/php-src/commit/c5e7490963a1c53ec12c09264d0dbe1654ec8cfc
• https://github.com/php/php-src/commit/6a7561203de89031147c0de6fe09cad905f82984

These required a bit of manual wrangling to turn into quilt patches that would apply cleanly, but nothing terribly involved (once I got the hang of the tooling, which was the biggest hurdle, heh). I've confirmed that the test cases added or modified in those patches do indeed pass.

I'll pick this up again tomorrow, and hopefully we can wrap this task up early this week, as long as there's some clarity around tideways.

Mentioned in SAL (#wikimedia-operations) [2025-07-15T14:06:50Z] <swfrench-wmf> reprepro include php8.3_8.3.23-1+wmf11u2 in component/php83 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-15T22:27:20Z] <swfrench-wmf> reprepro include php-excimer_1.2.5-1+wmf11u1 php-imagick_3.7.0-13+wmf11u1 php-luasandbox_4.1.2-1+wmf11u1 php-memcached_3.3.0-1+wmf11u1 php-pcov_1.0.12-1+wmf11u1 php-redis_6.2.0-1+wmf11u1 php-uuid_1.3.0-1+wmf11u1 php-wmerrors_2.0.0-1+wmf11u1 php-yaml_2.2.4-1+wmf11u1 wikidiff2_1.14.1-2+wmf11u1 xdebug_3.4.4-1+wmf11u1 in component/php83 - T398245

component/php83 is now populated with with all packages except tideways, reflecting the table of extension package versions in T398245#10965108.

This subset is sufficient to unblock the production images (T398246), but not sufficient to unblock production MediaWiki debug-image builds (which install tideways).

Before we close this out:

1. If anyone has concerns about the specific versions in T398245#10965108 (see note there around msgpack compatibility), please speak up.
2. We need to sort out how to proceed with tideways.

tideways: As described in T398245#10970535, tideways does not appear to be supported by the Debian php-team beyond PHP 8.2 on bookworm (i.e., does not exist in trixie). While I can modify the package to enable builds on 8.3, and indeed the extension compiles, my confidence is low that it will operate as expected: tests that assert properties of the collected call-graph stats fail on 8.3 (whereas they pass on 8.1).

P79140 shows one example. It's as if certain edges I would expect to exist do not (e.g., main() to top-level functions) and certain calls appear to recurse one level unexpectedly (e.g., tideways_xhprof_disable). It's unclear to me whether this could be an artifact of changes in the test harness (run-tests.php et al.) provided in php8.1-dev vs. php8.3-dev, rather than something more fundamental reflecting the operation of the extension.

One observation that makes me second guess whether there's an actual issue here is that building the library locally for PHP 8.2 on bookworm - which should ostensibly be non-broken if Debian offers the package for 8.2 - produces the same test failures (note that test failures have been made non-fatal to the overall build process for this package).

Ultimately, I'll still need input from folks with more expertise to assess whether this is acceptable. @MSantos - Could you help connect me with the right folks in MediaWiki Engineering to weigh in on this?

tideways: As described in T398245#10970535, tideways does not appear to be supported by the Debian php-team beyond PHP 8.2 on bookworm (i.e., does not exist in trixie). While I can modify the package to enable builds on 8.3, and indeed the extension compiles, my confidence is low that it will operate as expected: tests that assert properties of the collected call-graph stats fail on 8.3 (whereas they pass on 8.1).

The php-team definitely does a good support of the packages.

As I remember it, php-tideways was a fork of FaceBook XHProf, and Facebook certainly abandoned it as they had profiling built in HHVM. The fork is maintained by a small company in Germany and they are not more publishing it anymore. So essentially it is abandoned

Debian #1072571 php-tideways: Name mismatch between tideways_xhprof.so module name and package name

I'm an engineer at Tideways GmbH - Recently we've received a few bug reports for the php-tideways package as people have erroneously installed it instead of our proprietary package from the package servers we provide.

The confusion might stem from the package name being php-tideways instead of php-tideways-xhprof, matching the internal module name

While we used to maintain https://github.com/tideways/php-xhprof-extension in the past our tideways-php extension is not publically available at the moment.

And on Debian #1075765 RM: tideways -- ROM; deprecated by upstream, Ondrej mentions:

Either renaming the package or removing it from future Debian versions
would be appreciated.

So, we are just doing that. Enshittification in progress...

And that is how the package got removed (see https://tracker.debian.org/pkg/tideways):

• [2024-07-26] Removed 5.0.4-16 from unstable (Debian FTP Masters)
• [2024-07-27] tideways REMOVED from testing (Debian testing watch)

The upstream repository at https://github.com/tideways/php-xhprof-extension was marked archived on October 2023. Its description links to https://github.com/longxinH/xhprof (which is a fork for phacility/xhprof, Phacility was the company that supported Phabricator and was created by alumni from Facebook).

Since the tideways fork is no more available, upstream recommends xhprof:

• https://pecl.php.net/package/xhprof
• https://github.com/longxinH/xhprof

I imagine the code would need to be audited and tested for our use case (prod profiling). I guess it is worth filing an independent task.

Change #1170158 had a related patch set uploaded (by Jforrester; author: Jforrester):

[integration/config@master] Docker: [php83] Re-build based on Wikimedia-provided binaries

https://gerrit.wikimedia.org/r/1170158

Thanks for the additional context and details behind the decision to remove the package, @hashar.

Great, so if https://pecl.php.net/package/xhprof is considered a viable alternative - i.e., sufficient to warrant any necessary engineering work to adopt it over tideways (e.g., in XWD profiling support [0]) - then that is certainly something we can easily adopt from the packaging side. Although it does not exist in Debian as of yet, the php-team maintains a package thereof: https://salsa.debian.org/php-team/pecl/php-xhprof.

I think the key question is whether to initially roll the dice with tideways on 8.3, on the basis that it might work (but may also produce suspect profiling data), vs. front-loading any necessary work to switch to xhprof.

Given that the affected scope is "only" mw-debug (we do not install the extension anywhere else), the former seems reasonable to me. However, this is not a decision SRE should be making in isolation.

[0] From a cursory look at core's ProfilerXhprof, it maybe already supports plain xhprof?

Another issue: while using component/php83 @Jdforrester-WMF encountered:

$ dockerfiles/debug-image quibble-bullseye-php83 1.14.1-s8
nobody@5e4ab06ba224:/workspace$ php -v
PHP Warning:  PHP Startup: ^(text/|application/xhtml\+xml) (offset=0): unrecognised compile-time option bit(s) in Unknown on line 0
PHP 8.3.23 (cli) (built: Jul 14 2025 19:05:58) (NTS)

The reason is libpcre2 version. We built the image with Bullseye which comes with libpcre 10.36. PHP 8.3.0 came with libpcre 10.42 and it has an update for 10.39 which I think make it not back compatible.

Can you bump the package requirement? Our 8.3.23-1+wmf11u2 package has Depends: libpcre2-8-0 (>= 10.32). That should be bumped to I think 10.42. We had the same issue with the php 8.1 packages: T386006.

@hashar - Yes, we can consider doing that. In the meantime, presumably it can be solved the same way as with the 8.1 images: add an explicit libpcre2-8-0 to the installation list after component/php83 has been added.

Edit: For a bit more context, the reason why changing the Depends constraint in the control data is complicated is that it's dynamically generated based on the symbols referenced by the resulting shared objects (IIRC). It will take some doing to "force" that to 10.42, so in the interest of expedience, using the above trick is likely the best short term fix.

On a hunch given their common origin, I took a look at older xhprof issues that might rhyme with the tideways test failures we're seeing on 8.3.

Sure enough, https://github.com/longxinH/xhprof/issues/68 doesn't manifest quite the same way, but clearly results in spurious or "disconnected" frames in the callstack that look similar to what we're seeing (attributed to this upstream change).

After a very simple patch (P79272) to suppress these, all tests now pass on 8.3. Now, I'm not going to say that means tideways works, but it might be worth a shot to at least test it out.

In T398245#10965108, @Scott_French wrote:

[…] Note: The only package with a major version bump is php-msgpack (3.0.0). […], I suspect we don't actually use msgpack in production

Yeah, it looks like it's pulled in as dependency of php-memcached. In 2016, the upstream Debian package changed its "Suggests" declaration for php-igbinary and php-msgpack to a hard dependency. It says this is to "prevent loading errors" which I'm not sure what it refers to.

Our bare metal provision in Puppet did not declare it as dependency prior to 2018 (Puppet: mediawiki::packages::php7), but as part of the refactor to Puppet profile it became explicilty installed, although presumably apt-get install php-memcached would have started doing so implicitly around that time as well, after the next Debian upgrade.

In any event, we have no references to msgpack today in any PHP code in MediaWiki containers (MediaWiki core, extensions, or vendored libraries).

In T398245#11009656, @Scott_French wrote:

Thanks for the additional context and details behind the decision to remove the package, @hashar.

Great, so if https://pecl.php.net/package/xhprof is considered a viable alternative

Phacility (the makers of Phabricator) stopped maintaining XHProf and thus never officially supported PHP7. However, by the time that happened, WMF already moved off from PHP5/php-xhprof and onto HHVM (which shipped its own xhprof-compatible implementation). Meanwhile, a dozen unofficial forks of php-xhprof appeared. One of which was a stripped down version, branded tideways_xhprof. When WMF switched from HHVM to PHP 7.2, we adopted php-tideways-xhprof (T206152, (blogpost). A little bit later, https://github.com/longxinH/xhprof appeared as well, which after a php.net discussion succesfully overtook the PECL "xhprof" package in 2018, and thus their fork became the dominent fork after that.

From a cursory look at core's ProfilerXhprof, it maybe already supports plain xhprof?

MediaWiki indeed never dropped support for php-xhprof. We made sure to only use the subset of php-xhprof and php-tideways-xhprof, and during the HHVM-PHP7.2 migration added support for both function names in MediaWiki core.

Having said that, the version of php-xhprof has now increased from 0.9 to 2.x, and we have not tested php-xhprof for many years, so there may very well be breaking changes we need to account for. And either way, we do indeed need to update wmf-config's code for WikimediaDebug to specifically support php-xhprof since that code is specific to WMF and assumes tideways_xhprof.

I'm +1 for doing this now as part of the PHP 8.3 upgrade. MediaWiki core already promises to support it, so if someone else tests and reports a bug we'd deal with it now as well.

Based on user reports trying tideways_xhprof on PHP 8.1+ it seems that even when it compiles correctly, it may have unaddressed bugs with its runtime output that we'd need to fix, so it may actually be less effort to support php-xhprof than tideways_xhprof. If this works out, I would suggest that we actually provision and transition to php-xhprof on PHP 8.1 first, and then upgrade.

As always, note that neither php-xhprof nor php-tideways-xhprof should not be installed in the main containers, only in the mwdebug containers. Given that it is specific to debugging use cases, we can cleanly switch from one to the other, and then deploy the relevant wmf-config and WikimediaDebug changes right after that (after local testing confirms that this would work).

Thank you very much, @Krinkle - both for the historical background and weighing in on a path forward.

Front-loading validation and adoption (e.g., in mediawiki-config) of xhprof 2.x over tideways_xhprof on 8.1 sounds great. I've just now built the 2.3.10 from https://salsa.debian.org/php-team/pecl/php-xhprof for both 8.1 and 8.3, and will plan to include that in apt.w.o soon.

I'll also potentially upload the patched php-tideways for 8.3, though we may not end up needing it (in which case, we can consider removing it).

Edit: Also, yes - either package will only ever be installed in the special-case mwdebug or cli images (never the main production MediaWiki image), and making the change will involve swapping the package names in the relevant Dockerfile (e.g., for mwdebug).

Mentioned in SAL (#wikimedia-operations) [2025-07-24T15:13:06Z] <swfrench-wmf> reprepro include php-xhprof_2.3.10-1+wmf11u1 in component/php81 - T398245

Mentioned in SAL (#wikimedia-operations) [2025-07-24T15:13:52Z] <swfrench-wmf> reprepro include php-xhprof_2.3.10-1+wmf11u1 tideways_5.0.4-16+wmf11u2 in component/php83 - T398245

After discussion with folks in serviceops and Infrastructure-Foundations, I've gone ahead and included the respective xhprof 2.3.10-1 packages in component/php81 and component/php83.

I've also speculatively included the patched tideways package in component/php83.

Note that nothing (i.e., in the image build process, or in puppet for bare-metal use cases) knows about xhprof yet, so there is no risk of both being installed.

In any case, this puts us in a position where we can sequence the switch semi-independently of the introduction of MediaWiki PHP 8.3 images and / or early testing in mwdebug (as long as we're willing to try our luck with the patched tideways package).

In T398245#11010808, @hashar wrote:

Another issue: while using component/php83 @Jdforrester-WMF encountered:

$ dockerfiles/debug-image quibble-bullseye-php83 1.14.1-s8
nobody@5e4ab06ba224:/workspace$ php -v
PHP Warning:  PHP Startup: ^(text/|application/xhtml\+xml) (offset=0): unrecognised compile-time option bit(s) in Unknown on line 0
PHP 8.3.23 (cli) (built: Jul 14 2025 19:05:58) (NTS)

The reason is libpcre2 version. We built the image with Bullseye which comes with libpcre 10.36. PHP 8.3.0 came with libpcre 10.42 and it has an update for 10.39 which I think make it not back compatible.

Can you bump the package requirement? Our 8.3.23-1+wmf11u2 package has Depends: libpcre2-8-0 (>= 10.32). That should be bumped to I think 10.42. We had the same issue with the php 8.1 packages: T386006.

We had the same libpcre2 / PHP mismatch with the php 8.1 packages (T387276). I have filed T400693 to have it sorted out.

Many thanks to @Krinkle for validating xhprof as a suitable alternative for tideways in T400109.

My understanding is that, once profiling-related code in mediawiki-config is updated for xhprof compatibility (e.g., in Profiler.php), we should be good to transition production mw-debug to xhprof in advance of the 8.3 migration. Presumably that work will happen as a subtask of T348379: Migrate use of php-tideways_xhprof to php-xhprof, but I'll try to confirm.

At this point, I believe all the work explicitly tracked here is complete, with one late-breaking item: PHP 8.3.24 was just released, carrying a number of bugfixes. I'm building that now (with mail fixes backported, as before) and once those are included, we're done here.

Mentioned in SAL (#wikimedia-operations) [2025-08-04T20:32:22Z] <swfrench-wmf> reprepro include php8.3_8.3.24-1+wmf11u2 in component/php83 - T398245