Please backport my patch https://github.com/php/php-src/pull/7478 to our custom PHP 7.2 buster packages and deploy it.
The patch was recently accepted into the PHP 8.1.x branch but the issue it fixes is constantly generating Parsoid timeouts in production, so it would be good if it could be deployed now.
I've made an updated PHP 7.2 package with a 7.2 backport of https://github.com/php/php-src/commit/781e6b4d214012e9b9c0cf96a239cdf9f948da91
Not yet uploaded to apt.wikimedia.org, but https://people.wikimedia.org/~jmm/php/
What's the best way to validate that it works as expected, shall I e.g. upgrade scandium (the Parsoid test host)?
Sure, upgrading scandium would be the best way. There is a test running there right now, but I can stop it if you want to upgrade it now. Otherwise, tomorrow morning UTC would work.
Mentioned in SAL (#wikimedia-operations) [2021-09-16T08:04:48Z] <moritzm> upgrading scandium to PHP 7.2 backport of patch for enhanced DOM replaceChild/removeChild performance T291052
Mentioned in SAL (#wikimedia-operations) [2021-09-20T07:31:31Z] <moritzm> uploaded PHP 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 to apt.wikimedia.org (component/php7.2 for buster-wikimedia) T291052
We'll first roll out on our canaries and 5 parsoid servers, and continue with full roll out tomorrow.
Mentioned in SAL (#wikimedia-operations) [2021-09-20T16:41:06Z] <effie> upgrading php on wtp[1025-1029] to 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 - T291052
Mentioned in SAL (#wikimedia-operations) [2021-09-21T13:18:52Z] <effie> upgrading php on wtp* servers to 7.2.34-18+0~20210223.60+debian10~1.gbpb21322+wmf2 && rolling service restart - T291052
Mentioned in SAL (#wikimedia-operations) [2021-09-21T15:26:15Z] <effie> upgrade php7.2 on app-canaries and restart service - T291052
Mentioned in SAL (#wikimedia-operations) [2021-09-22T08:46:11Z] <effie> upgrade php7.2 on api-canaries and restart service - T291052
Mentioned in SAL (#wikimedia-operations) [2021-09-23T13:53:19Z] <effie> upgrade php7.2 on codfw - T291052
Change 723545 had a related patch set uploaded (by Effie Mouzeli; author: Effie Mouzeli):
[operations/docker-images/production-images@master] php7.2 images: include php7.2 patch (T291052)
Change 723545 merged by Effie Mouzeli:
[operations/docker-images/production-images@master] php7.2 images: include php7.2 patch (T291052)
Mentioned in SAL (#wikimedia-operations) [2021-09-27T06:07:17Z] <effie> upgrade php7.2 in eqiad - T291052
Mentioned in SAL (#wikimedia-operations) [2021-09-27T06:13:18Z] <effie> rolling restart php-fpm in eqiad - T291052
Just as a side note, the number of timeouts on parsoid went from ~ 5k/day before the change to ~ 3k/day or less afterwards. That's a 40% decrease in the number of timeouts, which is impressive.
Logstash: https://logstash.wikimedia.org/goto/393664cb3a4d7a228c0e0ed3ae0d0e99
This bug fix is definitely one of the reasons, but we also rolled out a number of other perf tweaks / improvements that Tim made in Parsoid itself in the same timeframe, so it is a little hard to disentangle the sources at this point, but, nevertheless, it is pretty good indeed.