Install php 7.4 in production
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Joe
	Jun 27 2022, 8:46 AM

Details

Subject	Repo	Branch	Lines +/-
lvs: check php 7.4 too on all appservers	operations/puppet	production	+9 -1
mediawiki: allow forcing the backend for blank page on wikipedia	operations/puppet	production	+18 -1
jobrunner: allow selecting explicitly the backend when performing health checks.	operations/puppet	production	+32 -0
mediawiki: install php7.4 on all appservers	operations/puppet	production	+16 -31
mediawiki: install php 7.4 on all maintenance servers	operations/puppet	production	+3 -3
mediawiki: install php7.4 on the maintenance server	operations/puppet	production	+3 -0
mediawiki: install php7.4 on jobrunners	operations/puppet	production	+5 -2
mediawiki: install php7.4 on the canaries	operations/puppet	production	+34 -0
mediawiki: install php7.4 on the mwdebug servers	operations/puppet	production	+3 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		None	T248925 Make MediaWiki release tarball compatible with PHP 8.0
Resolved		Jdforrester-WMF	T300463 Make PHP 8.0 voting on MW master
Resolved		Jdforrester-WMF	T313563 Bump lcobucci/jwt & league/uri-components for php8
Resolved		Jdforrester-WMF	T313564 Bump onoi/message-reporter in vendor.git to 1.4.2 for php 8 support
Resolved		Jdforrester-WMF	T247658 Make Wikimedia Production MediaWiki compatible with PHP 7.4
Resolved		• toan	T243590 Fix WikibaseDataModel CI for php 7.4
Resolved		Lucas_Werkmeister_WMDE	T316923 Restore skipped test in ReferenceListTest.php
Resolved		Joe	T318918 Undeploy patch to use old PHP serialization in PHP 7.4
Resolved		Jdforrester-WMF	T264168 Drop PHP 7.2 support from Wikibase master branch, once Wikimedia production is on 7.4
Resolved		Ladsgroup	T270740 Drop hacky support for doctrine/dbal class renaming
Invalid		None	T303505 [S] Remove Deprecated methods "serialize" and "unserialize" after php production upgrade to PHP 7.4
Resolved		Reedy	T251043 Cleanup css-sanitizer when it only requires PHP >= 7.4
Open		None	T166010 The Great Namespaceization Effort
Resolved		• tstarling	T277618 var_dump() on various objects writes gigabytes of data and takes minutes to run
Resolved		Jdforrester-WMF	T261872 Drop PHP 7.2 & 7.3 support from MediaWiki master branch, once Wikimedia production is on 7.4
Resolved	PRODUCTION ERROR	Legoktm	T293568 PHP Notice: Undefined offset in wikimedia/remex-html when rendering rest.php error page
Resolved		• tstarling	T297667 mysqli/mysqlnd memory leak
Resolved		Joe	T271736 Migrate WMF production from PHP 7.2 to PHP 7.4
Resolved		Joe	T311386 Install php 7.4 in production
Resolved		Joe	T312634 Properly support php7.4 across the observability stack
Resolved		Joe	T313505 Remove prometheus-phpfpm-exporter from physical hosts

Event Timeline

Joe created this task.Jun 27 2022, 8:46 AM

Change 808908 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] mediawiki: install php7.4 on the mwdebug servers

https://gerrit.wikimedia.org/r/808908

Change 808909 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] mediawiki: install php7.4 on the canaries

https://gerrit.wikimedia.org/r/808909

Change 808910 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] mediawiki: install php7.4 on jobrunners

https://gerrit.wikimedia.org/r/808910

Change 808911 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] mediawiki: install php7.4 on the maintenance server

https://gerrit.wikimedia.org/r/808911

Change 808912 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] mediawiki: install php7.4 on all appservers

https://gerrit.wikimedia.org/r/808912

I have updated our PHP build in component/php74 to the latest 7.4.x release; 7.4.30. Uploaded as 7.4.30-3+0~20220627.69+debian10~1.gbpf2b381+wmf1+buster1

Change 808908 merged by Giuseppe Lavagetto:

[operations/puppet@production] mediawiki: install php7.4 on the mwdebug servers

https://gerrit.wikimedia.org/r/808908

Sgs unsubscribed.Jun 28 2022, 11:21 AM

Joe updated the task description. (Show Details)Jun 28 2022, 4:02 PM

I've installed php 7.4 on the mwdebug servers. Modulo a series of small issues with resource ordering, and the package referencing a bullseye+ only script, the process went well.

I'll run some tests next, and also probably modify X-Wikimedia-Debug to be able to reach php 7.4 too.

Ladsgroup awarded a token.Jun 28 2022, 8:46 PM

@Krinkle noticed that mwdebug1001 was not actually serving pages with PHP 7.4 when requested via a cookie. I isolated the issue, reproduced it locally, and tested a fix.

/etc/apache2/conf-available/50-fcgi-proxies.conf has

# Configure the proxies

# Proxy for php version 7.2
<Proxy "unix:/run/php/fpm-www.sock|fcgi://localhost">
       ProxySet retry=0
</Proxy>
# Proxy for php version 7.4
<Proxy "unix:/run/php/fpm-www-7.4.sock|fcgi://localhost">
       ProxySet retry=0
</Proxy>

Only the part of the URL after the pipe is used for matching, so both of these Proxy workers match every PHP request. The first one is used. Then when the request is dispatched, it uses the UNIX path defined by the worker, not the one from the request.

My proposed solution is to use a single worker, configured similarly to the default worker. Omit the UNIX socket path, to ensure that worker->s->uds_path is null, since this triggers detection of the UNIX socket path from the request information. It's also necessary to disable connection reuse, otherwise the second request that comes in will use the same connection as the first, regardless of the desired UNIX socket path.

<Proxy "fcgi://localhost">
       ProxySet retry=0 disablereuse=on
</Proxy>

Also I noticed that the socket path is incorrect. In Apache it is /run/php/fpm-www-7.4.sock, but the actual socket defined by the PHP configuration is /run/php/fpm-www-7-4.sock, with a hyphen between 7 and 4. I prefer the one with the dot because that's how you spell version numbers. The issue with Proxy directives hid this issue.

Thanks @tstarling in fact we only tested tcp proxying in beta. I've gone a slightly different way than you suggested, adding an additional worker with a different name instead than just disabling reuse of it, which would be a performance penalty.

So now our proxies section looks like:

# Configure the proxies

# Proxy for php version 7.2
<Proxy "unix:/run/php/fpm-www.sock|fcgi://localhost">
       ProxySet retry=0
</Proxy>
# Proxy for php version 7.4
<Proxy "unix:/run/php/fpm-www-7.4.sock|fcgi://www-7.4">
       ProxySet retry=0
</Proxy>

and the corresponding routing in the vhosts is:

# Default: no/invalid cookie or default version
<If "reqenv('backend') in  {'', '7.2'}">
    SetHandler "proxy:unix:/run/php/fpm-www.sock|fcgi://localhost"
</If>
# Version 7.4
<ElseIf "reqenv('backend') == '7.4'">
    SetHandler "proxy:unix:/run/php/fpm-www-7.4.sock|fcgi://www-7.4"
</ElseIf>

Mentioned in SAL (#wikimedia-operations) [2022-06-30T13:00:33Z] <moritzm> uploaded php-defaults 76+wmf1+buster2 for component/php74 (drops a Breaks: on php72-common) T311386

Mentioned in SAL (#wikimedia-operations) [2022-06-30T13:05:48Z] <moritzm> upgrade mwdebug* servers to 2:76+wmf1~buster2 T311386

Change 810312 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] mediawiki: allow forcing the backend for blank page on wikipedia

https://gerrit.wikimedia.org/r/810312

Change 810313 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] lvs: check php 7.4 too on all appservers

https://gerrit.wikimedia.org/r/810313

Change 810348 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] jobrunner: allow selecting explicitly the backend when performing health checks.

https://gerrit.wikimedia.org/r/810348

Change 808909 merged by Giuseppe Lavagetto:

[operations/puppet@production] mediawiki: install php7.4 on the canaries

https://gerrit.wikimedia.org/r/808909

Mentioned in SAL (#wikimedia-operations) [2022-07-05T08:30:41Z] <moritzm> uploaded 7.4.30-3+0~20220627.69+debian10~1.gbpf2b381+wmf1+buster3 to component/php74 (pulling php-common with the socket helper) T311386

Joe updated the task description. (Show Details)Jul 5 2022, 2:57 PM

After the installation on the mediawiki canaries, I ran our standard "benchmw" set of benchmarks on both php 7.2 and 7.4 in codfw. In general we see better performance from php 7.4 especially when we do things that require a lot of php computations, like re-parsing.

Specifically, see

percentiles_a_reparse_heavy_c20.png (1×1 px, 26 KB)

for re-parsing of a heavy page. In this case, php 7.4 performs 25% better actoss pretty much all percentiles.

For stuff that requires a less heavy work, like a page view of english wikipedia, the difference is more like 10% in favour of php 7.4

percentiles_a_view_mainpage_c20.png (1×1 px, 22 KB)

A similar gain holds for all pageviews, see also a lighter wiki page view

percentiles_a_view_short_c20.png (1×1 px, 22 KB)

and a startup JS bundle produced by load.php still shows a significant gain

percentiles_a_rl_startup_c20.png (1×1 px, 23 KB)

while as one would expect the gain for loading a plain css file via load.php is smaller and mostly negligible

percentiles_a_rl_css_c20.png (1×1 px, 23 KB)

Change 808910 merged by Giuseppe Lavagetto:

[operations/puppet@production] mediawiki: install php7.4 on jobrunners

https://gerrit.wikimedia.org/r/808910

Change 808911 merged by Giuseppe Lavagetto:

[operations/puppet@production] mediawiki: install php7.4 on the maintenance server

https://gerrit.wikimedia.org/r/808911

Change 811585 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] mediawiki: install php 7.4 on all maintenance servers

https://gerrit.wikimedia.org/r/811585

Change 811585 merged by Giuseppe Lavagetto:

[operations/puppet@production] mediawiki: install php 7.4 on all maintenance servers

https://gerrit.wikimedia.org/r/811585

Change 808912 merged by Giuseppe Lavagetto:

[operations/puppet@production] mediawiki: install php7.4 on all appservers

https://gerrit.wikimedia.org/r/808912

Joe updated the task description. (Show Details)Jul 6 2022, 9:14 AM

Change 810348 merged by Giuseppe Lavagetto:

[operations/puppet@production] jobrunner: allow selecting explicitly the backend when performing health checks.

https://gerrit.wikimedia.org/r/810348

• brennen mentioned this in T313770: scap no longer restarts php-fpm on canary servers.Jul 25 2022, 10:33 PM

Change 810312 merged by Giuseppe Lavagetto:

[operations/puppet@production] mediawiki: allow forcing the backend for blank page on wikipedia

https://gerrit.wikimedia.org/r/810312

Change 810313 merged by Giuseppe Lavagetto:

[operations/puppet@production] lvs: check php 7.4 too on all appservers

https://gerrit.wikimedia.org/r/810313

Joe closed subtask T312634: Properly support php7.4 across the observability stack as Resolved.Aug 8 2022, 6:35 AM

Joe closed this task as Resolved.Aug 19 2022, 7:56 AM

	F35307462: percentiles_a_rl_startup_c20.png
	Jul 5 2022, 5:18 PM

	F35307457: percentiles_a_view_short_c20.png
	Jul 5 2022, 5:18 PM

	F35307449: percentiles_a_view_mainpage_c20.png
	Jul 5 2022, 5:18 PM

	F35307442: percentiles_a_reparse_heavy_c20.png
	Jul 5 2022, 5:18 PM

Install php 7.4 in productionClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Install php 7.4 in production
Closed, ResolvedPublic
Actions

Related Objects
Search...