Page MenuHomePhabricator

Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters
Open, NormalPublic

Description

We need to ramp-up usage of php7 to 100% both on the appserver and the API cluster.

This means progressively changing the sampling rate on both clusters to 5,10, 20, 50 and 100%.

For the API cluster, we will convert each API server to serve only via php7.2-fpm. Each API server serves ~2% of traffic.

Once that's done, and we're confident we won't need to revert, we will need to cleanup:

  • Remove the conditional in the virtual hosts, send all requests to php by default.
  • Remove all the code from the WikimediaEvents extension that sets the cookie for this transition
  • Remove any code in apache/varnish that does set the cookie for API requests
Record of going live for appservers:
LevelWent liveGit change
0.1%2019.03.255a270bbebf
1%2019.04.3088984d4d20
5%2019.05.028afada66e1
10%2019.05.21cee99d4ca6
20%2019.05.284ffc48ff58
Proportion of serversWent liveServer(s)Git Change
1/512019.08.01mw1270e0d83ea6
Record of going live for API cluster:
eqiad
Proportion of serversWent liveServer(s)Git Change
1/472019.05.30mw134835b2eca
2/472019.07.30mw134755e5091
5/512019.08.08mw122[1-3]845752ac43fc
10/512019.08.12mw122[4-8}1a7073eef400
codfw
Proportion of serversWent liveServer(s)Git Change
1/512019.05.30mw2135e13905d
2/512019.07.30mw213655e5091

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-04-30T14:44:15Z] <jijiki> Sending 1% of anonymous users to PHP7.2 - T219150

Change 506953 merged by Giuseppe Lavagetto:
[operations/mediawiki-config@master] Send 1% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/506953

Mentioned in SAL (#wikimedia-operations) [2019-04-30T15:09:58Z] <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Send 1% of anonymous users to PHP7.2 - T219150 (duration: 00m 54s)

Change 507741 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] Send 5% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/507741

Change 507741 merged by jenkins-bot:
[operations/mediawiki-config@master] Send 5% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/507741

Mentioned in SAL (#wikimedia-operations) [2019-05-02T08:49:04Z] <jijiki> Sending more traffic to PHP7.2 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-05-02T08:55:21Z] <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Send 5% of anonymous users to PHP7.2 - T219150 (duration: 01m 03s)

jijiki updated the task description. (Show Details)May 3 2019, 10:41 AM

Change 507939 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: if guard php72_only blocks

https://gerrit.wikimedia.org/r/507939

Change 511731 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Move 10% of traffic to php7

https://gerrit.wikimedia.org/r/511731

Change 511731 merged by jenkins-bot:
[operations/mediawiki-config@master] Move 10% of traffic to php7

https://gerrit.wikimedia.org/r/511731

Mentioned in SAL (#wikimedia-operations) [2019-05-21T15:37:16Z] <oblivian@deploy1001> Synchronized wmf-config/CommonSettings.php: Moving to 10% of users on php7 T219150 (duration: 00m 57s)

Change 512803 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] Send 20% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/512803

Change 512803 merged by jenkins-bot:
[operations/mediawiki-config@master] Send 20% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/512803

Mentioned in SAL (#wikimedia-operations) [2019-05-28T06:40:54Z] <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Send 20% of anonymous users to PHP7.2 - T219150 (duration: 00m 51s)

Mentioned in SAL (#wikimedia-operations) [2019-05-30T11:00:08Z] <jijiki> Disable puppet on mw* servers to merge 507939 - T219150

Change 507939 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: if guard php72_only blocks

https://gerrit.wikimedia.org/r/507939

Change 513273 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw2135

https://gerrit.wikimedia.org/r/513273

Change 513273 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw2135

https://gerrit.wikimedia.org/r/513273

Change 513277 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw2135

https://gerrit.wikimedia.org/r/513277

Change 513277 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw1348

https://gerrit.wikimedia.org/r/513277

Jdforrester-WMF updated the task description. (Show Details)
Fito added a subscriber: Fito.Jun 28 2019, 12:38 PM
Daimona added a subscriber: Daimona.Jul 3 2019, 9:31 AM

Mentioned in SAL (#wikimedia-operations) [2019-07-08T15:12:21Z] <jiji@deploy1001> Started deploy [cpjobqueue/deploy@7379e91]: Migrating refreshLinks to PHP7 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-07-08T15:13:46Z] <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@7379e91]: Migrating refreshLinks to PHP7 - T219150 (duration: 01m 26s)

Mentioned in SAL (#wikimedia-operations) [2019-07-09T14:28:46Z] <jiji@deploy1001> Started deploy [cpjobqueue/deploy@8517fec]: Migrating cirrus* jobs to PHP7 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-07-09T14:29:48Z] <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@8517fec]: Migrating cirrus* jobs to PHP7 - T219150 (duration: 01m 02s)

Mentioned in SAL (#wikimedia-operations) [2019-07-10T19:04:32Z] <jiji@deploy1001> Started deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-07-10T19:05:32Z] <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150 (duration: 01m 00s)

Mentioned in SAL (#wikimedia-operations) [2019-07-16T16:35:46Z] <jiji@deploy1001> Started deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-07-16T16:36:37Z] <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150 (duration: 00m 50s)

jijiki moved this task from Backlog to Next up on the serviceops board.Fri, Jul 26, 10:27 AM

Change 526434 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw1347 and mw2136

https://gerrit.wikimedia.org/r/526434

Mentioned in SAL (#wikimedia-operations) [2019-07-30T15:18:28Z] <jijiki> Disable puppet on mw1347 and mw2136, depool and pool back - T219150

Change 526434 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw1347 and mw2136

https://gerrit.wikimedia.org/r/526434

Joe updated the task description. (Show Details)Thu, Aug 1, 6:48 AM
Joe updated the task description. (Show Details)
Joe updated the task description. (Show Details)
jijiki added a comment.Tue, Aug 6, 8:56 AM

We switched mw1270 to PHP7 but we came across the following issues

  • a 10% increase in the median and 75% percentiles
  • we filled up apc, which resulted in severe degradation of performance

Looking at logs yielded that php-fpm was spawning new workers which is due to us using the processmanager dynamic but that's an expensive procedure we don't really need at full steam, so we switched it to static. Also it seemed we had some queued requests in php-fpm, so I raised the number of workers for appservers from 1.5 * ncores to 2 * ncores

Change 529061 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw122[1-3]

https://gerrit.wikimedia.org/r/529061

Change 529061 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw122[1-3]

https://gerrit.wikimedia.org/r/529061

jijiki updated the task description. (Show Details)Thu, Aug 8, 11:22 AM

Change 529796 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw122[4-8]

https://gerrit.wikimedia.org/r/529796

Change 529796 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw122[4-8]

https://gerrit.wikimedia.org/r/529796

Mentioned in SAL (#wikimedia-operations) [2019-08-12T15:05:31Z] <jijiki> rolling restat php-fpm on mw122[4-8] - T219150

jijiki updated the task description. (Show Details)Mon, Aug 12, 3:18 PM
jijiki updated the task description. (Show Details)
jijiki moved this task from Next up to Doing on the serviceops board.Tue, Aug 13, 8:32 AM

Change 529924 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] Send 33.3% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/529924