Page MenuHomePhabricator

Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters
Closed, ResolvedPublic

Description

We need to ramp-up usage of php7 to 100% both on the appserver and the API cluster.

This means progressively changing the sampling rate on both clusters to 5,10, 20, 50 and 100%.

For the API cluster, we will convert each API server to serve only via php7.2-fpm. Each API server serves ~2% of traffic.

Once that's done, and we're confident we won't need to revert, we will need to cleanup:

  • Remove the conditional in the virtual hosts, send all requests to php by default.
  • Remove all the code from the WikimediaEvents extension that sets the cookie for this transition
  • Remove any code in apache/varnish that does set the cookie for API requests
Record of going live for appservers:
LevelWent liveGit change
0.1%2019.03.255a270bbebf
1%2019.04.3088984d4d20
5%2019.05.028afada66e1
10%2019.05.21cee99d4ca6
20%2019.05.284ffc48ff58
33%2019.08.221b3990ef77
20%2019.08.227efa56c1f0
33%2019.09.09559c8afb19
50%2019.09.11fa81b83d78
100%2019.09.1711d6db5d7e
Proportion of serversWent liveServer(s)Git Change
1/532019.08.01mw1270e0d83ea6
53/532019.09.24
Record of going live for API cluster:
eqiad
Proportion of serversWent liveServer(s)Git Change
1/472019.05.30mw134835b2eca
2/472019.07.30mw134755e5091
5/512019.08.08mw122[1-3]845752ac43fc
10/512019.08.12mw122[4-8}1a7073eef400
51/512019.09.24
codfw
Proportion of serversWent liveServer(s)Git Change
1/512019.05.30mw2135e13905d
2/512019.07.30mw213655e5091

Details

Related Gerrit Patches:
operations/mediawiki-config : master50% of anonymous users via PHP7.2
operations/puppet : productionmediawiki: move all appservers to PHP7 as well.
operations/puppet : productionConvert 100% of API servers to only serve PHP7.2
operations/puppet : productionConvert 25% of app servers to only serve PHP7.2
operations/puppet : productionConvert 50% of API servers to only serve PHP7.2
operations/mediawiki-config : master100% of anonymous users via PHP7.2
operations/mediawiki-config : master50% of anonymous users via PHP7.2
operations/mediawiki-config : master33.3% of anonymous users via PHP7.2
operations/mediawiki-config : masterSend 33.3% of anonymous users to PHP7.2
operations/mediawiki-config : masterRevert "Send 33.3% of anonymous users to PHP7.2"
operations/mediawiki-config : masterSend 33.3% of anonymous users to PHP7.2
operations/puppet : productionhieradata: enable php72_only on mw122[4-8]
operations/puppet : productionhieradata: enable php72_only on mw122[1-3]
operations/puppet : productionhieradata: enable php72_only on mw1347 and mw2136
operations/puppet : productionhieradata: enable php72_only on mw1348
operations/puppet : productionhieradata: enable php72_only on mw2135
operations/puppet : productionmediawiki: if guard php72_only blocks
operations/mediawiki-config : masterSend 20% of anonymous users to PHP7.2
operations/mediawiki-config : masterMove 10% of traffic to php7
operations/mediawiki-config : masterSend 5% of anonymous users to PHP7.2
operations/mediawiki-config : masterSend 1% of anonymous users to PHP7.2

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-07-09T14:28:46Z] <jiji@deploy1001> Started deploy [cpjobqueue/deploy@8517fec]: Migrating cirrus* jobs to PHP7 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-07-09T14:29:48Z] <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@8517fec]: Migrating cirrus* jobs to PHP7 - T219150 (duration: 01m 02s)

Mentioned in SAL (#wikimedia-operations) [2019-07-10T19:04:32Z] <jiji@deploy1001> Started deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-07-10T19:05:32Z] <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@8761480]: Migrating rest of hightraffic jobs to PHP7 - T219150 (duration: 01m 00s)

Mentioned in SAL (#wikimedia-operations) [2019-07-16T16:35:46Z] <jiji@deploy1001> Started deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150

Mentioned in SAL (#wikimedia-operations) [2019-07-16T16:36:37Z] <jiji@deploy1001> Finished deploy [cpjobqueue/deploy@5d8128e]: Migrating videoscaling jobs to PHP7 - T219150 (duration: 00m 50s)

jijiki moved this task from Backlog to Next up on the serviceops board.Jul 26 2019, 10:27 AM

Change 526434 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw1347 and mw2136

https://gerrit.wikimedia.org/r/526434

Mentioned in SAL (#wikimedia-operations) [2019-07-30T15:18:28Z] <jijiki> Disable puppet on mw1347 and mw2136, depool and pool back - T219150

Change 526434 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw1347 and mw2136

https://gerrit.wikimedia.org/r/526434

Joe updated the task description. (Show Details)Aug 1 2019, 6:48 AM
Joe updated the task description. (Show Details)
Joe updated the task description. (Show Details)
jijiki added a comment.Aug 6 2019, 8:56 AM

We switched mw1270 to PHP7 but we came across the following issues

  • a 10% increase in the median and 75% percentiles
  • we filled up apc, which resulted in severe degradation of performance

Looking at logs yielded that php-fpm was spawning new workers which is due to us using the processmanager dynamic but that's an expensive procedure we don't really need at full steam, so we switched it to static. Also it seemed we had some queued requests in php-fpm, so I raised the number of workers for appservers from 1.5 * ncores to 2 * ncores

Change 529061 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw122[1-3]

https://gerrit.wikimedia.org/r/529061

Change 529061 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw122[1-3]

https://gerrit.wikimedia.org/r/529061

jijiki updated the task description. (Show Details)Aug 8 2019, 11:22 AM

Change 529796 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hieradata: enable php72_only on mw122[4-8]

https://gerrit.wikimedia.org/r/529796

Change 529796 merged by Effie Mouzeli:
[operations/puppet@production] hieradata: enable php72_only on mw122[4-8]

https://gerrit.wikimedia.org/r/529796

Mentioned in SAL (#wikimedia-operations) [2019-08-12T15:05:31Z] <jijiki> rolling restat php-fpm on mw122[4-8] - T219150

jijiki updated the task description. (Show Details)Aug 12 2019, 3:18 PM
jijiki updated the task description. (Show Details)
jijiki moved this task from Next up to Doing on the serviceops board.Aug 13 2019, 8:32 AM

Change 529924 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] Send 33.3% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/529924

Change 529924 merged by jenkins-bot:
[operations/mediawiki-config@master] Send 33.3% of anonymous users to PHP7.2

https://gerrit.wikimedia.org/r/529924

Mentioned in SAL (#wikimedia-operations) [2019-08-22T10:55:48Z] <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 33.3% - T219150 (duration: 01m 01s)

jijiki updated the task description. (Show Details)Aug 22 2019, 11:19 AM

Change 531695 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] Revert "Send 33.3% of anonymous users to PHP7.2"

https://gerrit.wikimedia.org/r/531695

Change 531695 merged by jenkins-bot:
[operations/mediawiki-config@master] Revert "Send 33.3% of anonymous users to PHP7.2"

https://gerrit.wikimedia.org/r/531695

Mentioned in SAL (#wikimedia-operations) [2019-08-22T13:32:07Z] <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Reverting PHP7 traffic back to 20% - T219150 (duration: 00m 57s)

jijiki updated the task description. (Show Details)Aug 22 2019, 1:45 PM

Change 534610 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/mediawiki-config@master] Make one user out of 3 use php7

https://gerrit.wikimedia.org/r/534610

Change 535134 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] 33.3% of anonymous users via PHP7.2

https://gerrit.wikimedia.org/r/535134

Change 534610 abandoned by Effie Mouzeli:
Send 33.3% of anonymous users to PHP7.2

Reason:
Merge conflicts, abandoned for I65b3f5a4eceb5cbf10bd4eceba50739f38f5e15c

https://gerrit.wikimedia.org/r/534610

Change 535134 merged by jenkins-bot:
[operations/mediawiki-config@master] 33.3% of anonymous users via PHP7.2

https://gerrit.wikimedia.org/r/535134

Mentioned in SAL (#wikimedia-operations) [2019-09-09T10:22:38Z] <effie> jiji@deploy1001:~$ scap sync-file wmf-config/CommonSettings.php "Push PHP7 traffic to 33.3% - T219150"

Jdforrester-WMF updated the task description. (Show Details)

Change 535770 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] 50% of anonymous users via PHP7.2

https://gerrit.wikimedia.org/r/535770

Change 535786 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] 50% of anonymous users via PHP7.2

https://gerrit.wikimedia.org/r/535786

MaxSem added a subscriber: MaxSem.Sep 11 2019, 7:37 AM

Change 535786 merged by jenkins-bot:
[operations/mediawiki-config@master] 50% of anonymous users via PHP7.2

https://gerrit.wikimedia.org/r/535786

Mentioned in SAL (#wikimedia-operations) [2019-09-11T09:46:46Z] <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 50% - T219150 (duration: 01m 03s)

Change 537328 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] 100% of anonymous users via PHP7.2

https://gerrit.wikimedia.org/r/537328

Change 537328 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/mediawiki-config@master] 100% of anonymous users via PHP7.2
https://gerrit.wikimedia.org/r/537328

💃💃💃

Change 537328 merged by jenkins-bot:
[operations/mediawiki-config@master] 100% of anonymous users via PHP7.2

https://gerrit.wikimedia.org/r/537328

Mentioned in SAL (#wikimedia-operations) [2019-09-17T09:05:23Z] <jiji@deploy1001> Synchronized wmf-config/CommonSettings.php: Push PHP7 traffic to 100% of users who accept cookies - T219150 (duration: 00m 57s)

Change 538565 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Convert 50% of API servers to only serve PHP7.2

https://gerrit.wikimedia.org/r/538565

Change 538571 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Convert 25% of API servers to only serve PHP7.2

https://gerrit.wikimedia.org/r/538571

Mentioned in SAL (#wikimedia-operations) [2019-09-23T09:02:48Z] <effie> Disable puppet and rolling restart php-fpm on mw[1312-1317,1339-1347]* - T219150

Change 538565 merged by Effie Mouzeli:
[operations/puppet@production] Convert 50% of API servers to only serve PHP7.2

https://gerrit.wikimedia.org/r/538565

Mentioned in SAL (#wikimedia-operations) [2019-09-23T11:12:58Z] <effie> Disable puppet and rolling restart of php7.2-fpm on mw[1321-1333] - T219150

Change 538571 merged by Effie Mouzeli:
[operations/puppet@production] Convert 25% of app servers to only serve PHP7.2

https://gerrit.wikimedia.org/r/538571

Change 538749 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Convert 100% of API servers to only serve PHP7.2

https://gerrit.wikimedia.org/r/538749

Mentioned in SAL (#wikimedia-operations) [2019-09-24T08:29:13Z] <effie> Disable puppet on api cluster and restart php-fpm to finish php7 migration - T219150

Change 538749 merged by Effie Mouzeli:
[operations/puppet@production] Convert 100% of API servers to only serve PHP7.2

https://gerrit.wikimedia.org/r/538749

Change 538855 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: move all appservers to PHP7 as well.

https://gerrit.wikimedia.org/r/538855

Change 538855 merged by Giuseppe Lavagetto:
[operations/puppet@production] mediawiki: move all appservers to PHP7 as well.

https://gerrit.wikimedia.org/r/538855

Mentioned in SAL (#wikimedia-operations) [2019-09-24T10:54:16Z] <_joe_> converting all appservers to php7, T219150

Mentioned in SAL (#wikimedia-operations) [2019-09-24T11:10:30Z] <_joe_> all wikis (including API) are now served by PHP7 T219150

Joe closed this task as Resolved.Sep 25 2019, 7:12 AM
Joe updated the task description. (Show Details)

Change 535770 abandoned by Jforrester:
50% of anonymous users via PHP7.2

Reason:
This got done in other patches.

https://gerrit.wikimedia.org/r/535770