Page MenuHomePhabricator

Tracking: Direct live production traffic at Parsoid/PHP
Closed, ResolvedPublic

Description

This is just a tracking task with blocking subtasks to track all the work that needs to happen before we can deploy to production

Here is the current deployment timeline:

  • Oct 29 - Nov 4: Parsoid/PHP deployed to the Parsoid cluster to process all changeprop traffic (mirrored from Parsoid/JS) from 10% to 100%.
  • Nov 19: Parsoid/PHP deployed to testwiki and test2wki (VE, Content Translation use Parsoid/PHP).
  • Nov 21: Parsoid/PHP deployed to mediawiki.org (VE uses Parsoid/PHP).
  • Nov25: Parsoid/PHP to be deployed to all private wikis (VE, Flow, Content Translation use Parsoid/PHP). An attempt on Nov 21 was rolled back because of T238849.
  • Nov 26: Enable Parsoid/PHP use for Flow on mediawiki.org
  • Nov 26: Parsoid/PHP to be deployed to group 0 and group 1 wikis (VE, Content Translation , Media Content Service use Parsoid/PHP). We have effectively deployed to all group 0 wikis since all the other closed wikis will have no impact from the switchover.
  • Dec 2: Flow on all group 0 and 1 wikis use Parsoid/PHP

Deployments have been really smooth so far, so, we'll probably finish group 2 wiki deployment on Dec 2 in multiple steps within a few hours:

  • Step 1: Enable Parsoid/PHP on zh, ru, sr wikis. This lets us exercise language variant code. We need MCS / Android product folks ready to test and verify. Based on crashers in production for language variant requests, we reverted the switchover on zh, sr wikis.
  • Step 2: Direct all language variant requests to Parsoid/JS always. This will let us fix the crashers and performance on it separately. Re-enable Parsoid/PHP on sr and zh wikis.
  • Step 3: Enable Parsoid/PHP on everything else but enwiki. Maybe wait for about an hour or less.
  • Step 4: Enable Parsoid/PHP on enwiki.
  • Step 5: Enable Flow everywhere (group 0 and group 1 previously)

At this point, Parsoid/PHP is enabled for all products except Linter.

  • Enabling linter posts from Parsoid/PHP (which will temporarily introduce duplicate lint posts -- verify we do the right thing) and disable linting from Parsoid/JS
  • Direct language variant requests to Parsoid/PHP -- completed Dec 18

At this point, Parsoid/JS will be operational in mirror / shadow mode for the really unlikely possibility of need to switch back to Parsoid/JS for some product somewhere.

  • Create a phab task to decommission Parsoid/JS (T241207)

Details

SubjectRepoBranchLines +/-
operations/mediawiki-configmaster+0 -2
operations/mediawiki-configmaster+1 -7
operations/mediawiki-configmaster+2 -4
mediawiki/services/restbase/deploymaster+3 -3
mediawiki/services/restbase/deploymaster+3 -13
mediawiki/services/restbase/deploymaster+1 -1
mediawiki/services/restbase/deploymaster+1 -1
operations/mediawiki-configmaster+2 -1
mediawiki/services/restbase/deploymaster+10 -1
operations/mediawiki-configmaster+1 -0
operations/mediawiki-configmaster+1 -2
operations/mediawiki-configmaster+2 -1
operations/mediawiki-configmaster+1 -1
operations/mediawiki-configmaster+9 -1
mediawiki/services/restbase/deploymaster+1 -0
mediawiki/corewmf/1.35.0-wmf.5+4 -0
mediawiki/coremaster+4 -0
mediawiki/services/restbase/deploymaster+7 -3
mediawiki/services/restbase/deploymaster+3 -1
mediawiki/services/parsoid/deploymaster+1 -1
operations/mediawiki-configmaster+3 -0
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
Resolvedssastry
Resolved mobrovac
DeclinedNone
ResolvedEevans
ResolvedEvanProdromou
DeclinedEvanProdromou
ResolvedTgr
ResolvedEvanProdromou
Resolved Pchelolo
ResolvedEvanProdromou
Resolvedtstarling
Resolvedtstarling
ResolvedEvanProdromou
Resolvedtstarling
ResolvedAnomie
ResolvedTgr
DuplicateNone
ResolvedEvanProdromou
ResolvedEvanProdromou
Resolvedtstarling
ResolvedEvanProdromou
ResolvedTgr
ResolvedTgr
ResolvedTgr
ResolvedBPirkle
ResolvedBPirkle
DeclinedEvanProdromou
DeclinedEvanProdromou
ResolvedEvanProdromou
ResolvedTgr
Resolvedsbassett
Resolvedkostajh
Resolvedssastry
Resolvedssastry
Resolvedssastry
DuplicateArlolra
Resolvedssastry
ResolvedGoalssastry
Resolvedssastry
ResolvedJoe
Resolvedssastry
Resolvedssastry
ResolvedArlolra
ResolvedDzahn
Resolvedssastry
ResolvedNone
ResolvedBUG REPORTssastry
Resolvedssastry
Resolvedssastry
Resolvedssastry
DuplicateNone
Resolvedcscott
Resolvedssastry
Resolved mobrovac
ResolvedDzahn
ResolvedJoe
ResolvedPRODUCTION ERRORDzahn
Resolved mobrovac
DuplicateNone
Resolved mobrovac
Resolved mobrovac
Resolvedssastry
Resolved Pchelolo
Resolved Pchelolo
Resolved Pchelolo
OpenNone
Resolvedcscott
Resolvedcscott
Resolvedssastry
ResolvedPRODUCTION ERRORcscott
ResolvedPRODUCTION ERRORssastry
ResolvedPRODUCTION ERRORcscott
ResolvedSbailey
Resolved mobrovac
ResolvedArlolra
InvalidSpikeNone
Resolved mobrovac
Resolvedssastry
ResolvedArlolra
Resolved mobrovac
Resolvedssastry
Resolvedssastry
Resolvedcscott
DuplicateArlolra
ResolvedPRODUCTION ERRORMSantos

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change 553034 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Switch group 0 and 1 to Parsoid/PHP

https://gerrit.wikimedia.org/r/553034

Change 553034 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Switch group 0 and 1 to Parsoid/PHP

https://gerrit.wikimedia.org/r/553034

Mentioned in SAL (#wikimedia-operations) [2019-11-26T17:10:06Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@0b74625]: Switch group 0 and 1 to Parsoid-PHP T229015

Mentioned in SAL (#wikimedia-operations) [2019-11-26T17:25:43Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@0b74625]: Switch group 0 and 1 to Parsoid-PHP T229015 (duration: 15m 38s)

Mentioned in SAL (#wikimedia-operations) [2019-12-02T12:59:31Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@eedba38]: Parsoid Proxy: Fixes - T229015

Change 554056 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Switch ru, sr and zh wikipedias to Parsoid/PHP

https://gerrit.wikimedia.org/r/554056

Mentioned in SAL (#wikimedia-operations) [2019-12-02T13:14:20Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@eedba38]: Parsoid Proxy: Fixes - T229015 (duration: 14m 49s)

Change 552972 merged by jenkins-bot:
[operations/mediawiki-config@master] Parsoid: Switch groups 0 and 1 to Parsoid/PHP

https://gerrit.wikimedia.org/r/552972

Mentioned in SAL (#wikimedia-operations) [2019-12-02T15:38:27Z] <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Parsoid VRS: Switch groups 0 and 1 to Parsoid/PHP - T229015 (duration: 00m 59s)

Change 554056 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Switch ru, sr and zh wikipedias to Parsoid/PHP

https://gerrit.wikimedia.org/r/554056

Mentioned in SAL (#wikimedia-operations) [2019-12-02T16:27:45Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@3516382]: Switch ru, sr and zh wikipediae to Parsoid/PHP - T229015

Mentioned in SAL (#wikimedia-operations) [2019-12-02T16:41:38Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@3516382]: Switch ru, sr and zh wikipediae to Parsoid/PHP - T229015 (duration: 13m 53s)

Change 554112 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Switch sr and zh wp back to Parsoid/PHP

https://gerrit.wikimedia.org/r/554112

Change 554112 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Switch sr and zh wp back to Parsoid/PHP

https://gerrit.wikimedia.org/r/554112

Mentioned in SAL (#wikimedia-operations) [2019-12-02T17:42:43Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@ff7862f]: Switch sr and zh wikipediae back to Parsoid/JS - T229015

Mentioned in SAL (#wikimedia-operations) [2019-12-02T17:56:49Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@ff7862f]: Switch sr and zh wikipediae back to Parsoid/JS - T229015 (duration: 14m 06s)

Mentioned in SAL (#wikimedia-operations) [2019-12-02T18:50:04Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@6a24685]: Parsoid Proxy: Direct html2html traffic to JS; Stop honouring the variant header; Switch sr and zh wikis to PHP - T229015

Mentioned in SAL (#wikimedia-operations) [2019-12-02T19:04:15Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@6a24685]: Parsoid Proxy: Direct html2html traffic to JS; Stop honouring the variant header; Switch sr and zh wikis to PHP - T229015 (duration: 14m 11s)

Change 554129 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Switch everything but enwiki to Parsoid/PHP

https://gerrit.wikimedia.org/r/554129

Change 554129 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Switch everything but enwiki to Parsoid/PHP

https://gerrit.wikimedia.org/r/554129

Mentioned in SAL (#wikimedia-operations) [2019-12-02T19:23:39Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@e69e2e5]: Switch everything but enwiki to Parsoid/PHP - T229015

Mentioned in SAL (#wikimedia-operations) [2019-12-02T19:37:27Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@e69e2e5]: Switch everything but enwiki to Parsoid/PHP - T229015 (duration: 13m 48s)

Change 554136 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Switch everything to Parsoid/PHP

https://gerrit.wikimedia.org/r/554136

Change 554136 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Switch everything to Parsoid/PHP

https://gerrit.wikimedia.org/r/554136

Mentioned in SAL (#wikimedia-operations) [2019-12-02T20:06:58Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@92acf1e]: Switch everything to Parsoid/PHP - T229015

Mentioned in SAL (#wikimedia-operations) [2019-12-02T20:21:57Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@92acf1e]: Switch everything to Parsoid/PHP - T229015 (duration: 14m 59s)

Change 554146 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/mediawiki-config@master] Parsoid VRS: Switch all clients to Parsoid/PHP

https://gerrit.wikimedia.org/r/554146

Change 554146 merged by Mobrovac:
[operations/mediawiki-config@master] Parsoid VRS: Switch all clients to Parsoid/PHP

https://gerrit.wikimedia.org/r/554146

Mentioned in SAL (#wikimedia-operations) [2019-12-02T20:36:14Z] <mobrovac@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Switch Flow on all wikis to Parsoid/PHP - T229015 (duration: 00m 59s)

Mentioned in SAL (#wikimedia-operations) [2019-12-03T11:58:02Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@b346ebf]: Mirror html2html traffic to Parsoid/PHP - T229015 T239643

Mentioned in SAL (#wikimedia-operations) [2019-12-03T11:58:10Z] <mobrovac@deploy1001> deploy aborted: Mirror html2html traffic to Parsoid/PHP - T229015 T239643 (duration: 00m 00s)

Mentioned in SAL (#wikimedia-operations) [2019-12-03T11:58:33Z] <mobrovac@deploy1001> Started deploy [restbase/deploy@b346ebf]: Mirror html2html traffic to Parsoid/PHP - T229015 T239643

Mentioned in SAL (#wikimedia-operations) [2019-12-03T12:12:02Z] <mobrovac@deploy1001> Finished deploy [restbase/deploy@b346ebf]: Mirror html2html traffic to Parsoid/PHP - T229015 T239643 (duration: 13m 29s)

Mentioned in SAL (#wikimedia-operations) [2019-12-18T17:18:02Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@c9d8ef1]: Parsoid-PHP: mirror 100% of all traffic T229015

Mentioned in SAL (#wikimedia-operations) [2019-12-18T17:33:59Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@c9d8ef1]: Parsoid-PHP: mirror 100% of all traffic T229015 (duration: 16m 00s)

Mentioned in SAL (#wikimedia-operations) [2019-12-18T20:36:47Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@6e24349]: Disable all parsoid-php vs parsoid-js special cases T229015

Mentioned in SAL (#wikimedia-operations) [2019-12-18T20:50:43Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@6e24349]: Disable all parsoid-php vs parsoid-js special cases T229015 (duration: 13m 56s)

ssastry claimed this task.
ssastry updated the task description. (Show Details)

This is now all done. The unresolved subtasks can be handled independently and should be updated to remove this as the parent task.

Change 579005 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[operations/mediawiki-config@master] Remove no-longer-necessary $wmgParsoidVariant

https://gerrit.wikimedia.org/r/579005

Change 579020 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] ProductionServices: Stop defining the 'parsoid' JS service, unused

https://gerrit.wikimedia.org/r/579020

Change 579005 merged by jenkins-bot:
[operations/mediawiki-config@master] Remove no-longer-necessary $wmgParsoidVariant

https://gerrit.wikimedia.org/r/579005

Mentioned in SAL (#wikimedia-operations) [2020-03-11T18:21:19Z] <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Stop using wmgParsoidVariant, no longer varied T229015 (duration: 01m 08s)

Mentioned in SAL (#wikimedia-operations) [2020-03-11T18:22:52Z] <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Stop setting wmgParsoidVariant, no longer read T229015 (duration: 01m 07s)

Change 579020 abandoned by Jforrester:
ProductionServices: Stop defining the 'parsoid' JS service, unused

Reason:
Going with I7c2f114add instead.

https://gerrit.wikimedia.org/r/579020