Page MenuHomePhabricator

Tracking: Shadow Parsoid/PHP deployment to production cluster to handle mirrored reparse traffic
Closed, ResolvedPublic

Description

This is a tracker task with a bunch of subtasks. The plan is to enable this the week of Oct 21 at the earliest possible date. We will deploy the latest Parsoid code to the production cluster on Monday, but RESTBase will separate deploy a configuration change to mirror changeprop requests to Parsoid/PHP besides Parsoid/JS.

Details

Related Gerrit Patches:
mediawiki/services/restbase/deploy : masterBump Parsoid-PHP traffic mirroring to 25%
mediawiki/services/restbase/deploy : masterMirror 10% of /page/html/ traffic to Parsoid/PHP
mediawiki/services/restbase/deploy : masterMirror 10% of production traffic to Parsoid-PHP
mediawiki/services/restbase/deploy : masterBeta: Mirror all traffic to Parsoid/PHP

Related Objects

StatusAssignedTask
OpenNone
Resolvedssastry
Resolvedssastry
Resolvedssastry
DuplicateNone
Resolvedcscott
Resolvedssastry
Resolvedmobrovac
ResolvedDzahn
ResolvedJoe
ResolvedDzahn
Resolvedmobrovac
DuplicateNone
Resolvedmobrovac
Resolvedmobrovac
Resolvedssastry
ResolvedPchelolo
ResolvedPchelolo
ResolvedPchelolo

Event Timeline

ssastry triaged this task as High priority.Fri, Oct 18, 5:13 PM
ssastry created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFri, Oct 18, 5:13 PM
ssastry moved this task from Backlog to Deployment on the Parsoid-PHP board.Fri, Oct 18, 5:24 PM
This comment was removed by mobrovac.
ssastry renamed this task from Tracking: Shadow Parsoid/PHP deployment to production cluster to handle changeprop reparse traffic to Tracking: Shadow Parsoid/PHP deployment to production cluster to handle mirrored reparse traffic.Fri, Oct 18, 7:24 PM
ssastry updated the task description. (Show Details)
Dzahn added a comment.EditedFri, Oct 18, 11:08 PM

All wtp servers have been converted to parsoid-php-appservers and got MediaWiki with this change:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/544232

Lots of new monitoring, that is all green: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=wtp

They are all pooled in conftool for the "parsoid-php" service.

also see T233654#5588312

Change 546395 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Beta: Mirror all traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546395

Change 546395 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Beta: Mirror all traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546395

Change 546397 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Mirror 10% of /page/html/ traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546397

mobrovac added a subscriber: Pchelolo.

@ssastry RESTBase is production is now able to mirror traffic to Parsoid/PHP (it is already mirroring all the traffic in Beta). However, this is currently disabled. I put up Gerrit 546397 which sets RESTBase to mirror 10% of /page/html/ traffic (including both live and update traffic) to Parsoid/PHP. If you want to activate mirroring today, please sync up with @Pchelolo to do so, otherwise I can do it tomorrow EU morning.

We'll also have to figure out the pace at which to ramp up the percentage. I would suggest to go with 10% today, and if all is good tomorrow, ramp up to 25% followed by 50% on Wednesday. I would stop there for the week and go to full 100% Monday if Friday all is looking good from both sides.

Note that I am off Wednesday - Friday, and so @ssastry and @Pchelolo, you two will need to coordinate for the schedule of Wednesday and Thursday.

Change 546681 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Mirror 10% of production traffic to Parsoid-PHP

https://gerrit.wikimedia.org/r/546681

Change 546681 abandoned by Ppchelko:
Mirror 10% of production traffic to Parsoid-PHP

https://gerrit.wikimedia.org/r/546681

Change 546397 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Mirror 10% of /page/html/ traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546397

Mentioned in SAL (#wikimedia-operations) [2019-10-29T18:07:37Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@cf80130]: Mirror 10% of /page/html/ traffic to Parsoid/PHP T235902

Mentioned in SAL (#wikimedia-operations) [2019-10-29T18:21:49Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@cf80130]: Mirror 10% of /page/html/ traffic to Parsoid/PHP T235902 (duration: 14m 13s)

Change 547324 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Bump Parsoid-PHP traffic mirroring to 25%

https://gerrit.wikimedia.org/r/547324

Change 547324 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Bump Parsoid-PHP traffic mirroring to 25%

https://gerrit.wikimedia.org/r/547324

Mentioned in SAL (#wikimedia-operations) [2019-10-30T21:55:05Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@fa934c8]: Bump parsoid mirroring to 25% and fix 412: T235902, T236837

Mentioned in SAL (#wikimedia-operations) [2019-10-30T22:08:59Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@fa934c8]: Bump parsoid mirroring to 25% and fix 412: T235902, T236837 (duration: 13m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-10-31T21:18:51Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@9cac9ac]: Bump Parsoid-PHP traffic mirroring to 50% T235902

Mentioned in SAL (#wikimedia-operations) [2019-10-31T21:32:35Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@9cac9ac]: Bump Parsoid-PHP traffic mirroring to 50% T235902 (duration: 13m 44s)

Mentioned in SAL (#wikimedia-operations) [2019-11-04T18:09:34Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@20c710d]: Bump Parsoid-PHP mirroring to 100% T235902

Mentioned in SAL (#wikimedia-operations) [2019-11-04T18:24:03Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@20c710d]: Bump Parsoid-PHP mirroring to 100% T235902 (duration: 14m 30s)

ssastry closed this task as Resolved.Mon, Nov 4, 9:26 PM
ssastry claimed this task.

We are now at 100% and all looks good (error logs, performance, cluster load) wise. Resolving.