Page MenuHomePhabricator
Create Task
Maniphest T235902

Tracking: Shadow Parsoid/PHP deployment to production cluster to handle mirrored reparse traffic
Closed, ResolvedPublic
Actions

Description

This is a tracker task with a bunch of subtasks. The plan is to enable this the week of Oct 21 at the earliest possible date. We will deploy the latest Parsoid code to the production cluster on Monday, but RESTBase will separate deploy a configuration change to mirror changeprop requests to Parsoid/PHP besides Parsoid/JS.

Details

SubjectRepoBranchLines +/-
Bump Parsoid-PHP traffic mirroring to 25%mediawiki/services/restbase/deploymaster+1 -1
Mirror 10% of /page/html/ traffic to Parsoid/PHPmediawiki/services/restbase/deploymaster+2 -3
Mirror 10% of production traffic to Parsoid-PHPmediawiki/services/restbase/deploymaster+1 -1
Beta: Mirror all traffic to Parsoid/PHPmediawiki/services/restbase/deploymaster+1 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedssastryT229015 Tracking: Direct live production traffic at Parsoid/PHP
 ResolvedssastryT235902 Tracking: Shadow Parsoid/PHP deployment to production cluster to handle mirrored reparse traffic
 ResolvedssastryT235899 Direct all Parsoid/PHP logs from wtp* servers to the parsoid-php channel instead of the mediawiki channel
 ResolvedssastryT232182 Parsoid/PHP performance benchmarking on scandium / eqiad cluster
 DuplicateNoneT229025 Add ability to RESTBase to partition client traffic to Parsoid/PHP
 ResolvedcscottT235897 Enable performance metrics logging
 ResolvedssastryT235900 Create a Parsoid-PHP kibana dashboard to monitor logs
 Resolved• mobrovacT235898 Enable Parsoid extension loading on all wtp* cluster servers
 ResolvedDzahnT233654 Make the parsoid cluster support parsoid/PHP
 ResolvedJoeT233722 Set up LVS for parsoid/PHP
 ResolvedPRODUCTION ERRORDzahnT237304 EasyTimeline extension shell error
 Resolved• mobrovacT230791 Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP
 DuplicateNoneT235375 Parsoid/PHP responds with 404 for titles with slashes and revision
 Resolved• mobrovacT236382 Parsoid/PHP HTML output for PageBundle should contain Content-Language and Vary headers
 Resolved• mobrovacT236388 Linting is disabled on beta cluster, but needs to be enabled
 ResolvedssastryT236428 Update the Parsoid grafana dashboards to monitor Parsoid/PHP performance
 Resolved• PcheloloT236836 RESTBase warning missing Content-Language or Vary header in pb.body.html.headers
 Resolved• PcheloloT236837 High rate of 412 responses from Parsoid-PHP
 Resolved• PcheloloT236838 RESTBase mirror mode for Parsoid-PHP doesn't honor storage

Event Timeline

ssastry triaged this task as High priority.Oct 18 2019, 5:13 PM
ssastry created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 18 2019, 5:13 PM
ssastry added subtasks: T235899: Direct all Parsoid/PHP logs from wtp* servers to the parsoid-php channel instead of the mediawiki channel, T232182: Parsoid/PHP performance benchmarking on scandium / eqiad cluster, T227209: Security Review For Parsoid-PHP, T229025: Add ability to RESTBase to partition client traffic to Parsoid/PHP, T229018: RESTBase should be able to store Parsoid/PHP contents in Cassandra alongwith Parsoid/JS contents, T235897: Enable performance metrics logging, T235900: Create a Parsoid-PHP kibana dashboard to monitor logs.Oct 18 2019, 5:15 PM
ssastry added a subtask: T235898: Enable Parsoid extension loading on all wtp* cluster servers.
ssastry added a parent task: T229015: Tracking: Direct live production traffic at Parsoid/PHP.Oct 18 2019, 5:19 PM
ssastry removed a subtask: T227209: Security Review For Parsoid-PHP.
ssastry removed a subtask: T229018: RESTBase should be able to store Parsoid/PHP contents in Cassandra alongwith Parsoid/JS contents.
ssastry mentioned this in T229018: RESTBase should be able to store Parsoid/PHP contents in Cassandra alongwith Parsoid/JS contents.Oct 18 2019, 5:21 PM
ssastry moved this task from Backlog to Deployment on the Parsoid-PHP board.Oct 18 2019, 5:24 PM
Dzahn added a subtask: T233654: Make the parsoid cluster support parsoid/PHP.Oct 18 2019, 5:31 PM
• mobrovac added a subtask: T230791: Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP.Oct 18 2019, 5:58 PM
• mobrovac added a comment.Oct 18 2019, 6:02 PM
This comment was removed by • mobrovac.
ssastry renamed this task from Tracking: Shadow Parsoid/PHP deployment to production cluster to handle changeprop reparse traffic to Tracking: Shadow Parsoid/PHP deployment to production cluster to handle mirrored reparse traffic.Oct 18 2019, 7:24 PM
ssastry updated the task description. (Show Details)
ssastry mentioned this in T227209: Security Review For Parsoid-PHP.Oct 18 2019, 10:47 PM
Dzahn added a comment.EditedOct 18 2019, 11:08 PM

All wtp servers have been converted to parsoid-php-appservers and got MediaWiki with this change:

https://gerrit.wikimedia.org/r/c/operations/puppet/+/544232

Lots of new monitoring, that is all green: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=wtp

They are all pooled in conftool for the "parsoid-php" service.

also see T233654#5588312

ssastry closed subtask T232182: Parsoid/PHP performance benchmarking on scandium / eqiad cluster as Resolved.Oct 21 2019, 4:07 PM
• mobrovac closed subtask T235898: Enable Parsoid extension loading on all wtp* cluster servers as Resolved.Oct 24 2019, 6:02 PM
ssastry closed subtask T235897: Enable performance metrics logging as Resolved.Oct 24 2019, 6:03 PM
• mobrovac changed the status of subtask T230791: Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP from Stalled to Open.Oct 24 2019, 6:22 PM
gerritbot added a comment.Oct 28 2019, 7:31 AM

Change 546395 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Beta: Mirror all traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546395

gerritbot added a project: Patch-For-Review.Oct 28 2019, 7:31 AM

Change 546395 merged by Mobrovac:
[mediawiki/services/restbase/deploy@master] Beta: Mirror all traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546395

• mobrovac mentioned this in rGRBD528b8930f031: Beta: Mirror all traffic to Parsoid/PHP.Oct 28 2019, 7:33 AM
gerritbot added a comment.Oct 28 2019, 7:37 AM

Change 546397 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/restbase/deploy@master] Mirror 10% of /page/html/ traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546397

• mobrovac closed subtask T230791: Have a Mechanism for Storing and Retrieving Parsoid HTML from JS and PHP as Resolved.Oct 28 2019, 10:04 AM
• mobrovac added projects: RESTBase, Platform Team Workboards (Clinic Duty Team).Oct 28 2019, 10:12 AM
• mobrovac added a subscriber: • Pchelolo.

@ssastry RESTBase is production is now able to mirror traffic to Parsoid/PHP (it is already mirroring all the traffic in Beta). However, this is currently disabled. I put up Gerrit 546397 which sets RESTBase to mirror 10% of /page/html/ traffic (including both live and update traffic) to Parsoid/PHP. If you want to activate mirroring today, please sync up with @Pchelolo to do so, otherwise I can do it tomorrow EU morning.

• mobrovac moved this task from Inbox to Doing(WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.Oct 28 2019, 10:15 AM

We'll also have to figure out the pace at which to ramp up the percentage. I would suggest to go with 10% today, and if all is good tomorrow, ramp up to 25% followed by 50% on Wednesday. I would stop there for the week and go to full 100% Monday if Friday all is looking good from both sides.

Note that I am off Wednesday - Friday, and so @ssastry and @Pchelolo, you two will need to coordinate for the schedule of Wednesday and Thursday.

gerritbot added a comment.Oct 28 2019, 7:11 PM

Change 546681 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Mirror 10% of production traffic to Parsoid-PHP

https://gerrit.wikimedia.org/r/546681

gerritbot added a comment.Oct 29 2019, 2:21 PM

Change 546681 abandoned by Ppchelko:
Mirror 10% of production traffic to Parsoid-PHP

https://gerrit.wikimedia.org/r/546681

ssastry closed subtask T235899: Direct all Parsoid/PHP logs from wtp* servers to the parsoid-php channel instead of the mediawiki channel as Resolved.Oct 29 2019, 4:04 PM
Dzahn closed subtask T233654: Make the parsoid cluster support parsoid/PHP as Resolved.Oct 29 2019, 5:45 PM
gerritbot added a comment.Oct 29 2019, 6:05 PM

Change 546397 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Mirror 10% of /page/html/ traffic to Parsoid/PHP

https://gerrit.wikimedia.org/r/546397

• Pchelolo mentioned this in rGRBDcf8013098e52: Mirror 10% of /page/html/ traffic to Parsoid/PHP.Oct 29 2019, 6:05 PM
Stashbot added a comment.Oct 29 2019, 6:07 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-29T18:07:37Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@cf80130]: Mirror 10% of /page/html/ traffic to Parsoid/PHP T235902

Maintenance_bot removed a project: Patch-For-Review.Oct 29 2019, 6:10 PM
Stashbot added a comment.Oct 29 2019, 6:21 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-29T18:21:49Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@cf80130]: Mirror 10% of /page/html/ traffic to Parsoid/PHP T235902 (duration: 14m 13s)

ssastry closed subtask T235900: Create a Parsoid-PHP kibana dashboard to monitor logs as Resolved.Oct 29 2019, 7:09 PM
ssastry closed subtask T236428: Update the Parsoid grafana dashboards to monitor Parsoid/PHP performance as Resolved.Oct 29 2019, 9:46 PM
• Pchelolo closed subtask T236836: RESTBase warning missing Content-Language or Vary header in pb.body.html.headers as Resolved.Oct 30 2019, 4:18 PM
• Pchelolo closed subtask T236838: RESTBase mirror mode for Parsoid-PHP doesn't honor storage as Resolved.Oct 30 2019, 9:39 PM
gerritbot added a comment.Oct 30 2019, 9:53 PM

Change 547324 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Bump Parsoid-PHP traffic mirroring to 25%

https://gerrit.wikimedia.org/r/547324

gerritbot added a project: Patch-For-Review.Oct 30 2019, 9:53 PM

Change 547324 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Bump Parsoid-PHP traffic mirroring to 25%

https://gerrit.wikimedia.org/r/547324

Stashbot added a comment.Oct 30 2019, 9:55 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-30T21:55:05Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@fa934c8]: Bump parsoid mirroring to 25% and fix 412: T235902, T236837

Stashbot mentioned this in T236837: High rate of 412 responses from Parsoid-PHP.Oct 30 2019, 9:55 PM
• Pchelolo mentioned this in rGRBDfa934c8ffa69: Bump Parsoid-PHP traffic mirroring to 25%.Oct 30 2019, 9:55 PM
Stashbot added a comment.Oct 30 2019, 10:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-30T22:08:59Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@fa934c8]: Bump parsoid mirroring to 25% and fix 412: T235902, T236837 (duration: 13m 54s)

Maintenance_bot removed a project: Patch-For-Review.Oct 30 2019, 10:10 PM
• Pchelolo closed subtask T236837: High rate of 412 responses from Parsoid-PHP as Resolved.Oct 30 2019, 10:11 PM
• Pchelolo mentioned this in rGRBD9cac9acf75be: Bump Parsoid-PHP traffic mirroring to 50%.Oct 31 2019, 7:11 PM
Stashbot added a comment.Oct 31 2019, 9:18 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-31T21:18:51Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@9cac9ac]: Bump Parsoid-PHP traffic mirroring to 50% T235902

Stashbot added a comment.Oct 31 2019, 9:32 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-31T21:32:35Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@9cac9ac]: Bump Parsoid-PHP traffic mirroring to 50% T235902 (duration: 13m 44s)

Stashbot added a comment.Nov 4 2019, 6:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-04T18:09:34Z] <ppchelko@deploy1001> Started deploy [restbase/deploy@20c710d]: Bump Parsoid-PHP mirroring to 100% T235902

• Pchelolo mentioned this in rGRBD20c710d0a52b: Bump Parsoid-PHP traffic mirroring to 100%.Nov 4 2019, 6:17 PM
Stashbot added a comment.Nov 4 2019, 6:24 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-04T18:24:03Z] <ppchelko@deploy1001> Finished deploy [restbase/deploy@20c710d]: Bump Parsoid-PHP mirroring to 100% T235902 (duration: 14m 30s)

ssastry closed this task as Resolved.Nov 4 2019, 9:26 PM
ssastry claimed this task.

We are now at 100% and all looks good (error logs, performance, cluster load) wise. Resolving.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL