Page MenuHomePhabricator
Create Task
Maniphest T89066

Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service on group1/group2 wikis
Closed, ResolvedPublic
Actions

Description

The Virtual Rest Service for Parsoid content currently uses the Parsoid v1 API. Once RESTBase is deployed and well tested we should start switching this over to RESTBase, so that VisualEditor, Flow and other consumers can profit from the smaller HTML without inline data-parsoid & faster load times.

For a smooth migration we should support both backends & have the option to switch between them using a config flag in MediaWiki. Both infrastructures (RESTBase and Parsoid Varnishes) will be available in parallel during the transition period.

This is also a good opportunity to clean up some things in this area:

  • share / generalize the config & setup for VRS & Parsoid / RESTBase: factory function / singleton? (Could perhaps take a similar approach Ori took in his statsd patch.)
  • Adjust URL schema to current restbase layout (/page/html/{title}/{revision}) and mount it at /local/ for now. In the longer term, we'll want to support a domain at that level, but for now we only need to support /local/.
  • eventually, remove the /parsoid/ prefix (requires some changes in the VRS base class)

Details

SubjectRepoBranchLines +/-
Disable restbase on wikipediasoperations/mediawiki-configmaster+0 -1
Use RESTBase with VisualEditor on frwikioperations/mediawiki-configmaster+1 -0
Enable RESTBase for visual editing on itwiki and plwiki as welloperations/mediawiki-configmaster+2 -0
Use the RESTBase back-end if availablemediawiki/extensions/VisualEditormaster+53 -12
Introduce the Restbase Virtual REST Service classmediawiki/coremaster+213 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolved• GWickeT144814 Services Team Goals 2016/2017 Q2: October - December
 Resolved• mobrovacT136942 Services Team Goals July - September 2016 (Q1 2016-17)
 Resolved• mobrovacT118871 Services team goals April - June 2016 (Q4 2015/16)
 ResolvedNoneT118868 Services team goals January - March 2016 (Q3 2015/16)
 Resolved• GWickeT111819 Services team goals October - December 2015 (Q2 2015/16)
 Resolved• GWickeT102306 Services team roadmap July - September 2015 (Q1 2015/16)
 ResolvedNoneT92468 Services Roadmap April - June 2015 (Q4 2014/2015)
 Resolved• GWickeT91533 Services team Q3 (Jan - March 2015) quarterly goal tracking
 Resolved• mobrovacT89066 Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service on group1/group2 wikis
 Resolved• GWickeT89481 RESTBase beta release (revision storage / content API)
 Resolved• GWickeT1228 RESTbase deployment
 ResolvedfgiunchediT76986 RESTBase production hardware
 ResolvedfgiunchediT88805 rack and setup restbase production cluster in eqiad
 Resolved• CmjohnsonT83551 Split memcached in eqiad across multiple racks/rows
 Resolved• CmjohnsonT82259 mc1016 mgmt not working
 ResolvedJoeT89345 Missing memcached logs
 ResolvedfgiunchediT89639 restbase1006 faulty disk controller
 Resolved• GWickeT75955 RESTBase / Parsoid integration - waiting for parsoid deploy
 Resolved• JdouglasT88456 On-demand generation of HTML and data-parsoid
 Resolved• GWickeT88457 html2wt conversion support
 Resolved• GWickeT88588 Formalize the RESTBase v1 specification
 ResolvedArlolraT89411 Parsoid v2 API selser might not be working correctly
 Resolved• GWickeT76370 RESTBase performance testing
 Resolved• mobrovacT76165 Handle revision deletion and suppression in RESTBase
 Resolved• mobrovacT87520 Set up update jobs for RESTBase: code done and deployed, to be configured
 Resolved• mobrovacT88652 Integrate revisions into /page structure. Was: Create a /page/revision/{revision} entry point
 Resolved• mobrovacT87392 Add property & index on namespace in page_revisions
 Duplicate• JdouglasT78214 Documentation location and deployment process
 ResolvedAndrewT88850 Import Cassandra packages for Jessie
 Resolved• GWickeT89461 trebuchet puppet provider broken on systems without upstart
 ResolvedfgiunchediT89636 setup internal LVS for restbase eqiad servers
 ResolvedfgiunchediT78514 Detailed cassandra monitoring: metrics and dashboards done, need to set up alerts
 ResolvedfgiunchediT93886 Cassandra/CQL query interface monitoring
 ResolvedEevansT93884 graphs for Cassandra metrics
 ResolvedfgiunchediT97024 some cassandra metrics sent with invalid values
 Resolved• GWickeT78194 Public entry point for RESTBase
 Resolved• GWickeT89567 Load test production cluster
 ResolvedfgiunchediT89366 Access to restbase / cassandra cluster
 Resolved• mobrovacT91102 Update / maintain Beta Cluster restbase cluster: Up & working with VE
 Resolved• GWickeT91188 Need to change data model for >~300 wikis in single cassandra cluster

Event Timeline

• GWicke created this task.Feb 10 2015, 1:21 AM
• GWicke raised the priority of this task from to Needs Triage.
• GWicke updated the task description. (Show Details)
• GWicke added a project: MediaWiki-General.
• GWicke subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2015, 1:21 AM
• GWicke updated the task description. (Show Details)Feb 10 2015, 1:23 AM
• GWicke added projects: RESTBase, VisualEditor, Performance Issue.
• GWicke set Security to None.
• GWicke added a project: Services.
• GWicke triaged this task as Medium priority.Feb 10 2015, 1:25 AM
Jdforrester-WMF added a subtask: T78194: Public entry point for RESTBase.Feb 10 2015, 1:26 AM
• MZMcBride subscribed.Feb 10 2015, 1:41 AM
Liuxinyu970226 subscribed.Feb 10 2015, 1:55 AM
• GWicke added a subtask: T1228: RESTbase deployment.Feb 10 2015, 2:28 AM
Jdforrester-WMF moved this task from To Triage to Freezer on the VisualEditor board.Feb 11 2015, 5:25 PM
• GWicke edited subtasks, added: T89481: RESTBase beta release (revision storage / content API); removed: T1228: RESTbase deployment, T78194: Public entry point for RESTBase.Feb 13 2015, 7:01 PM
• GWicke mentioned this in T90374: Retrieve page HTML directly from RESTbase.Feb 23 2015, 6:56 PM
• GWicke added a subscriber: • mobrovac.EditedFeb 26 2015, 4:57 PM

Some recent discussion on the old gerrit patch for parsoid integration: https://gerrit.wikimedia.org/r/#/c/181011/12/ApiVisualEditor.php

• GWicke raised the priority of this task from Medium to High.Feb 26 2015, 5:03 PM
• GWicke updated the task description. (Show Details)
• GWicke updated the task description. (Show Details)Feb 26 2015, 5:10 PM
Krenair subscribed.Feb 26 2015, 9:38 PM
• mobrovac claimed this task.Feb 27 2015, 2:11 PM
• GWicke updated the task description. (Show Details)Feb 27 2015, 4:34 PM
greg subscribed.Feb 27 2015, 6:22 PM

Where are we on the timeline for this? (Deployment calendar says "late feb early march")

• mobrovac added a comment.Feb 27 2015, 6:25 PM

@greg should be right on time. We are contemplating a possible multi-step solution where the first step would be to refactor things in core so that it's possible to switch from Parsoid to RESTBase without making changes in VE itself. This is somewhat a messier solution, but I'd allow us to start testing VE with RESTBase pretty soon.

gerritbot subscribed.Mar 2 2015, 2:48 PM

Change 193826 had a related patch set uploaded (by Mobrovac):
Introduce the Restbase Virtual REST Service class

https://gerrit.wikimedia.org/r/193826

gerritbot added a project: Patch-For-Review.Mar 2 2015, 2:48 PM
gerritbot added a comment.Mar 4 2015, 3:02 PM

Change 194338 had a related patch set uploaded (by Mobrovac):
Use the RESTBase back-end if available

https://gerrit.wikimedia.org/r/194338

• GWicke added a parent task: T91533: Services team Q3 (Jan - March 2015) quarterly goal tracking.Mar 4 2015, 6:41 PM
• GWicke mentioned this in T91533: Services team Q3 (Jan - March 2015) quarterly goal tracking.Mar 4 2015, 6:51 PM
• mobrovac mentioned this in T91661: Have a RESTbase vagrant role.Mar 5 2015, 5:24 PM
• GWicke mentioned this in T89481: RESTBase beta release (revision storage / content API).Mar 5 2015, 9:16 PM
greg added a project: Roadmap.Mar 6 2015, 7:00 PM
greg moved this task from Unscheduled to March 2015: Platform on the Roadmap board.
greg moved this task from March 2015: Platform to March 9-13 on the Roadmap board.
• GWicke added a comment.Mar 6 2015, 7:03 PM

@mobrovac, I'll amend your patch to address some of the comments today.

I hope that we can get this in early next week, so that we can test with VE in the second half of the week.

Eloquence renamed this task from Use RESTBase from the MediaWiki Virtual Rest Service to VisualEditor performance: Use RESTBase from the MediaWiki Virtual Rest Service.Mar 6 2015, 7:37 PM
Jdforrester-WMF subscribed.Mar 6 2015, 7:57 PM

This is about all Parsoid HTML consumers of the MediaWiki core service, not just VisualEditor, isn't it? Flow, etc.

Eloquence renamed this task from VisualEditor performance: Use RESTBase from the MediaWiki Virtual Rest Service to Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service.Mar 7 2015, 7:09 PM
• mobrovac added a comment.Mar 8 2015, 8:24 PM

The patches for MW core and for VE have been tested on a local MW-Vagrant install and work flawlessly: VE gets the HTML from RESTBase on load and passes it back for transformation.

gerritbot added a comment.Mar 10 2015, 5:08 PM

Change 193826 merged by jenkins-bot:
Introduce the Restbase Virtual REST Service class

https://gerrit.wikimedia.org/r/193826

• mobrovac mentioned this in rMWb13bd0996cbd: Introduce the Restbase Virtual REST Service class.Mar 10 2015, 5:08 PM
Jdforrester-WMF added a project: WMF-deploy-2015-03-11_(1.25wmf21).Mar 10 2015, 9:46 PM
Jdforrester-WMF closed this task as Resolved.Mar 10 2015, 10:23 PM
Jdforrester-WMF edited projects, added VisualEditor 2014/15 Q3 blockers; removed Patch-For-Review.
gerritbot added a comment.Mar 10 2015, 10:23 PM

Change 194338 merged by jenkins-bot:
Use the RESTBase back-end if available

https://gerrit.wikimedia.org/r/194338

• mobrovac mentioned this in rEVEDb46a2bd105b3: Use the RESTBase back-end if available.Mar 10 2015, 10:24 PM
Jdforrester-WMF moved this task from Nominated to Done on the VisualEditor 2014/15 Q3 blockers board.Mar 10 2015, 10:24 PM
Liuxinyu970226 unsubscribed.Mar 10 2015, 11:36 PM
• GWicke reopened this task as Open.EditedMar 12 2015, 12:21 AM

Re-opening as this is not fully done yet.

The integration is now being tested on https://test.wikipedia.org/. Things are working fine, although performance on test.wikipedia.org seems to be a bit mixed. VE initializes quickly using RESTBase-provided content, but saves seem to be relatively slow in both VE and the wikitext editor.

Next step (possibly tomorrow) is to enable this across phase0 wikis, eventually followed by real production.

• GWicke closed subtask T89481: RESTBase beta release (revision storage / content API) as Resolved.EditedMar 13 2015, 1:33 AM

VE is now using RESTBase on all group0 wikis including https://www.mediawiki.org. Further roll-out to follow next week.

• GWicke renamed this task from Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service to Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service: phase0 done, others to follow.Mar 13 2015, 1:36 AM
Eloquence renamed this task from Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service: phase0 done, others to follow to Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service on phase1/phase2 wikis.Mar 13 2015, 4:47 PM
Eloquence moved this task from March 9-13 to March 16-20 on the Roadmap board.
Krenair renamed this task from Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service on phase1/phase2 wikis to Parsoid performance: Use RESTBase from the MediaWiki Virtual Rest Service on group1/group2 wikis.Mar 13 2015, 5:56 PM
Ricordisamoa subscribed.Mar 13 2015, 11:05 PM
• GWicke mentioned this in T86648: Upgrade all HTTP frontends to Debian jessie.Mar 15 2015, 4:01 PM
• GWicke added a comment.EditedMar 16 2015, 6:13 PM

Further roll-out is starting now:

  1. ptwiki and ruwiki: done
  2. itwiki, plwiki: done
  3. frwiki: done
  4. remaining wikipedias: done

Edit: It turns out that none of these actually had an effect, as the VE extension in wmf20 did not include the RESTBase integration yet. It is going to be updated with the train on Wednesday.

gerritbot added a comment.Mar 16 2015, 6:20 PM

Change 197104 had a related patch set uploaded (by GWicke):
Enable RESTBase for visual editing on itwiki and plwiki as well

https://gerrit.wikimedia.org/r/197104

gerritbot added a project: Patch-For-Review.Mar 16 2015, 6:20 PM
gerritbot added a comment.Mar 16 2015, 6:26 PM

Change 197104 merged by jenkins-bot:
Enable RESTBase for visual editing on itwiki and plwiki as well

https://gerrit.wikimedia.org/r/197104

gerritbot added a comment.Mar 16 2015, 6:40 PM

Change 197108 had a related patch set uploaded (by GWicke):
Use RESTBase with VisualEditor on frwiki

https://gerrit.wikimedia.org/r/197108

gerritbot added a comment.Mar 16 2015, 6:43 PM

Change 197108 merged by jenkins-bot:
Use RESTBase with VisualEditor on frwiki

https://gerrit.wikimedia.org/r/197108

• GWicke added a comment.EditedMar 16 2015, 11:31 PM

It turns out that the VE extension deployed in wmf20 does not actually have the restbase integration code yet, so the config changes on *.wikipedia.org were basically no-ops.

• GWicke added a comment.Mar 16 2015, 11:43 PM

Patch to disable the setting until the train deploy on Wednesday updates the VE extension to wmf21 at https://gerrit.wikimedia.org/r/#/c/197250/.

gerritbot added a comment.Mar 17 2015, 2:38 PM

Change 197250 had a related patch set uploaded (by Anomie):
Disable restbase on wikipedias

https://gerrit.wikimedia.org/r/197250

gerritbot added a comment.Mar 17 2015, 3:15 PM

Change 197250 merged by jenkins-bot:
Disable restbase on wikipedias

https://gerrit.wikimedia.org/r/197250

• GWicke moved this task from Backlog to In-progress on the Services board.Mar 17 2015, 8:03 PM
• GWicke moved this task from Backlog to In progress on the RESTBase board.Mar 17 2015, 8:21 PM
• GWicke added a comment.EditedMar 19 2015, 5:56 PM

As of now RESTBase is powering VisualEditor on all wikipedias. First performance metrics are looking promising:

pasted_file (663×1 px, 127 KB)

• GWicke closed this task as Resolved.Mar 19 2015, 6:02 PM

Lets track group1 in a follow-up task. Closing this task as resolved.

Ricordisamoa removed a project: Patch-For-Review.Mar 19 2015, 8:06 PM
Eloquence moved this task from March 16-20 to March 2015: Platform on the Roadmap board.Mar 20 2015, 2:36 AM
• GWicke mentioned this in rOMWCa6ea4ae596d6: Use RESTBase for visual editing on ruwiki and ptwiki.Mar 24 2015, 5:40 PM
• GWicke mentioned this in rOMWC2529b1c0596e: Enable RESTBase for visual editing on itwiki and plwiki as well.
• GWicke mentioned this in rOMWC05a37965c877: Use RESTBase with VisualEditor on frwiki.
Anomie mentioned this in rOMWC41fb1b94102e: Disable restbase on wikipedias.Mar 24 2015, 5:40 PM
• mobrovac moved this task from In-progress to Done on the Services board.Jun 23 2015, 8:49 AM
Restricted Application added a project: Notice. · View Herald TranscriptJun 23 2015, 8:49 AM
gpaumier moved this task from Backlog to Triaged on the Notice board.Jun 25 2015, 3:13 PM
gpaumier moved this task from Triaged to Archive on the Notice board.Jun 26 2015, 8:27 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL