Page MenuHomePhabricator
Create Task
Maniphest T320529

Configure VE backend to use Parsoid directly, instead of calling RESTbase
Closed, ResolvedPublic
Actions

Assigned To
daniel
Authored By
daniel
Oct 11 2022, 3:47 PM
Tags
Referenced Files
F37104900: Screenshot 2023-06-15 160146.png
Jun 15 2023, 2:10 PM
F37104898: Screenshot 2023-06-15 160120.png
Jun 15 2023, 2:10 PM
F37104896: Screenshot 2023-06-15 160036.png
Jun 15 2023, 2:10 PM
Restricted File
Jun 12 2023, 2:59 PM
Restricted File
Jun 12 2023, 2:59 PM
Subscribers
Aklapper
daniel
Frostly
Izno
Ladsgroup
matmarex
Neolexx
View All 9 Subscribers
Tokens
"Party Time" token, awarded by DAlangi_WMF."Like" token, awarded by Lens0021.

Description

The VisualEditor backend on WMF production sites should no longer use RESTbase.

Rationale: This will remove one of the primary consumers of the parsoid endpoints on RESTbase, and one of two reason we need to store pre-generated parsoid output on restbase (the other reason being PCS, for now).

  • beta (some) T320531
  • testwiki and labtestwiki T320531
  • officewiki
  • labswiki (aka wikitech)
  • mediawikiwiki
  • hewiki
  • small wikis
  • medium wikis
  • enwiki
  • all wikis

Relevant configuration:

$wmgUseRestbaseVRS= false;
NOTE: Notify the community via TechNews when this is happening!

Details

SubjectRepoBranchLines +/-
Switch VisualEditor to bypass RESTbase on all wikis.operations/mediawiki-configmaster+4 -24
Switch VisualEditor to not use RESTbase on English Wikipedia.operations/mediawiki-configmaster+2 -0
Switch VisualEditor to not use RESTbase on small and medium wikisoperations/mediawiki-configmaster+8 -3
Make VE on officewiki use Parsoid directlyoperations/mediawiki-configmaster+7 -0
hewiki: set VisualEditor to direct modeoperations/mediawiki-configmaster+5 -1
hewiki: enable parser cache writes for parsoid's page/html endpoint.operations/mediawiki-configmaster+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT324931 Clean up open RESTBase related tickets
 In ProgressNoneT262315 <CORE TECHNOLOGY> API Migration & RESTbase Sunset
 OpenNoneT264669 Move VE API from RESTBase into core.
 OpenNoneT344944 Move Parsoid endpoints out of RESTbase
 In ProgressNoneT328559 Replace usage of RESTbase parsoid endpoints
 ResolveddanielT320529 Configure VE backend to use Parsoid directly, instead of calling RESTbase
 ResolvedNoneT310376 Make VE Action API use Parsoid directly
 ResolvedNoneT310377 Make parsoid transformation API available internally
 ResolvedmatmarexT314565 ApiVisualEditor: Convert ApiParsoidTrait into a helper object.
 ResolvedPRODUCTION ERRORdanielT320703 VisualEditor can't save pages on de.wikipedia.beta
 ResolveddanielT320704 When VisualEditor saves an edit, it should use the same backend that the HTML was originally loaded from
 DuplicatexSavitarT320532 Create dual backend mode for VE API
 ResolveddanielT320536 Ensure that stashing backend for the VE API has sufficient capacity
 ResolveddanielT309016 Determine storage requirements for stashing parsoid output for VE edits
 ResolveddanielT320531 Configure VE backend to use Parsoid directly on the beta cluster and testwiki
 ResolvedKCVelaga_WMFT338634 Calculate edit session length by platform and editing interface
 ResolveddanielT338638 [SPIKE] What – if any – user-facing regressions might RESTbase deprecation have caused
 ResolveddanielT359426 REST: endpoints that return rendered page content should support all content models
 ResolveddanielT320534 Put Parsoid output into the ParserCache on every edit
 ResolveddanielT320535 Put Parsoid output into the ParserCache on the beta cluster and testwiki
 ResolveddanielT322427 Job for pre-emptive population of the ParserCache with Parsoid output
 ResolvedNoneT322672 Make ParsoidHandler::wt2html write to parser cache
 ResolvedovasilevaT326147 Stop fragmenting ParserCache entries for mobile frontend
Declined• nrayT328802 Devise Test cases for functional fallout of ParserCache fix
 ResolvedDAlangi_WMFT327164 Trigger ParsoidCachePrewarmJob on page views
 ResolvedssastryT328034 ParsoidCachePrewarmJob should set the useParsoid ParserOption
 ResolveddanielT341123 High insertion rate of ParsoidCachePrewarmJob causes substantial backlog
 ResolveddanielT329366 Enable WarmParsoidParserCache on all wikis
 ResolvedClement_GoubertT333528 Increase memory_limit for jobrunners to $wmgMemoryLimitParsoid
 ResolveddanielT333684 VE load latency degraded on hewiki
 ResolvedKrinkleT327162 RefreshLinksJob should update ParserCache

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
daniel moved this task from Doing to Blocked on the RESTBase Sunsetting board.Jan 11 2023, 5:12 PM
daniel added a parent task: T328559: Replace usage of RESTbase parsoid endpoints.Feb 1 2023, 2:22 PM
gerritbot added a comment.Mar 13 2023, 4:58 PM

Change 896104 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/mediawiki-config@master] Make VE on officewiki use Parsoid directly

https://gerrit.wikimedia.org/r/896104

gerritbot added a project: Patch-For-Review.Mar 13 2023, 4:58 PM
daniel added a subtask: T333684: VE load latency degraded on hewiki.Mar 31 2023, 1:11 PM
gerritbot added a comment.Apr 12 2023, 1:29 PM

Change 896104 merged by jenkins-bot:

[operations/mediawiki-config@master] Make VE on officewiki use Parsoid directly

https://gerrit.wikimedia.org/r/896104

Stashbot mentioned this in T333402: Switching from source editing to visual editing mode is broken with the REST API .Apr 12 2023, 1:29 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-12T13:29:31Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:896104|Make VE on officewiki use Parsoid directly (T320529 T333402)]]

Stashbot added a comment.Apr 12 2023, 1:30 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-12T13:30:54Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde and daniel: Backport for [[gerrit:896104|Make VE on officewiki use Parsoid directly (T320529 T333402)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet

Maintenance_bot removed a project: Patch-For-Review.Apr 12 2023, 1:31 PM
Stashbot added a comment.Apr 12 2023, 1:39 PM

Mentioned in SAL (#wikimedia-operations) [2023-04-12T13:39:20Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:896104|Make VE on officewiki use Parsoid directly (T320529 T333402)]] (duration: 09m 48s)

daniel updated the task description. (Show Details)May 26 2023, 2:52 PM
daniel closed subtask T333684: VE load latency degraded on hewiki as Resolved.May 26 2023, 3:03 PM
daniel added a subtask: T328034: ParsoidCachePrewarmJob should set the useParsoid ParserOption.May 26 2023, 3:05 PM
daniel closed subtask T320534: Put Parsoid output into the ParserCache on every edit as Resolved.
gerritbot added a comment.May 26 2023, 3:16 PM

Change 923650 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/mediawiki-config@master] Switch VisualEditor to not use RESTbase on small and medium wikis

https://gerrit.wikimedia.org/r/923650

gerritbot added a project: Patch-For-Review.May 26 2023, 3:16 PM
Quiddity updated the task description. (Show Details)May 26 2023, 5:32 PM
Quiddity added a project: User-notice.
Quiddity moved this task from To Triage to In current Tech/News draft on the User-notice board.
Quiddity subscribed.
Lens0021 awarded a token.May 29 2023, 4:25 AM
Izno subscribed.May 29 2023, 10:12 PM
Frostly subscribed.May 29 2023, 11:06 PM
gerritbot added a comment.May 30 2023, 8:01 AM

Change 923650 merged by jenkins-bot:

[operations/mediawiki-config@master] Switch VisualEditor to not use RESTbase on small and medium wikis

https://gerrit.wikimedia.org/r/923650

Ladsgroup added a comment.May 30 2023, 8:06 AM

I reverted this ^

Maintenance_bot removed a project: Patch-For-Review.May 30 2023, 8:10 AM
daniel updated the task description. (Show Details)May 31 2023, 1:59 PM
daniel added a comment.May 31 2023, 7:44 PM
In T320529#8887622, @Ladsgroup wrote:

I reverted this ^

For the record, the config change is good and was deployed, the reson for the revert was purely technical.
We now have VE using direct mode on all small and medium wikis.

Quiddity moved this task from In current Tech/News draft to Already announced/Archive on the User-notice board.Jun 1 2023, 5:23 PM
gerritbot added a comment.Jun 8 2023, 3:35 PM

Change 928590 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/mediawiki-config@master] Switch VisualEditor to not use RESTbase on English Wikipedia.

https://gerrit.wikimedia.org/r/928590

gerritbot added a project: Patch-For-Review.Jun 8 2023, 3:35 PM
• ppelberg subscribed.Jun 9 2023, 4:59 PM
• ppelberg mentioned this in T338638: [SPIKE] What – if any – user-facing regressions might RESTbase deprecation have caused.Jun 9 2023, 6:26 PM
gerritbot added a comment.Jun 12 2023, 1:03 PM

Change 928590 merged by jenkins-bot:

[operations/mediawiki-config@master] Switch VisualEditor to not use RESTbase on English Wikipedia.

https://gerrit.wikimedia.org/r/928590

Stashbot added a comment.Jun 12 2023, 1:04 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-12T13:04:08Z] <lucaswerkmeister-wmde@deploy1002> Started scap: Backport for [[gerrit:928590|Switch VisualEditor to not use RESTbase on English Wikipedia. (T320529)]]

Stashbot added a comment.Jun 12 2023, 1:05 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-12T13:05:28Z] <lucaswerkmeister-wmde@deploy1002> lucaswerkmeister-wmde and daniel: Backport for [[gerrit:928590|Switch VisualEditor to not use RESTbase on English Wikipedia. (T320529)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet

Maintenance_bot removed a project: Patch-For-Review.Jun 12 2023, 1:10 PM
Stashbot added a comment.Jun 12 2023, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-12T13:14:59Z] <lucaswerkmeister-wmde@deploy1002> Finished scap: Backport for [[gerrit:928590|Switch VisualEditor to not use RESTbase on English Wikipedia. (T320529)]] (duration: 10m 51s)

daniel updated the task description. (Show Details)Jun 12 2023, 1:37 PM
daniel added a comment.EditedJun 12 2023, 1:39 PM

Deployment to enwiki is looking good. Client side latency didn't change, server side performance got a good boost:

{F37101528}

{F37101529}

gerritbot added a comment.Jun 12 2023, 3:06 PM

Change 929364 had a related patch set uploaded (by Daniel Kinzler; author: Daniel Kinzler):

[operations/mediawiki-config@master] Switch VisualEditor to bypass RESTbase on all wikis.

https://gerrit.wikimedia.org/r/929364

gerritbot added a project: Patch-For-Review.Jun 12 2023, 3:06 PM
gerritbot added a comment.Jun 15 2023, 1:08 PM

Change 929364 merged by jenkins-bot:

[operations/mediawiki-config@master] Switch VisualEditor to bypass RESTbase on all wikis.

https://gerrit.wikimedia.org/r/929364

Stashbot added a comment.Jun 15 2023, 1:09 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-15T13:09:17Z] <daniel@deploy1002> Started scap: Backport for [[gerrit:929364|Switch VisualEditor to bypass RESTbase on all wikis. (T320529)]]

Maintenance_bot removed a project: Patch-For-Review.Jun 15 2023, 1:10 PM
Stashbot added a comment.Jun 15 2023, 1:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-15T13:10:41Z] <daniel@deploy1002> daniel: Backport for [[gerrit:929364|Switch VisualEditor to bypass RESTbase on all wikis. (T320529)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet

Stashbot added a comment.Jun 15 2023, 1:21 PM

Mentioned in SAL (#wikimedia-operations) [2023-06-15T13:21:06Z] <daniel@deploy1002> Finished scap: Backport for [[gerrit:929364|Switch VisualEditor to bypass RESTbase on all wikis. (T320529)]] (duration: 11m 48s)

daniel updated the task description. (Show Details)Jun 15 2023, 2:02 PM
daniel added a comment.EditedJun 15 2023, 2:10 PM

VE has been switched to direct mode on all wikis as of 13:20 UTC. So far, all metrics look good.

Screenshot 2023-06-15 160036.png (953×1 px, 199 KB)

Client side activation latency was unaffected. Utilization of the RESTbase API dropped to nearly zero, as expected. The remaining calls are from ongoing eiting sessions and should be gone within 24 hours.

Screenshot 2023-06-15 160120.png (951×1 px, 188 KB)

Server side latency of the VE API modules went down as expected, since direct mode removes the overhead of connecting to RESTbase. getPageHTML didn't benefit much got a bit slower (from 88ms up to around 100ms) since it now needs to load data from ParserCache, but HTML-to-wikitext fransformation became significantly faster, going from 270ms down to 110ms. This would benefit switchoing from visual to source mode, and should also improve save times.

Screenshot 2023-06-15 160146.png (903×1 px, 162 KB)

Stash write rates more than doubled, as expected based on the number of edits on the wikis where direct mode was previously disabled. The write rate went from about 60 to roughly 150 writes per minute. This change however barely registers on the database cluster that backs the stash: write operations on the x2 cluster remained stable at about 1500 per minute, replication lag was unaffected.

daniel closed this task as Resolved.Jun 15 2023, 2:12 PM
daniel claimed this task.
daniel updated the task description. (Show Details)
DAlangi_WMF updated the task description. (Show Details)Jun 15 2023, 2:58 PM
DAlangi_WMF awarded a token.Jun 15 2023, 3:02 PM
Neolexx subscribed.Jun 15 2023, 6:39 PM
In T320529#8935063, @daniel wrote:

VE has been switched to direct mode on all wikis as of 13:20 UTC. So far, all metrics look good.

I am curious if the phab metrics are affected by the Country protection list - there is ruwiki (one of top-10 of projects) - thus silently disregarded.

Because ever since the event - a mass problem reported at the local techforum, see (in Russian) Википедия:Форум/Технический#Визред - with VE stopped to load completely.

It look like besides other changes ve.init.Target.actionGroups stopped to work.

Or your metrics are fine and it is just a fraction of ruwiki VE users (represented at the techforum) who are affected.

daniel added a comment.Jun 15 2023, 7:12 PM
In T320529#8936419, @Neolexx wrote:

Because ever since the event - a mass problem reported at the local techforum, see (in Russian) Википедия:Форум/Технический#Визред - with VE stopped to load completely.

My metrics only show that loading VE dodn't get slower - it would not show VE failing to load.

Thank you for the report!

daniel added a comment.Jun 15 2023, 7:15 PM

@Neolexx I have just successfully made a couple of edits on Russion wikipedia using Visual Editor: https://ru.wikipedia.org/w/index.php?title=%D0%A3%D1%87%D0%B0%D1%81%D1%82%D0%BD%D0%B8%D0%BA:DKinzler_(WMF)/Sandbox&action=history

If VE is failing, please file a separate bug report. You can tag me on the report, and also link to this task if you think it is related. Please include as much information as possible in the report.

matmarex added a comment.EditedJun 15 2023, 8:02 PM

@Neolexx That seems like an unrelated problem. I think this might be caused by change https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/928049, which slightly changed how the VE toolbar is defined.

edit: Filed T339292

matmarex mentioned this in T339292: Issues with gadgets adding tools to VisualEditor "Page options" dropdown (ve.init.Target.actionGroups[1] is undefined).Jun 15 2023, 8:06 PM
Neolexx added a comment.Jun 15 2023, 8:17 PM
In T320529#8936510, @daniel wrote:

I have just successfully made a couple of edits on Russion wikipedia using Visual Editor

Sorry for troubles, it seems to be fixed even prior to my post - https://ru.wikipedia.org/w/index.php?title=MediaWiki:Gadget-registerTool.js&action=history

daniel added a comment.Jun 15 2023, 8:19 PM
In T320529#8936670, @Neolexx wrote:

Sorry for troubles, it seems to be fixed even prior to my post - https://ru.wikipedia.org/w/index.php?title=MediaWiki:Gadget-registerTool.js&action=history

No problem, thanks for reporting!

KartikMistry mentioned this in T339322: Use Parsoid in all Wikis for Content Translation.Jun 16 2023, 5:31 AM
matmarex mentioned this in T257124: Page is blank when edited a page on gomwiktionary with VisualEditor (new pages are missing on some RESTBase servers).Jun 20 2023, 1:59 PM
Maintenance_bot edited projects, added User-notice-archive; removed User-notice.Jun 30 2023, 2:31 PM
• ppelberg closed subtask T338638: [SPIKE] What – if any – user-facing regressions might RESTbase deprecation have caused as Resolved.Jul 1 2023, 12:29 AM
matmarex mentioned this in T255963: "Error contacting the Parsoid/RESTBase server (HTTP 404)" in reply tool for unknown reason.Aug 14 2023, 10:25 PM
matmarex mentioned this in T263962: "Error contacting the Parsoid/RESTBase server (HTTP 404)" in VisualEditor upon saving when a page has been deleted while you were editing it.Aug 14 2023, 10:28 PM
matmarex mentioned this in T235822: VisualEditor can't save edits (HTTP 404 error) when it was open for more than 24 hours while editing an old revision or after switching from wikitext.
daniel closed subtask T320536: Ensure that stashing backend for the VE API has sufficient capacity as Resolved.Aug 25 2023, 8:33 AM
ssastry closed subtask T328034: ParsoidCachePrewarmJob should set the useParsoid ParserOption as Resolved.Oct 26 2023, 3:17 PM
matmarex mentioned this in T261581: Request to parsoid failed on Docker Developer Environment at 80 port.Jan 8 2024, 6:46 PM
matmarex mentioned this in T261390: Zeroconf Parsoid + VisualEditor doesn't work with MediaWiki-Docker.
matmarex mentioned this in T260002: Bundled Parsoid for VisualEditor doesn't work in mediawiki-docker-dev.
matmarex mentioned this in T326026: Error contacting the Parsoid/RESTBase server" (HTTP 403).Jan 8 2024, 7:00 PM
matmarex mentioned this in T309438: Error loading data from server: Error contacting the Parsoid/RESTBase server (HTTP 500).
matmarex mentioned this in T299526: Visual Editor in MediaWiki 1.37.1: Error contacting the Parasoid/RESTBase Server: http-bad-status..
matmarex mentioned this in T290666: Visual Editor issue - Error contacting the Parsoid/RESTBase server: http-request-error.
matmarex mentioned this in T279203: Error contacting the Parsoid/RESTBase server: (curl error: 7) Couldn't connect to server.
matmarex mentioned this in T279039: Error contacting the Parsoid/RESTBase server (HTTP 400).
matmarex mentioned this in T274989: Error contacting the Parsoid/RESTBase server (HTTP 404).
matmarex mentioned this in T266037: MediaWiki 1.35 Error contacting the Parsoid/RESTBase server: (curl error: 28) Timeout was reached.
matmarex mentioned this in T270377: VisualEditor throws "Error contacting the Parsoid/RESTBase server (HTTP 500)" on fresh installation.
matmarex mentioned this in T266970: VisualEditor in 1.36-alpha with zeroconfig Parsoid throws 'Error contacting the Parsoid/RESTBase server (HTTP 404)'.
matmarex mentioned this in T298161: Unclear error related to Revision IDs returned by the server do not match (document: undefined, metadata / possibly related to "The requested relative path () did not match any known handler".Jan 8 2024, 7:10 PM
matmarex mentioned this in T265614: Visual Editor + IIS bug. Visual Editor not working when editing a page with "/" or "\".
matmarex mentioned this in T264680: Lockdown extension and VE: "Error loading from server: apierror-visualeditor-docserver-http: HTTP 500.".
matmarex mentioned this in T234049: "Error loading data from server: apierror-visualeditor-docserver-http-error: (curl error: 7) Couldn't connect to server.".
matmarex mentioned this in T306048: VisualEditor reports "Revision IDs returned by the server do not match (document: undefined, metadata: nnn)".Jan 8 2024, 7:16 PM
matmarex mentioned this in T306024: VE does not respect forwardCookies and fails on private wikis.
matmarex mentioned this in T272627: VisualEditor (MW 1.35): apierror-visualeditor-docserver-http-error.
matmarex mentioned this in T269174: HTTP 404 error for Visual editor when trying to edit a subpage (top-level page works).
matmarex mentioned this in T268277: MW 1.35.0 "Error contacting the Parsoid/RESTBase server: http-bad-status" on a private wiki.
matmarex mentioned this in T284075: Visual Editor doesn't work with sqlite and without curl.Jan 8 2024, 9:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL