Yeah, this is known and "expected"... Pending more migrations

I can't find the relevant task offhand though

I *think* this is because wikitech doesn't use RESTBase and talks to Parsoid directly... and I think *that's* because wikitech is on a different cluster and can't physically talk to RESTBase (and now Parsoid)? Can someone confirm that? I *think* this is more of an ops problem than a Parsoid bug...

Any ETA on when it will be fixed?

In T241961#5890295, @ayounsi wrote:

Any ETA on when it will be fixed?

I would love to know this as well. I'm still waiting for anyone to explain what needs to be done. If the answer is that for whatever reason we can only support VE from the main production wiki farm in the Parsoid/PHP era then we could try an prioritize T237773: Move Wikitech onto the production MW cluster. If we just need to setup a Parsoid/PHP deploy on the labweb* hosts which currently power wikitech.wikimedia.org we could weigh that work vs the move to the main cluster.

AIUI, options are:

A. Heroïc levels of engineering to lash up a production-like hacky Parsoid cluster and hope it doesn't break;
B. Migrate Wikitech to productions hacky systems and hope it doesn't break; or
C. Wait for proper MW integration of Parsoid, T239660, at which point VE's queries will be routed inside MW and this will "Just Work™".

We went with C as the least-effort option in December, but the other two options are possible (but just temporary extra work ahead of C).

Just for planning purposes, option C is "18 months out" (very rough estimate), although in theory we'd like to have a "all in one box" solution for VE/Parsoid by MW LTS 1.35 (May) -- but that LTS solution won't have proper caching (aka "RestBASE") it will generate Parsoid output directly when requested by VE. Let's call that option D, "let's hope no one is really going to edit WikiTech seriously w/ VE because latency and/or load will be high". So Aug 2021ish for option C, or May 2020ish for option D, or however long you think A and/or B would take.

In T241961#5894194, @cscott wrote:

Just for planning purposes, option C is "18 months out" (very rough estimate), although in theory we'd like to have a "all in one box" solution for VE/Parsoid by MW LTS 1.35 (May) -- but that LTS solution won't have proper caching (aka "RestBASE") it will generate Parsoid output directly when requested by VE. Let's call that option D, "let's hope no one is really going to edit WikiTech seriously w/ VE because latency and/or load will be high". So Aug 2021ish for option C, or May 2020ish for option D, or however long you think A and/or B would take.

Wikitech is not enwiki. Its not even what I would consider a big personal wiki by content page count or edit rate. Lack of pre-computed annotated DOMs for VE will be very unlikely to be a problem for our editing community.

Note that wikitech never had RESTbase and it was "fine", so option D sounds good enough.

OK. I don't think option D is officially on our roadmap, just a "nice to have"; @ssastry should probably weigh in re: whether we are going to commit to getting "VE+Parsoid/PHP w/o RESTBase" done before LTS. Requires us to export some temporary APIs in 1.35 that we'll probably immediately deprecate in 1.36 once "real" parser cache integration/RESTBase replacement occurs.

In T241961#5778315, @cscott wrote:

I *think* this is because wikitech doesn't use RESTBase and talks to Parsoid directly... and I think *that's* because wikitech is on a different cluster and can't physically talk to RESTBase (and now Parsoid)? Can someone confirm that? I *think* this is more of an ops problem than a Parsoid bug...

This is incorrect. The wikitech servers can talk to RESTBase

labweb1001:~$ telnet restbase.discovery.wmnet 7231
Trying 10.2.2.17...
Connected to restbase.discovery.wmnet.
Escape character is '^]'.

So if there is a reason not to use restbase for wikitech it doesn't have to do with reachability. I strongly suspect this is a remainder of when wikitech was much more different from the main cluster than it is today.

I think the problem could have to do with the fact wikitech doesn't emit events to kafka (I think?) and doesn't use the jobqueue. This means we don't have changeprop to propagate evictions / cache purges to restbase.

Based on what @Joe said, if MediaWiki in the production cluster can access the wikitech database, VE in wikitech can just ask Parsoid in the production cluster to serve its requests, this can work.

But, if not, the real solution is what @Jdforrester-WMF outlined as solution C and which @cscott refined as solution D. Scott and I have slightly different understanding of what option D is, but, TLDR is VE has always worked without RESTBase (modulo perf, and some dirty diff scenarios) and so, once Parsoid is accessible in MediaWiki (vs. special deployments to only the Parsoid cluster), we can work through our different understanding of what is needed to make this work and get it done.

In T241961#5894457, @ssastry wrote:

Based on what @Joe said, if MediaWiki in the production cluster can access the wikitech database, VE in wikitech can just ask Parsoid in the production cluster to serve its requests, this can work.

@Joe is correct that the wikitech servers can communicate with RESTBase. The Parsoid/PHP deployment for the main wiki cluster reading the database for wikitech which is hosted on the m5 "misc" cluster is another matter. That may be possible, but as far as I know is currently untested.

We should be able to deploy Parsoid/PHP on the labsweb servers if that is needed. The 2 servers which serve wikitech are connected to scap and scap3 for deploying MediaWiki and other services running there. It sounds however like there may currently be a hard dependency on RESTBase that deploying Parsoid/PHP specifically for wikitech would not solve?

This is all sounding more and more to me like T237773: Move Wikitech onto the production MW cluster is the most expedient fix for the lost VE feature.

In T241961#5894853, @bd808 wrote:

We should be able to deploy Parsoid/PHP on the labsweb servers if that is needed. The 2 servers which serve wikitech are connected to scap and scap3 for deploying MediaWiki and other services running there. It sounds however like there may currently be a hard dependency on RESTBase that deploying Parsoid/PHP specifically for wikitech would not solve?

No, Parsoid/PHP or VE does not have a hard dependency on RESTBase. Otherwise, officewiki wouldn't have VE available. As long as VE can talk to a Parsoid/PHP instance, you can have VE. The problem right now is that VE on wikitech cannot talk to Parsoid/PHP in the production cluster (since Parsoid/PHP on that cluster cannot access the wikitech database). But, there are some more details here however.

Parsoid (as of today) has a special configuration that enables it only on the Parsoid cluster. But, soon, Parsoid will be available everywhere as a composer library. However, since we don't want to make Parsoid a DDOS vector on the production cluster, its API endpoints will still only be available on the Parsoid cluster which VE / Flow / RESTBase talk with.

So, to make VE available on wikitech's cluster in this new setup, we'll either have to make Parsoid's REST API public on wikitech, or write more code to have VE talk to Parsoid via the *library* API, not the HTTP API. So someone will still have to write that internal routing code (probably something with VirtualRestService classes) and make that configuration option available. Additionally, for performance reasons, eventually, this internal routing will have to go through parser cache (and this part is what Scott was referring to as 18 months out). But, for wikitech, this performance concern doesn't exist and so we can just get away with the internal routing code without needing parser cache work. This is Option D that Scott referred to. It would be nice to have that done before the next LTS release in May so that 3rd parties in general can benefit from VE out of the box (without having to making Parsoid's REST API public on their wikis). So, we are still targeting that but haven't yet done the phab task creation / project mgmt work to surface this work. I don't think this is substantial work - someone familiar with VRS code can get it done.

So, yes, either wikitech chooses to make the Parsoid REST API public (which is uncached parse endpoint that anyone can hit) OR we have to wait for the VRS code to be written.

Or, yes, you can deploy "Mediawiki wth Parsoid/PHP" to internal servers that only wikitech wiki can talk with and bypass all these considerations - this would be a transient solution while we wait on the above work to get done.

In T241961#5894853, @bd808 wrote:

This is all sounding more and more to me like T237773: Move Wikitech onto the production MW cluster is the most expedient fix for the lost VE feature.

Or, yes, this is also a reasonable solution that solves this and any other issue that arises from Wikitech being a special snowflake that bypasses all the conversation above. :-)

Added T248343 and T237773 as parent tasks based on the two main options for fixing wikitech; note that only one or the other has to be done to unblock this task.

In T241961#5993030, @cscott wrote:

Added T248343 and T237773 as parent tasks based on the two main options for fixing wikitech; note that only one or the other has to be done to unblock this task.

You want them as subtasks of this task, not parent tasks.

Fixed, thanks!

This is probably something that will Just Work™ with wmf.41 and the new config option. Will investigate after the train deployment tomorrow.

@Jdforrester-WMF Let's manually install/load the Parsoid extension and explicitly set up the VRS routing ($wgVirtualRestConfig['modules']['parsoid'] = []; should be enough) rather than use the $wgVisualEditorParsoidAutoConfig stuff, though. That way it will be a little more robust to upgrades than "VE's copy of the Parsoid extension" is, which will probably break between 1.35 and 1.36.

Change 612875 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] Add VisualEditor support back to wikitech

https://gerrit.wikimedia.org/r/612875

Change 612875 merged by jenkins-bot:
[operations/mediawiki-config@master] Add VisualEditor support back to wikitech

https://gerrit.wikimedia.org/r/612875

Mentioned in SAL (#wikimedia-operations) [2020-07-15T14:26:30Z] <jforrester@deploy1001> Synchronized wmf-config/CommonSettings.php: Add exceptional wikitech VE/Parsoid config T241961 (duration: 01m 05s)

Mentioned in SAL (#wikimedia-operations) [2020-07-15T14:28:34Z] <jforrester@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Add exceptional wikitech VE/Parsoid config T241961 (duration: 01m 04s)

Change 612879 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[operations/mediawiki-config@master] Alternate configuration mechanism for Parsoid

https://gerrit.wikimedia.org/r/612879

^ slight counter-proposal, to keep wikitech a little bit more in sync with how Parsoid is run on the other production machines (Parsoid loaded from vendor/wikimedia/parsoid, rather from the copy included in VisualEditor 1.35).

Change 612879 merged by jenkins-bot:
[operations/mediawiki-config@master] Alternate configuration mechanism for Parsoid

https://gerrit.wikimedia.org/r/612879

Mentioned in SAL (#wikimedia-operations) [2020-08-24T23:06:29Z] <urbanecm@deploy1001> Synchronized wmf-config/CommonSettings.php: 778f710bbbdb24730f7ce4c75d5ff1ca7a5ce3b3: Alternate configuration mechanism for Parsoid (T241961) (duration: 00m 58s)

Change 949592 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[operations/mediawiki-config@master] Remove unusual VisualEditor config for Wikitech

https://gerrit.wikimedia.org/r/949592

Change 949592 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove unusual VisualEditor config for Wikitech

https://gerrit.wikimedia.org/r/949592

Mentioned in SAL (#wikimedia-operations) [2023-08-16T20:04:36Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:949592|Remove unusual VisualEditor config for Wikitech (T241961)]], [[gerrit:949545|Disable upcoming wgMFShowEditNotices in production (T312587)]], [[gerrit:943558|Explicitly set DiscussionToolsAutoTopicSubEditor to discussiontoolsapi]]

Mentioned in SAL (#wikimedia-operations) [2023-08-16T20:06:16Z] <urbanecm@deploy1002> esanders and urbanecm and matmarex: Backport for [[gerrit:949592|Remove unusual VisualEditor config for Wikitech (T241961)]], [[gerrit:949545|Disable upcoming wgMFShowEditNotices in production (T312587)]], [[gerrit:943558|Explicitly set DiscussionToolsAutoTopicSubEditor to discussiontoolsapi]] synced to the testservers mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwde

Mentioned in SAL (#wikimedia-operations) [2023-08-16T20:14:04Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:949592|Remove unusual VisualEditor config for Wikitech (T241961)]], [[gerrit:949545|Disable upcoming wgMFShowEditNotices in production (T312587)]], [[gerrit:943558|Explicitly set DiscussionToolsAutoTopicSubEditor to discussiontoolsapi]] (duration: 09m 27s)