The visual editor is not working on Wikitech since at least a few weeks.
|Resolved||Jdforrester-WMF||T241961 VisualEditor was removed from Wikitech because Parsoid/PHP isn't yet compatible with how Wikitech is set up|
|Resolved||cscott||T248343 Uncached VisualEditor w/ Parsoid/PHP (no JS, no RESTBase) for MW 1.35 LTS|
|Resolved||cscott||T259685 Zeroconf VisualEditor/Parsoid doesn't work on SQLite|
|Resolved||Legoktm||T259693 VisualEditor triggers HTTP 500 Parsoid error on Windows because of exception in GlobalIdGenerator|
- Mentioned In
- T237773: Move Wikitech onto the production MW cluster
T248343: Uncached VisualEditor w/ Parsoid/PHP (no JS, no RESTBase) for MW 1.35 LTS
- Mentioned Here
- rOMWC778f710bbbdb: Alternate configuration mechanism for Parsoid
T248343: Uncached VisualEditor w/ Parsoid/PHP (no JS, no RESTBase) for MW 1.35 LTS
T239660: Integrate Parsoid/PHP with core as a composer library
T237773: Move Wikitech onto the production MW cluster
I *think* this is because wikitech doesn't use RESTBase and talks to Parsoid directly... and I think *that's* because wikitech is on a different cluster and can't physically talk to RESTBase (and now Parsoid)? Can someone confirm that? I *think* this is more of an ops problem than a Parsoid bug...
I would love to know this as well. I'm still waiting for anyone to explain what needs to be done. If the answer is that for whatever reason we can only support VE from the main production wiki farm in the Parsoid/PHP era then we could try an prioritize T237773: Move Wikitech onto the production MW cluster. If we just need to setup a Parsoid/PHP deploy on the labweb* hosts which currently power wikitech.wikimedia.org we could weigh that work vs the move to the main cluster.
AIUI, options are:
A. Heroïc levels of engineering to lash up a production-like hacky Parsoid cluster and hope it doesn't break;
B. Migrate Wikitech to productions hacky systems and hope it doesn't break; or
C. Wait for proper MW integration of Parsoid, T239660, at which point VE's queries will be routed inside MW and this will "Just Work™".
We went with C as the least-effort option in December, but the other two options are possible (but just temporary extra work ahead of C).
Just for planning purposes, option C is "18 months out" (very rough estimate), although in theory we'd like to have a "all in one box" solution for VE/Parsoid by MW LTS 1.35 (May) -- but that LTS solution won't have proper caching (aka "RestBASE") it will generate Parsoid output directly when requested by VE. Let's call that option D, "let's hope no one is really going to edit WikiTech seriously w/ VE because latency and/or load will be high". So Aug 2021ish for option C, or May 2020ish for option D, or however long you think A and/or B would take.
Wikitech is not enwiki. Its not even what I would consider a big personal wiki by content page count or edit rate. Lack of pre-computed annotated DOMs for VE will be very unlikely to be a problem for our editing community.
OK. I don't think option D is officially on our roadmap, just a "nice to have"; @ssastry should probably weigh in re: whether we are going to commit to getting "VE+Parsoid/PHP w/o RESTBase" done before LTS. Requires us to export some temporary APIs in 1.35 that we'll probably immediately deprecate in 1.36 once "real" parser cache integration/RESTBase replacement occurs.
This is incorrect. The wikitech servers can talk to RESTBase
labweb1001:~$ telnet restbase.discovery.wmnet 7231 Trying 10.2.2.17... Connected to restbase.discovery.wmnet. Escape character is '^]'.
So if there is a reason not to use restbase for wikitech it doesn't have to do with reachability. I strongly suspect this is a remainder of when wikitech was much more different from the main cluster than it is today.
I think the problem could have to do with the fact wikitech doesn't emit events to kafka (I think?) and doesn't use the jobqueue. This means we don't have changeprop to propagate evictions / cache purges to restbase.
Based on what @Joe said, if MediaWiki in the production cluster can access the wikitech database, VE in wikitech can just ask Parsoid in the production cluster to serve its requests, this can work.
But, if not, the real solution is what @Jdforrester-WMF outlined as solution C and which @cscott refined as solution D. Scott and I have slightly different understanding of what option D is, but, TLDR is VE has always worked without RESTBase (modulo perf, and some dirty diff scenarios) and so, once Parsoid is accessible in MediaWiki (vs. special deployments to only the Parsoid cluster), we can work through our different understanding of what is needed to make this work and get it done.
@Joe is correct that the wikitech servers can communicate with RESTBase. The Parsoid/PHP deployment for the main wiki cluster reading the database for wikitech which is hosted on the m5 "misc" cluster is another matter. That may be possible, but as far as I know is currently untested.
We should be able to deploy Parsoid/PHP on the labsweb servers if that is needed. The 2 servers which serve wikitech are connected to scap and scap3 for deploying MediaWiki and other services running there. It sounds however like there may currently be a hard dependency on RESTBase that deploying Parsoid/PHP specifically for wikitech would not solve?
This is all sounding more and more to me like T237773: Move Wikitech onto the production MW cluster is the most expedient fix for the lost VE feature.
No, Parsoid/PHP or VE does not have a hard dependency on RESTBase. Otherwise, officewiki wouldn't have VE available. As long as VE can talk to a Parsoid/PHP instance, you can have VE. The problem right now is that VE on wikitech cannot talk to Parsoid/PHP in the production cluster (since Parsoid/PHP on that cluster cannot access the wikitech database). But, there are some more details here however.
Parsoid (as of today) has a special configuration that enables it only on the Parsoid cluster. But, soon, Parsoid will be available everywhere as a composer library. However, since we don't want to make Parsoid a DDOS vector on the production cluster, its API endpoints will still only be available on the Parsoid cluster which VE / Flow / RESTBase talk with.
So, to make VE available on wikitech's cluster in this new setup, we'll either have to make Parsoid's REST API public on wikitech, or write more code to have VE talk to Parsoid via the *library* API, not the HTTP API. So someone will still have to write that internal routing code (probably something with VirtualRestService classes) and make that configuration option available. Additionally, for performance reasons, eventually, this internal routing will have to go through parser cache (and this part is what Scott was referring to as 18 months out). But, for wikitech, this performance concern doesn't exist and so we can just get away with the internal routing code without needing parser cache work. This is Option D that Scott referred to. It would be nice to have that done before the next LTS release in May so that 3rd parties in general can benefit from VE out of the box (without having to making Parsoid's REST API public on their wikis). So, we are still targeting that but haven't yet done the phab task creation / project mgmt work to surface this work. I don't think this is substantial work - someone familiar with VRS code can get it done.
So, yes, either wikitech chooses to make the Parsoid REST API public (which is uncached parse endpoint that anyone can hit) OR we have to wait for the VRS code to be written.
Or, yes, you can deploy "Mediawiki wth Parsoid/PHP" to internal servers that only wikitech wiki can talk with and bypass all these considerations - this would be a transient solution while we wait on the above work to get done.
Or, yes, this is also a reasonable solution that solves this and any other issue that arises from Wikitech being a special snowflake that bypasses all the conversation above. :-)
@Jdforrester-WMF Let's manually install/load the Parsoid extension and explicitly set up the VRS routing ($wgVirtualRestConfig['modules']['parsoid'] = ; should be enough) rather than use the $wgVisualEditorParsoidAutoConfig stuff, though. That way it will be a little more robust to upgrades than "VE's copy of the Parsoid extension" is, which will probably break between 1.35 and 1.36.
^ slight counter-proposal, to keep wikitech a little bit more in sync with how Parsoid is run on the other production machines (Parsoid loaded from vendor/wikimedia/parsoid, rather from the copy included in VisualEditor 1.35).