I saw this too right after the DC switch. I imagine it has something to do with the restbase/parsoid setup. I would honestly consider this low to lowest priority. If it doesn't magically get better when eqiad is the primary DC again then we can dig into it.

Confirmed that VE editing works normally following switch back to eqiad as primary DC.

The issue is happening again with the switch over to codfw today:

{"error":{"code":"apierror-visualeditor-docserver-http","info":"HTTP 500","*":"See https://wikitech.wikimedia.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes."},"servedby":"labweb1002"}

wtp2013 /srv/log/parsoid/main.log:

{"name":"parsoid","hostname":"wtp2013","pid":12,"level":60,"levelPath":"fatal/request","msg":"Config Request failure for \"https://wikitech.wikimedia.org/w/api.php\": 404","time":"2018-09-11T21:04:56.556Z","v":0}

RB or Parsoid mis-configuration?

RB or Parsoid mis-configuration?

RESTBase is still not involved with wikitech, so you can rule RB out of the picture entirely.

In T163438#4575450, @Jdforrester-WMF wrote:

RB or Parsoid mis-configuration?

I guess parsoid, but it's probably subtle; AFAIR parsoid fetches info about the wikis from siteinfo, which AFAIR should still be fetched from eqiad.

Oh, yeah, that's probably why it breaks (the edge case we never test).

So the only config difference is that in codfw we call the mediawiki API via HTTPS, while in eqiad we call it via HTTP and I think there are some subtle differences to how we do it, that might explain why wikitech would fail via https - it's not hosted on the main cluster.

Why is Parsoid saying Config Request failure for "https://wikitech.wikimedia.org/w/api.php": 404 though? There has to be more to that request than it's logging.

Ok, I think I found the issue:

in eqiad, we define defaultAPIProxyURI and not mwApiServer, which means we follow this code path in ApiRequest:

https://github.com/wikimedia/parsoid/blob/597932be72082eb00999eb71565c2ac43f724205/lib/mw/ApiRequest.js#L294-L310

as I found out this morning, when we set up the proxy object referenced there during the initialization of the sitemaps, we explicitly set the proxy to null for private wikis, fishbowl wikis and - drumroll - 'labswiki' and 'labtestwiki':

https://github.com/wikimedia/parsoid/blob/994611e64cdd8409c74976657ee642ab4e7a17ec/lib/config/ParsoidConfig.js#L435-L446

This means that our requests to those wikis will not be proxied but will go through the open internet - this also explains why calls to wikitech from parsoid go through cache-text.

Of course this doesn't happen when we use mwApiServer and we just need to add that as a condition in ApiRequest in parsoid.

Change 459912 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/services/parsoid@master] Respect the proxy overriding even when going through the mwApiServer

https://gerrit.wikimedia.org/r/459912

In T163438#4576283, @Joe wrote:

Ok, I think I found the issue:

in eqiad, we define defaultAPIProxyURI and not mwApiServer, which means we follow this code path in ApiRequest:

https://github.com/wikimedia/parsoid/blob/597932be72082eb00999eb71565c2ac43f724205/lib/mw/ApiRequest.js#L294-L310

as I found out this morning, when we set up the proxy object referenced there during the initialization of the sitemaps, we explicitly set the proxy to null for private wikis, fishbowl wikis and - drumroll - 'labswiki' and 'labtestwiki':

https://github.com/wikimedia/parsoid/blob/994611e64cdd8409c74976657ee642ab4e7a17ec/lib/config/ParsoidConfig.js#L435-L446

This means that our requests to those wikis will not be proxied but will go through the open internet - this also explains why calls to wikitech from parsoid go through cache-text.

Of course this doesn't happen when we use mwApiServer and we just need to add that as a condition in ApiRequest in parsoid.

Good find!

Change 459912 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Respect the proxy overriding even when going through the mwApiServer

https://gerrit.wikimedia.org/r/459912

VE is functional on wikitech once more.

Mentioned in SAL (#wikimedia-operations) [2018-09-27T18:12:14Z] <arlolra> Updated Parsoid to rGPARaf3a920c97fd (T198511, T163438, T108776, T205334, T114413)

Change 463733 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[mediawiki/services/parsoid@master] Fix the logic (and comments) when skipping the funnel into mwApiServer

https://gerrit.wikimedia.org/r/463733

Change 463733 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Fix the logic (and comments) when skipping the funnel into mwApiServer

https://gerrit.wikimedia.org/r/463733

In T163438#4577979, @akosiaris wrote:

VE is functional on wikitech once more.

It appears to be malfunctioning again. Getting on each edit attempt:

Error loading data from server:
apierror-visualeditor-docserver-http: HTTP 500.

I was going to file a new task, but given this task was active just yesterday, re-opening for now.

Yup, https://gerrit.wikimedia.org/r/463733 is meant to fix it

In T163438#4633123, @akosiaris wrote:

Yup, https://gerrit.wikimedia.org/r/463733 is meant to fix it

Right, but it's seems to have had the opposite effect. Wikitech editing was working until this patch merged yesterday. (Or maybe something else caused it.)

That patch hasn't been deployed yet

Mentioned in SAL (#wikimedia-operations) [2018-10-02T18:07:58Z] <arlolra> Updated Parsoid to 65d6f82 (T163438, T205674, T205673)