Page MenuHomePhabricator

Beta Cluster: Parsoid config request failures from the MediaWiki API
Closed, ResolvedPublic

Description

Parsoid parses on the beta cluster are currently failing right at the beginning with a config request failure with a Error: Protocol "http:" not supported. Expected "https:"error.

Is this some transient issue or did something change with the beta labs config?

Event Timeline

Pchelolo added a subscriber: Pchelolo.

The problem is that MW API is configured as 'http://', but for some reason request uses the https Agent - thus the failure.

I don't think anything has changed in the beta cluster setup to cause this, but probably some dependencies have changed. Did you recently upgrade request library? We had a similar issue in preq at some point - request changed the api and now just setting the agent to the req options is not a good way, the better way is to set agentClass and agentOptions - then request internally pools the agents. If you just set the agent instance in my experience it might be ignored with some combinations of options. See https://github.com/wikimedia/preq/blob/master/index.js#L104-L105

mobrovac added a subscriber: mobrovac.

The error, as seen by RESTBase:

error: Config Request failure for \"https://en.wikipedia.beta.wmflabs.org/w/api.php\": Error: Protocol \"http:\" not supported. Expected \"https:\" path: /en.wikipedia.beta.wmflabs.org/v3/page/pagebundle/Main_Page/377057
Error: Config Request failure for \"https://en.wikipedia.beta.wmflabs.org/w/api.php\": Error: Protocol \"http:\" not supported. Expected \"https:\"
    at ConfigRequest.ApiRequest._requestCB (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/src/lib/mw/ApiRequest.js:390:15)
    at self.callback (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:185:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.start (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:749:10)
    at Request.end (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:1506:10)
    at end (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:560:14)
    at Immediate.<anonymous> (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:574:7)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
09:37 Pchelolo: arlolra: I think I found why parsoid is failing in beta
09:39 arlolra: Pchelolo: /me perks up ... I haven't actually looked yet
09:40 arlolra: I just assumed it was a configuration change
09:41 Pchelolo: when using ApiRequest, https://en.wikipedia.beta.wmflabs.org/w/api.php is provided as a uri which is conf.wiki.apiURI. http.Agent is selected based on it and https agent is assigned. then on line 291 the uri is assigned to mwApiServer===conf.parsoid.mwApiServer which is http://deployment-mediawiki-07.deployment-prep.eqiad.wmflabs/w/api.php
09:41 Pchelolo: so the protocol starts to mismatch the agent
09:41 Pchelolo: we just need to move the agent selection code way to the bottom of ApiRequest.prototype.request method and we should be fine
09:41 Pchelolo: verifying
09:43 Pchelolo: vuala, parsoid works in beta
09:44 Pchelolo: I'm not sure my approach is entirely correct, but parsoid is life-hacked on deployment-parsoid09 now and it works

I will leave it to the professionals to verify this is the right approach, but it seems correct, the mwApi and mwApiServer protocols can mismatch, so we need to assign a correct agent.

Change 464364 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Set request agent after updating uri with mwApiServer

https://gerrit.wikimedia.org/r/464364

While @Arlolra patch surely does a sensible thing, I am not sure we have an https interface to mediawiki in beta. I didn't think of this when I did the transition in production, probably. This is easy to solve though, and can go in parallel with what @Arlolra is doing (which is indeed correct anyways).

Change 464364 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Set request agent after updating uri with mwApiServer

https://gerrit.wikimedia.org/r/464364

Sorry, I misinterpreted the issue. We are connecting via http even though the configuration says https, so Arlo's patch is all that's needed.

Arlolra claimed this task.

The config still doesn't seem to be working for the edge case of en.wp.org:

error: Config Request failure for \"https://en.wikipedia.org/w/api.php\": 404 path: /en.wikipedia.org/v3/page/pagebundle/Main_Page/847600508
Error: Config Request failure for \"https://en.wikipedia.org/w/api.php\": 404
    at ConfigRequest.ApiRequest._requestCB (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/src/lib/mw/ApiRequest.js:410:11)
    at Request.self.callback (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/node_modules/request/request.js:185:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/node_modules/request/request.js:1157:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/node_modules/request/request.js:1079:12)
    at IncomingMessage.g (events.js:292:16)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

Change 464553 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid/deploy@master] Fix enwikiprod on the beta cluster

https://gerrit.wikimedia.org/r/464553

@mobrovac The above patch is deployed. Please confirm it meets your expectations

Yup, that did the trick! Thank you, @Arlolra !

Change 464553 merged by jenkins-bot:
[mediawiki/services/parsoid/deploy@master] Fix enwikiprod on the beta cluster

https://gerrit.wikimedia.org/r/464553