Page MenuHomePhabricator

Beta Cluster: Parsoid config request failures from the MediaWiki API
Closed, ResolvedPublic

Description

Parsoid parses on the beta cluster are currently failing right at the beginning with a config request failure with a Error: Protocol "http:" not supported. Expected "https:"error.

Is this some transient issue or did something change with the beta labs config?

Event Timeline

ssastry created this task.Oct 2 2018, 6:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 2 2018, 6:04 PM
Pchelolo added a subscriber: Pchelolo.

The problem is that MW API is configured as 'http://', but for some reason request uses the https Agent - thus the failure.

I don't think anything has changed in the beta cluster setup to cause this, but probably some dependencies have changed. Did you recently upgrade request library? We had a similar issue in preq at some point - request changed the api and now just setting the agent to the req options is not a good way, the better way is to set agentClass and agentOptions - then request internally pools the agents. If you just set the agent instance in my experience it might be ignored with some combinations of options. See https://github.com/wikimedia/preq/blob/master/index.js#L104-L105

mobrovac added a subscriber: mobrovac.

The error, as seen by RESTBase:

error: Config Request failure for \"https://en.wikipedia.beta.wmflabs.org/w/api.php\": Error: Protocol \"http:\" not supported. Expected \"https:\" path: /en.wikipedia.beta.wmflabs.org/v3/page/pagebundle/Main_Page/377057
Error: Config Request failure for \"https://en.wikipedia.beta.wmflabs.org/w/api.php\": Error: Protocol \"http:\" not supported. Expected \"https:\"
    at ConfigRequest.ApiRequest._requestCB (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/src/lib/mw/ApiRequest.js:390:15)
    at self.callback (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:185:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at Request.start (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:749:10)
    at Request.end (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:1506:10)
    at end (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:560:14)
    at Immediate.<anonymous> (/srv/deployment/parsoid/deploy-cache/revs/8ff45dbe8244a853d497c9d4d3a53acd158dbb3e/node_modules/request/request.js:574:7)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
Krenair added a subscriber: Krenair.Oct 3 2018, 2:29 PM
Pchelolo added a comment.EditedOct 3 2018, 4:49 PM
09:37 Pchelolo: arlolra: I think I found why parsoid is failing in beta
09:39 arlolra: Pchelolo: /me perks up ... I haven't actually looked yet
09:40 arlolra: I just assumed it was a configuration change
09:41 Pchelolo: when using ApiRequest, https://en.wikipedia.beta.wmflabs.org/w/api.php is provided as a uri which is conf.wiki.apiURI. http.Agent is selected based on it and https agent is assigned. then on line 291 the uri is assigned to mwApiServer===conf.parsoid.mwApiServer which is http://deployment-mediawiki-07.deployment-prep.eqiad.wmflabs/w/api.php
09:41 Pchelolo: so the protocol starts to mismatch the agent
09:41 Pchelolo: we just need to move the agent selection code way to the bottom of ApiRequest.prototype.request method and we should be fine
09:41 Pchelolo: verifying
09:43 Pchelolo: vuala, parsoid works in beta
09:44 Pchelolo: I'm not sure my approach is entirely correct, but parsoid is life-hacked on deployment-parsoid09 now and it works

I will leave it to the professionals to verify this is the right approach, but it seems correct, the mwApi and mwApiServer protocols can mismatch, so we need to assign a correct agent.

Change 464364 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid@master] Set request agent after updating uri with mwApiServer

https://gerrit.wikimedia.org/r/464364

Joe added a comment.Oct 3 2018, 5:58 PM

While @Arlolra patch surely does a sensible thing, I am not sure we have an https interface to mediawiki in beta. I didn't think of this when I did the transition in production, probably. This is easy to solve though, and can go in parallel with what @Arlolra is doing (which is indeed correct anyways).

Change 464364 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Set request agent after updating uri with mwApiServer

https://gerrit.wikimedia.org/r/464364

Joe added a comment.Oct 3 2018, 6:10 PM

Sorry, I misinterpreted the issue. We are connecting via http even though the configuration says https, so Arlo's patch is all that's needed.

Arlolra closed this task as Resolved.Oct 3 2018, 6:12 PM
Arlolra claimed this task.
mobrovac reopened this task as Open.Oct 4 2018, 11:56 AM

The config still doesn't seem to be working for the edge case of en.wp.org:

error: Config Request failure for \"https://en.wikipedia.org/w/api.php\": 404 path: /en.wikipedia.org/v3/page/pagebundle/Main_Page/847600508
Error: Config Request failure for \"https://en.wikipedia.org/w/api.php\": 404
    at ConfigRequest.ApiRequest._requestCB (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/src/lib/mw/ApiRequest.js:410:11)
    at Request.self.callback (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/node_modules/request/request.js:185:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/node_modules/request/request.js:1157:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous> (/srv/deployment/parsoid/deploy-cache/revs/894d12f4a01e5445b0716d16f70e81d0be4f78fd/node_modules/request/request.js:1079:12)
    at IncomingMessage.g (events.js:292:16)
    at emitNone (events.js:91:20)
    at IncomingMessage.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)

Change 464553 had a related patch set uploaded (by Arlolra; owner: Arlolra):
[mediawiki/services/parsoid/deploy@master] Fix enwikiprod on the beta cluster

https://gerrit.wikimedia.org/r/464553

@mobrovac The above patch is deployed. Please confirm it meets your expectations

mobrovac closed this task as Resolved.Oct 4 2018, 1:14 PM

Yup, that did the trick! Thank you, @Arlolra !

Change 464553 merged by jenkins-bot:
[mediawiki/services/parsoid/deploy@master] Fix enwikiprod on the beta cluster

https://gerrit.wikimedia.org/r/464553