Page MenuHomePhabricator

Parsoid/RESTbase seems to be unavailable in Beta
Closed, ResolvedPublic

Description

I am getting the following error while saving an Edit on VE, and while adding a comment using DiscussionTools.

Error contacting the Parsoid/RESTBase server (HTTP 404)

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
OpenNone
OpenNone
OpenNone
ResolvedGoalcscott
Resolvedcscott
Resolvedcscott
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedKrenair
ResolvedKrenair
ResolvedDzahn
ResolvedKrenair
Resolvedcscott

Event Timeline

Ryasmeen created this task.Mar 3 2020, 10:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2020, 10:09 PM

@Jdforrester-WMF thinks this is because of yesterday's changes to wmf-config that got merged and deployed where Parsoid is run from vendor/ code and not from the deploy repository.
I wonder if path issues are involved but this is something for @cscott and @Jdforrester-WMF to investigate. In case it helps that investigation, Marko and I ran into path issues when we were first deploying Parsoid-PHP to beta ( T231569#5508124 )

Yeah, this is probably that the Parsoid server(s?) in Beta Cluster don't have SERVERGROUP set to 'parsoid'.

Change 576466 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/puppet@production] [Beta Cluster] Tell parsoid10 that it's in the parsoid group

https://gerrit.wikimedia.org/r/576466

Change 576466 merged by Dzahn:
[operations/puppet@production] [Beta Cluster] Tell parsoid10 that it's in the parsoid group

https://gerrit.wikimedia.org/r/576466

ssastry closed this task as Resolved.Mar 4 2020, 12:12 AM
ssastry assigned this task to Jdforrester-WMF.

Thanks @Jdforrester-WMF ! I tested an edit on the beta cluster and it went through.

Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptMar 4 2020, 12:12 AM

Hmm, I'm still getting 404s for some attempts but not others.

Is it possible that it's cached somewhere?

Is it possible that it's cached somewhere?

jforrester@deployment-mediawiki-parsoid10:~$ sudo -u www-data php /srv/mediawiki/multiversion/MWScript.php shell.php enwiki
Psy Shell v0.9.12 (PHP 7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1 — cli) by Justin Hileman
>>> $_SERVER['SERVERGROUP'];
PHP Notice:  Undefined index: SERVERGROUP in /srv/mediawiki/php-mastereval()'d code on line 2

No, looks like it's not getting set. But then I don't know how any thing works, except some things certainly are (like the DiscussionTools editor).

Jdforrester-WMF reopened this task as Open.Mar 4 2020, 12:49 AM

Yes, that's this task; did you mean a different one?

cscott added a comment.Mar 4 2020, 2:49 PM

Yes, T246760 is almost certainly a dup.

cscott added a comment.EditedMar 4 2020, 3:58 PM

tl;dr: the root cause is that the parsoid-beta machine has an unusual configuration that didn't track the changes we made to prod; seems like James is doing the work to fix this in T246854: Replace deployment-mediawiki-parsoid10 with a "purer" deployment-parsoid11 box, which I've made a child task.

DLynch added a subscriber: DLynch.Mar 5 2020, 5:44 PM
ssastry moved this task from Needs Triage to Testing on the Parsoid board.Mar 5 2020, 11:05 PM
ssastry triaged this task as High priority.Mar 8 2020, 1:00 AM

Change 578529 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] [Beta Cluster] Point to deployment-parsoid11 for Parsoid services

https://gerrit.wikimedia.org/r/578529

Change 578529 merged by jenkins-bot:
[operations/mediawiki-config@master] [Beta Cluster] Point to deployment-parsoid11 for Parsoid services

https://gerrit.wikimedia.org/r/578529

jforrester@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:8001 http://en.wikipedia.wmflabs.org/w/rest.php/en.wikipedia.wmflabs.org/v3/page/html/Dog/413818
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /w/rest.php/en.wikipedia.wmflabs.org/v3/page/html/Dog/413818</pre>
</body>
</html>
jforrester@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:80 http://en.wikipedia.wmflabs.org/w/rest.php/en.wikipedia.wmflabs.org/v3/page/html/Dog/413818
<!DOCTYPE html>
<html lang=en>
<meta charset="utf-8">
<title>Unconfigured domain</title>
<link rel="shortcut icon" href="//foundation.wikimedia.org/favicon.ico">
<style>
* { margin: 0; padding: 0; }
body { background: #fff; margin: 7% auto 0; padding: 2em 1em 1em; font: 15px/1.6 sans-serif; color: #333; max-width: 640px; }
img { float: left; margin: 0 2em 2em 0; }
a img { border: 0; }
h1 { margin-top: 1em; font-size: 1.2em; }
p { margin: 0.7em 0 1em 0; }
a { color: #0645AD; text-decoration: none; }
a:hover { text-decoration: underline; }
</style>
<a href="https://www.wikimedia.org"><img src="https://www.wikimedia.org/static/images/wmf.png" srcset="https://www.wikimedia.org/static/images/wmf-2x.png 2x" alt=Wikimedia width=135 height=135></a>
<h1>Domain not configured</h1>
<p>This domain points to a <a href="https://www.wikimedia.org">Wikimedia Foundation</a> server, but is not configured on this server.</p>
</html>

This worked for me. I see that you missed the ".beta." in your urls

ssastry@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:80 http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African_linsang
<!doctype html><title>Redirect</title><a href="https://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African%20linsang/397377">https://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African%20linsang/397377</a>

ssastry@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:80 http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African_linsang/397377
<html prefix="dc: http://purl.org/dc/terms/ mw: http://mediawiki.org/rdf/" about="https://en.wikipedia.beta.wmflabs.org/wiki/Special:Redirect/revision/397377"><head prefix="mwr: https://en.wikipedia.beta.wmflabs.org/wiki/Special:Redirect/"><meta charset="utf-8"/><meta property="mw:pageId" content="89120"/><meta property="mw:pageNamespace" content="0"/><link rel="dc:replaces" resource="mwr:revision/397376"/><meta property="mw:revisionSHA1" content="f6c77ea40fc9c3131c1f2bb2a1463986ecc25786"/><meta property="dc:modified" content="2019-09-18T08:17:57.000Z"/><meta property="mw:html:version" content="2.1.0"/><link rel="dc:isVersionOf" href="https://en.wikipedia.beta.wmflabs.org/wiki/African_linsang"/><title>African linsang</title><base href="https://en.wikipedia.beta.wmflabs.org/wiki/"/><link rel="stylesheet" href="/w/load.php?modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Cskins.vector.styles%7Csite.styles%7Cext.cite.style%7Cext.cite.styles%7Cmediawiki.page.gallery.styles&amp;only=styles&amp;skin=vector"/><!--[if lt IE 9]><script src="/w/load.php?modules=html5shiv&amp;only=scripts&amp;skin=vector&amp;sync=1"></script><script>html5.addElements('figure-inline');</script><![endif]--><meta http-equiv="content-language" content="en"/><meta http-equiv="vary" content="Accept"/></head><body data-parsoid='{"dsr":[0,4874,0,0]}' lang="en" class="mw-content-ltr sitedir-ltr ltr mw-body-content parsoid-body mediawiki mw-parser-output" dir="ltr"><section data-mw-section-id="0" data-parsoid="{}"><div role="note" class="hatnote navigation-not-searchable" about="#mwt1" typeof="mw:Transclusion" data-parsoid='{"stx":"html","dsr":[0,78,null,null],"pi":[[{"k":"1"},{"k":"2"},{"k":"3"}]]}' data-mw='{"parts":[{"template":{"target":{"wt":"About","href":"./Template:About"},"params":{"1":{"wt":"the species of linsang"},"2":{"wt":"the genus of linsangs"},"3":{"wt":"African linsang (genus)"}},"i":0}}]}'>This article is about the species of linsang. For the genus of linsangs, see <a rel="mw:WikiLink" href="./African_linsang_(genus)" title="African linsang (genus)" class="new">African linsang (genus)</a>.</div>
...
....

But, when I open VE (which open successfully probably because the page is cached in RB), and click review changes after an edit, I get a "Error contacting the Parsoid/RESTBase server (HTTP 404)". So, RB / VE <--> Parsoid config needs some tweaking probably.

Change 579035 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[operations/puppet@production] Parsoid10 is dead, long live parsoid11

https://gerrit.wikimedia.org/r/579035

Change 579035 merged by Dzahn:
[operations/puppet@production] Parsoid10 is dead, long live parsoid11

https://gerrit.wikimedia.org/r/579035

Jdforrester-WMF closed this task as Resolved.Mar 11 2020, 11:38 PM
Jdforrester-WMF reassigned this task from Jdforrester-WMF to cscott.

A lot of the final push to get this fixed in Restbase and elsewhere was landed by Scott and Daniel. Thank you!

Change 579018 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[operations/mediawiki-config@master] Update linter whitelist w/ parsoid11's IP address

https://gerrit.wikimedia.org/r/579018

Change 579510 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Switch to deployment-parsoid11

https://gerrit.wikimedia.org/r/579510

Change 579510 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Switch to deployment-parsoid11

https://gerrit.wikimedia.org/r/579510

Change 579018 merged by jenkins-bot:
[operations/mediawiki-config@master] Update linter whitelist w/ parsoid11's IP address

https://gerrit.wikimedia.org/r/579018

Mentioned in SAL (#wikimedia-operations) [2020-03-18T18:27:26Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:579018|Update linter whitelist w/ parsoid11's IP address (T246833)]] (beta-only) (duration: 01m 04s)