Page MenuHomePhabricator

Parsoid/RESTbase seems to be unavailable in Beta
Closed, ResolvedPublic

Description

I am getting the following error while saving an Edit on VE, and while adding a comment using DiscussionTools.

Error contacting the Parsoid/RESTBase server (HTTP 404)

Screen Shot 2020-03-03 at 2.04.27 PM.png (575×1 px, 170 KB)

Related Objects

Event Timeline

@Jdforrester-WMF thinks this is because of yesterday's changes to wmf-config that got merged and deployed where Parsoid is run from vendor/ code and not from the deploy repository.
I wonder if path issues are involved but this is something for @cscott and @Jdforrester-WMF to investigate. In case it helps that investigation, Marko and I ran into path issues when we were first deploying Parsoid-PHP to beta ( T231569#5508124 )

Yeah, this is probably that the Parsoid server(s?) in Beta Cluster don't have SERVERGROUP set to 'parsoid'.

Change 576466 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/puppet@production] [Beta Cluster] Tell parsoid10 that it's in the parsoid group

https://gerrit.wikimedia.org/r/576466

Change 576466 merged by Dzahn:
[operations/puppet@production] [Beta Cluster] Tell parsoid10 that it's in the parsoid group

https://gerrit.wikimedia.org/r/576466

ssastry assigned this task to Jdforrester-WMF.

Thanks @Jdforrester-WMF ! I tested an edit on the beta cluster and it went through.

Hmm, I'm still getting 404s for some attempts but not others.

Is it possible that it's cached somewhere?

Is it possible that it's cached somewhere?

jforrester@deployment-mediawiki-parsoid10:~$ sudo -u www-data php /srv/mediawiki/multiversion/MWScript.php shell.php enwiki
Psy Shell v0.9.12 (PHP 7.2.26-1+0~20191218.33+debian9~1.gbpb5a340+wmf1 — cli) by Justin Hileman
>>> $_SERVER['SERVERGROUP'];
PHP Notice:  Undefined index: SERVERGROUP in /srv/mediawiki/php-mastereval()'d code on line 2

No, looks like it's not getting set. But then I don't know how any thing works, except some things certainly are (like the DiscussionTools editor).

tl;dr: the root cause is that the parsoid-beta machine has an unusual configuration that didn't track the changes we made to prod; seems like James is doing the work to fix this in T246854: Replace deployment-mediawiki-parsoid10 with a "purer" deployment-parsoid11 box, which I've made a child task.

ssastry triaged this task as High priority.Mar 8 2020, 1:00 AM

Change 578529 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[operations/mediawiki-config@master] [Beta Cluster] Point to deployment-parsoid11 for Parsoid services

https://gerrit.wikimedia.org/r/578529

Change 578529 merged by jenkins-bot:
[operations/mediawiki-config@master] [Beta Cluster] Point to deployment-parsoid11 for Parsoid services

https://gerrit.wikimedia.org/r/578529

jforrester@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:8001 http://en.wikipedia.wmflabs.org/w/rest.php/en.wikipedia.wmflabs.org/v3/page/html/Dog/413818
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /w/rest.php/en.wikipedia.wmflabs.org/v3/page/html/Dog/413818</pre>
</body>
</html>
jforrester@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:80 http://en.wikipedia.wmflabs.org/w/rest.php/en.wikipedia.wmflabs.org/v3/page/html/Dog/413818
<!DOCTYPE html>
<html lang=en>
<meta charset="utf-8">
<title>Unconfigured domain</title>
<link rel="shortcut icon" href="//foundation.wikimedia.org/favicon.ico">
<style>
* { margin: 0; padding: 0; }
body { background: #fff; margin: 7% auto 0; padding: 2em 1em 1em; font: 15px/1.6 sans-serif; color: #333; max-width: 640px; }
img { float: left; margin: 0 2em 2em 0; }
a img { border: 0; }
h1 { margin-top: 1em; font-size: 1.2em; }
p { margin: 0.7em 0 1em 0; }
a { color: #0645AD; text-decoration: none; }
a:hover { text-decoration: underline; }
</style>
<a href="https://www.wikimedia.org"><img src="https://www.wikimedia.org/static/images/wmf.png" srcset="https://www.wikimedia.org/static/images/wmf-2x.png 2x" alt=Wikimedia width=135 height=135></a>
<h1>Domain not configured</h1>
<p>This domain points to a <a href="https://www.wikimedia.org">Wikimedia Foundation</a> server, but is not configured on this server.</p>
</html>

This worked for me. I see that you missed the ".beta." in your urls

ssastry@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:80 http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African_linsang
<!doctype html><title>Redirect</title><a href="https://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African%20linsang/397377">https://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African%20linsang/397377</a>

ssastry@deployment-parsoid11:~$ curl -x deployment-parsoid11.deployment-prep.eqiad.wmflabs:80 http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/African_linsang/397377
<html prefix="dc: http://purl.org/dc/terms/ mw: http://mediawiki.org/rdf/" about="https://en.wikipedia.beta.wmflabs.org/wiki/Special:Redirect/revision/397377"><head prefix="mwr: https://en.wikipedia.beta.wmflabs.org/wiki/Special:Redirect/"><meta charset="utf-8"/><meta property="mw:pageId" content="89120"/><meta property="mw:pageNamespace" content="0"/><link rel="dc:replaces" resource="mwr:revision/397376"/><meta property="mw:revisionSHA1" content="f6c77ea40fc9c3131c1f2bb2a1463986ecc25786"/><meta property="dc:modified" content="2019-09-18T08:17:57.000Z"/><meta property="mw:html:version" content="2.1.0"/><link rel="dc:isVersionOf" href="https://en.wikipedia.beta.wmflabs.org/wiki/African_linsang"/><title>African linsang</title><base href="https://en.wikipedia.beta.wmflabs.org/wiki/"/><link rel="stylesheet" href="/w/load.php?modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Cskins.vector.styles%7Csite.styles%7Cext.cite.style%7Cext.cite.styles%7Cmediawiki.page.gallery.styles&amp;only=styles&amp;skin=vector"/><!--[if lt IE 9]><script src="/w/load.php?modules=html5shiv&amp;only=scripts&amp;skin=vector&amp;sync=1"></script><script>html5.addElements('figure-inline');</script><![endif]--><meta http-equiv="content-language" content="en"/><meta http-equiv="vary" content="Accept"/></head><body data-parsoid='{"dsr":[0,4874,0,0]}' lang="en" class="mw-content-ltr sitedir-ltr ltr mw-body-content parsoid-body mediawiki mw-parser-output" dir="ltr"><section data-mw-section-id="0" data-parsoid="{}"><div role="note" class="hatnote navigation-not-searchable" about="#mwt1" typeof="mw:Transclusion" data-parsoid='{"stx":"html","dsr":[0,78,null,null],"pi":[[{"k":"1"},{"k":"2"},{"k":"3"}]]}' data-mw='{"parts":[{"template":{"target":{"wt":"About","href":"./Template:About"},"params":{"1":{"wt":"the species of linsang"},"2":{"wt":"the genus of linsangs"},"3":{"wt":"African linsang (genus)"}},"i":0}}]}'>This article is about the species of linsang. For the genus of linsangs, see <a rel="mw:WikiLink" href="./African_linsang_(genus)" title="African linsang (genus)" class="new">African linsang (genus)</a>.</div>
...
....

But, when I open VE (which open successfully probably because the page is cached in RB), and click review changes after an edit, I get a "Error contacting the Parsoid/RESTBase server (HTTP 404)". So, RB / VE <--> Parsoid config needs some tweaking probably.

Change 579035 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[operations/puppet@production] Parsoid10 is dead, long live parsoid11

https://gerrit.wikimedia.org/r/579035

Change 579035 merged by Dzahn:
[operations/puppet@production] Parsoid10 is dead, long live parsoid11

https://gerrit.wikimedia.org/r/579035

Jdforrester-WMF reassigned this task from Jdforrester-WMF to cscott.

A lot of the final push to get this fixed in Restbase and elsewhere was landed by Scott and Daniel. Thank you!

Change 579018 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[operations/mediawiki-config@master] Update linter whitelist w/ parsoid11's IP address

https://gerrit.wikimedia.org/r/579018

Change 579510 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/restbase/deploy@master] Switch to deployment-parsoid11

https://gerrit.wikimedia.org/r/579510

Change 579510 merged by Ppchelko:
[mediawiki/services/restbase/deploy@master] Switch to deployment-parsoid11

https://gerrit.wikimedia.org/r/579510

Change 579018 merged by jenkins-bot:
[operations/mediawiki-config@master] Update linter whitelist w/ parsoid11's IP address

https://gerrit.wikimedia.org/r/579018

Mentioned in SAL (#wikimedia-operations) [2020-03-18T18:27:26Z] <lucaswerkmeister-wmde@deploy1001> Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:579018|Update linter whitelist w/ parsoid11's IP address (T246833)]] (beta-only) (duration: 01m 04s)