This is also a problem on deployment-cache-text02 and likely others as well.

In T103660#1397903, @thcipriani wrote:

This is also a problem on deployment-cache-text02 and likely others as well.

Looks like it's just parsoidcache02 now.

The other caches have been migrated to Jessie which comes with systemd (T98758). deployment-parsoidcache02 still uses Trusty hence the failure.

Turns out we had an instance created using Jessie:

deployment-cache-parsoid04
debian-8.1-jessie	
10.68.19.197

parsoid-beta.wmflabs.org points to the public IP 208.80.155.156 which is the labs shared proxy. That is entirely wrong: beta-cluster should use a dedicated cache.

Turns out we had an instance created using Jessie:

deployment-cache-parsoid04
debian-8.1-jessie	
10.68.19.197

parsoid-beta.wmflabs.org points to the public IP 208.80.155.156 which is the labs shared proxy. That is entirely wrong: beta-cluster should use a dedicated cache.

We should delete:

To be replaced with:

Change 249366 had a related patch set uploaded (by Mobrovac):
Labs: Parsoid Cache: Use new IP address for deployment-parsoidcache02

https://gerrit.wikimedia.org/r/249366

Change 249367 had a related patch set uploaded (by Mobrovac):
BetaCluster: Use deployment-cache-parsoid05

https://gerrit.wikimedia.org/r/249367

Change 249366 merged by jenkins-bot:
beta: use new IP for Parsoid Cache

https://gerrit.wikimedia.org/r/249366

Change 249367 merged by jenkins-bot:
BetaCluster: Use deployment-cache-parsoid05

https://gerrit.wikimedia.org/r/249367

Doing a VE edit on beta I get:

Error loading data from server: HTTP 504.

In the browser console there is:

http://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/User%3AHashar
Failed to load resource: the server responded with a status of 504 (Gateway Timeout)

And:

$ curl http://en.wikipedia.beta.wmflabs.org/api/rest_v1/page/html/User%3AHashar|python -m json.tool
{
    "detail": "Error: connect ECONNREFUSED",
    "method": "get",
    "type": "https://restbase.org/errors/internal_http_error",
    "uri": "http://deployment-parsoid05.deployment-prep.eqiad.wmflabs:8000/v2/en.wikipedia.beta.wmflabs.org/pagebundle/User%3AHashar/85930"
}

On the Parsoid instance:

deployment-parsoid05:~$ curl http://127.0.0.1:8000/v2/en.wikipedia.beta.wmflabs.org/pagebundle/User%3AHashar/85930
curl: (7) Failed to connect to 127.0.0.1 port 8000: Connection refused

And indeed parsoid is not running there...

The merge of https://gerrit.wikimedia.org/r/#/c/249367/ did trigger the Jenkins job beta-parsoid-update-eqiad with:

+ sudo /etc/init.d/parsoid restart
00:02:46.782 parsoid start/running, process 27941

Parsoid fails with:

module.js:340
    throw err;
          ^
Error: Cannot find module '/srv/deployment/parsoid/parsoid/api/server.js'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:902:3

Lot of files have been renamed in the parsoid source repo but the setting files to start the service have not been updated yet. So it is pending T116901: Parsoid refuses to start on beta cluster

So Parsoid is running fine:

root@deployment-parsoid05:~# curl http://localhost:8000/_version|python -m json.tool
{
	    "name": "parsoid",
	    "sha": "47b8013ae7b15428122579bfff4943191c1866db",
	    "version": "0.4.1-git"
}

Gotta verify whether the Varnish Parsoid cache is properly working. No idea how to assert that though :-(

Could use help from Parsoid people to verify whether the new Parsoid Varnish cache is properly working.

The instance is using role::cache::parsoid, it is now a Jessie distribution:

If it is found to work fine, I guess we can finally resolve this task.

So there is still a web proxy entry:

Moved it to the new instance: