Page MenuHomePhabricator

Deploy Parsoid-PHP (integrated with Mediawiki) to the beta cluster
Closed, ResolvedPublic

Description

We should deploy Parsoid/PHP to the beta cluster so that RESTBase can test integration with it.

  • New vm instance that is a MediaWiki appserver and Parsoid server
  • HTTP proxy set up on the VM (@Pchelolo on this) - http://parsoid-php-beta.wmflabs.org
  • New vm instance is a scap target for both MediaWiki and Parsoid ( patches in gerrit )
  • MediaWiki config for labs for MediaWiki to load Parsoid/PHP as an extension ( patch in gerrit )
  • Move Parsoid/JS' port in beta to a different port
  • Update references to Parsoid/JS in beta (proxy, RB)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
ssastry renamed this task from Deploy Parsoid-PHP with Mediawiki to the beta cluster to Deploy Parsoid-PHP (integrated with Mediawiki) to the beta cluster.Aug 29 2019, 3:34 PM
ssastry triaged this task as High priority.
ssastry updated the task description. (Show Details)
ssastry moved this task from Backlog to Testing / QA on the Parsoid-PHP board.

To match what will happen in production (parsoid cluster will become appservers), it makes sense to make the beta cluster parsoid server an appserver too. This will require puppet changes and something that @mobrovac says he will investigate.

Separately, I'll create a wmf-config repo patch to update the settings to load the Parsoid extension on the appservers (if parsoid exists on disk).

@Pchelolo note that you will no longer have the ability to ask Parsoid/PHP to talk with the production enwiki wiki. Strictly speaking, we can enable that (since we have the option for development), but it will be ultra slow since all api requests will be serialized and practically useless.

Change 534215 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/mediawiki-config@master] Enable loading Parsoid/PHP as an extension on the beta cluster

https://gerrit.wikimedia.org/r/534215

@Pchelolo note that you will no longer have the ability to ask Parsoid/PHP to talk with the production enwiki wiki. Strictly speaking, we can enable that (since we have the option for development), but it will be ultra slow since all api requests will be serialized and practically useless.

Hm.. this is an interesting problem. There're some tests in RESTBase + services depending on parsed in beta working for prod enwiki.. We probably need to review all these tests, copy relevant pages to beta (or create new pages with similar features) and change the tests. We need a subtask for that.

Change 536694 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid/deploy@master] Make the new betacluster VM a Parsoid scap target

https://gerrit.wikimedia.org/r/536694

Change 536694 merged by jenkins-bot:
[mediawiki/services/parsoid/deploy@master] Make the new betacluster VM a Parsoid scap target

https://gerrit.wikimedia.org/r/536694

Change 534215 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable loading Parsoid/PHP as an extension on the beta cluster

https://gerrit.wikimedia.org/r/534215

Mentioned in SAL (#wikimedia-operations) [2019-09-16T08:56:30Z] <mobrovac@deploy1001> Synchronized wmf-config/CommonSettings.php: Beta: enable the Parsoid extension - T231569 (duration: 01m 01s)

We have a conflict in beta now: Both Parsoid/JS and PHP7 use the port 8000, so on deployment-mediawiki-parsoid10 Parsoid/JS is constantly failing to start. I propose to change the port in beta and update the relevant references (proxy, RB) to resolve this. We should remember this for production roll-out later on.

Change 536975 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] Parsoid: Change the beta port to 8001 to avoid conflict with PHP7

https://gerrit.wikimedia.org/r/536975

Change 536975 merged by Effie Mouzeli:
[operations/puppet@production] Parsoid: Change the beta port to 8001 to avoid conflict with PHP7

https://gerrit.wikimedia.org/r/536975

Parsoid/JS is now running on port 8001 in beta, so it's active on both parsoid09 as well as mediawiki-parsoid10. All the references have also been updated. @ssastry could you do a deployment round of both variants and confirm everything is in order now?

Change 537554 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[mediawiki/services/parsoid/deploy@master] Bump src to d704d6bc for deploy to beta cluster

https://gerrit.wikimedia.org/r/537554

Change 537554 merged by jenkins-bot:
[mediawiki/services/parsoid/deploy@master] Bump src to d704d6bc for deploy to beta cluster

https://gerrit.wikimedia.org/r/537554

I Imagine some config is still using 8000 for testing whether the JS service is up. I don't know what that is and what needs fixing, but other than that, looks like the code got deployed. I'll verify Parsoid/JS manually shortly.

22:00:42 Started deploy [parsoid/deploy@7d2519e] (beta)
22:00:42 Deploying Rev: HEAD = 7d2519e6cf3155819ca7ddb75c81104fb62c37fe
22:00:42 Started deploy [parsoid/deploy@7d2519e] (beta): Updating Parsoid to 6fbdd703 (T231569)
22:00:42 
== DEFAULT ==
:* deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
:* deployment-parsoid09.deployment-prep.eqiad.wmflabs
parsoid/deploy: fetch stage(s): 100% (ok: 2; fail: 0; left: 0)                  
parsoid/deploy: config_deploy stage(s): 100% (ok: 2; fail: 0; left: 0)          
22:03:03 ['/usr/bin/scap', 'deploy-local', '-v', '--repo', 'parsoid/deploy', '-g', 'default', 'promote', '--refresh-config'] on deployment-parsoid09.deployment-prep.eqiad.wmflabs returned [70]: Linking config files at: /srv/deployment/parsoid/deploy-cache/revs/7d2519e6cf3155819ca7ddb75c81104fb62c37fe/.git/config-files
Check depool is empty and will not be run
Check repool is empty and will not be run
Restarting service 'parsoid'
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Port 8000 not up. Waiting 3.00s
Unhandled error:
deploy-local failed: <OSError> {}

parsoid/deploy: promote and restart_service stage(s): 100% (ok: 1; fail: 1; left: 0)
22:03:03 1 targets had deploy errors
22:03:03 1 targets failed
22:03:03 
== DEFAULT ==
:* deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
:* deployment-parsoid09.deployment-prep.eqiad.wmflabs
parsoid/deploy: finalize stage(s): 100% (ok: 2; fail: 0; left: 0)               
22:03:04 Finished deploy [parsoid/deploy@7d2519e] (beta): Updating Parsoid to 6fbdd703 (T231569) (duration: 02m 22s)
22:03:04 Finished deploy [parsoid/deploy@7d2519e] (beta) (duration: 02m 22s)

So, some other config seems to be off.

VE window spins for ever at this point. On the backend, I see deployment-parsoid09 getting the requests and successfully completing them, but looks like the response never makes it back to VE. So, I imagine there is some RESTBase config that needs updating to deal with the port changes.

@mobrovac, Handing this back to you.

Change 537592 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/parsoid/deploy@master] Beta: Set the port to 8001

https://gerrit.wikimedia.org/r/537592

Change 537592 merged by Mobrovac:
[mediawiki/services/parsoid/deploy@master] Beta: Set the port to 8001

https://gerrit.wikimedia.org/r/537592

I Imagine some config is still using 8000 for testing whether the JS service is up. I don't know what that is and what needs fixing, but other than that, looks like the code got deployed.

Oops, I forgot to tell scap to check a different port. That's now done with I31663013c80706086c8f95d3f4bd2379e45f8db0 from above.

VE window spins for ever at this point. On the backend, I see deployment-parsoid09 getting the requests and successfully completing them, but looks like the response never makes it back to VE. So, I imagine there is some RESTBase config that needs updating to deal with the port changes.

This seems to be an issue with VE, not RB or Parsoid (or configuration). When I follow your instructions, I can see that VE receives a successful diff response, but blocks for some reason:

{"visualeditoredit":{"result":"success","diff":"<table class=\"diff diff-contentalign-left\" data-mw=\"interface\">\n\t\t\t\t<col class=\"diff-marker\" />\n\t\t\t\t<col class=\"diff-content\" />\n\t\t\t\t<col class=\"diff-marker\" />\n\t\t\t\t<col class=\"diff-content\" />\n\t\t\t\t<tr class=\"diff-title\" lang=\"en\">\n\t\t\t\t<td colspan=\"2\" class=\"diff-otitle\">Latest revision</td>\n\t\t\t\t<td colspan=\"2\" class=\"diff-ntitle\">Your text</td>\n\t\t\t\t</tr><tr>\n  <td colspan=\"2\" class=\"diff-lineno\">Line 19:</td>\n  <td colspan=\"2\" class=\"diff-lineno\">Line 19:</td>\n</tr>\n<tr>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"><div>}}</div></td>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"><div>}}</div></td>\n</tr>\n<tr>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"></td>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"></td>\n</tr>\n<tr>\n  <td class=\"diff-marker\">−</td>\n  <td class=\"diff-deletedline\"><div>The '''African linsang''' (''Poiana richardsonii''), or '''oyan''', is a mammal that belongs to the civet family([[Viverridae]]).&lt;ref name=msw3/&gt;</div></td>\n  <td class=\"diff-marker\">+</td>\n  <td class=\"diff-addedline\"><div>The '''African linsang''' (''Poiana richardsonii''), or '''oyan''', is a<ins class=\"diffchange diffchange-inline\"> nice</ins> mammal that belongs to the civet family([[Viverridae]]).&lt;ref name=msw3/&gt;</div></td>\n</tr>\n<tr>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"></td>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"></td>\n</tr>\n<tr>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"><div>== Habitat ==</div></td>\n  <td class=\"diff-marker\">&#160;</td>\n  <td class=\"diff-context\"><div>== Habitat ==</div></td>\n</tr>\n</table>"}}

On the other hand, when I click on Publish changes, the edit goes through and a new revision is created.

@ssastry were you able to confirm Parsoid/PHP works as advertised on parsoid-php-beta.wmflabs.org ?

@ssastry were you able to confirm Parsoid/PHP works as advertised on parsoid-php-beta.wmflabs.org ?

I will need to figure out how to test since that domain isn't configured anywhere in config. So, I imagine I need to use some host headers / proxies or something to actually test. Do you know how I might be able to test this?

@ssastry were you able to confirm Parsoid/PHP works as advertised on parsoid-php-beta.wmflabs.org ?

I will need to figure out how to test since that domain isn't configured anywhere in config. So, I imagine I need to use some host headers / proxies or something to actually test. Do you know how I might be able to test this?

ssastry@deployment-mediawiki-parsoid10:~$ curl -v https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page -x deployment-mediawiki-parsoid10:80
<
< ... skipped ...
< 
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 18 Sep 2019 14:47:19 GMT
< Server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< X-Powered-By: HHVM/3.18.6-dev
< Cache-control: no-cache
<
< ... skipped ...
<
* Received HTTP code 500 from proxy after CONNECT

So, looks like it is being served by HHVM, not PHP 7.2 .. I'll have to look at what puppet classes are missing.

Among others there are the following Hiera settings done in role/common/parsoid/testing.yaml which affects the host scandium:

profile::mediawiki::php::php_version: "7.2"
profile::mediawiki::install_hhvm: false
profile::mediawiki::webserver::has_tls: false
has_lvs: false
profile::mediawiki::vhost_feature_flags:
  php72_only: true

If "install_hhvm" is set to false that should remove HHVM. LVS and TLS are also disabled on the standalone test box and PHP version is set to 7.2.

This can be done either in repo or in Horizon.

@mobrovac, Can I punt this puppet updates to you? We cannot do this via the horizon UI and this needs a puppet patch. I tried enabling role::parsoid::testing via the UI and it needs a parsoid_port hieradata property which isn't available right now.

@mobrovac, Can I punt this puppet updates to you? We cannot do this via the horizon UI and this needs a puppet patch. I tried enabling role::parsoid::testing via the UI and it needs a parsoid_port hieradata property which isn't available right now.

Yup, sure, I'll work on it tomorrow.

Change 538099 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[operations/puppet@production] [Beta] Parsoid: Add PHP7 config

https://gerrit.wikimedia.org/r/538099

ssastry@deployment-mediawiki-parsoid10:~$ curl -v https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page -x deployment-mediawiki-parsoid10:80

So, looks like it is being served by HHVM, not PHP 7.2 .. I'll have to look at what puppet classes are missing.

With I164223e0209aba83906ceb6de39fe3e97b0afd8a applied in Beta this now produces:

$ curl -v http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page -x deployment-mediawiki-parsoid10:80
> GET http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page HTTP/1.1
> Host: en.wikipedia.beta.wmflabs.org
> User-Agent: curl/7.52.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Date: Thu, 19 Sep 2019 20:00:32 GMT
< Server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< X-Powered-By: PHP/7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1
< X-Content-Type-Options: nosniff
< P3P: CP="See https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAutoLogin/P3P for more info."
< Content-language: en
< X-Analytics: ns=0;page_id=1
< Vary: Accept-Encoding,Cookie,Authorization
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
< Last-Modified: Thu, 19 Sep 2019 13:27:16 GMT
< Backend-Timing: D=7777437 t=1568923232847921
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8

Victory \o/.

@ssastry how can we now test that Parsoid/PHP works / is available?

Change 538099 merged by Dzahn:
[operations/puppet@production] [Beta] Parsoid: Add PHP7 config

https://gerrit.wikimedia.org/r/538099

ssastry@deployment-mediawiki-parsoid10:~$ curl -v https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page -x deployment-mediawiki-parsoid10:80

So, looks like it is being served by HHVM, not PHP 7.2 .. I'll have to look at what puppet classes are missing.

With I164223e0209aba83906ceb6de39fe3e97b0afd8a applied in Beta this now produces:

$ curl -v http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page -x deployment-mediawiki-parsoid10:80
> GET http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page HTTP/1.1
> Host: en.wikipedia.beta.wmflabs.org
> User-Agent: curl/7.52.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Date: Thu, 19 Sep 2019 20:00:32 GMT
< Server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< X-Powered-By: PHP/7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1
< X-Content-Type-Options: nosniff
< P3P: CP="See https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAutoLogin/P3P for more info."
< Content-language: en
< X-Analytics: ns=0;page_id=1
< Vary: Accept-Encoding,Cookie,Authorization
< Cache-Control: s-maxage=1209600, must-revalidate, max-age=0
< Last-Modified: Thu, 19 Sep 2019 13:27:16 GMT
< Backend-Timing: D=7777437 t=1568923232847921
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=UTF-8

Victory \o/.

@ssastry how can we now test that Parsoid/PHP works / is available?

On scandium curl -x scandium.eqiad.wmnet:80 http://en.wikipedia.org/w/rest.php/en.wikipedia.org/v3/page/html/Aam_Aadmi_Party/912987256 works. So, in principle, something similar to that. I had a few mins and did a quick tries of a couple urls but that returned http 404/500s. So, maybe try that. If not, have to check the error logs .. not sure where they end up .. presumably logstash?

From logstash-beta.wmflabs.org: PHP Fatal error: Class 'Parsoid\Config\SiteConfig' not found in /srv/deployment/parsoid/deploy-cache/revs/77630c59f081ecef658fea29861a6c2835b7add9/src/extension/src/Config/SiteConfig.php on line 29

Will take a look. I imagine some other piece of config needs a tweak.

Aha .. from https://github.com/wikimedia/mediawiki-services-parsoid-deploy/commit/401476389eb6a37a74ccc60805eae28cbe00b7c9:

This will still require a symlink in the submodule to the parent vendor/ directory. This will need to be done as part of the update_parsoid.sh script that we use to update Parsoid on scandium.

For just this once, I'll manually add the symlink, but, will need to figure out how to add this symlink as a post-scap-deploy activity. Does scap let you do this?

..... aaaannndddd .... successs .....

ssastry@deployment-mediawiki-parsoid10:/srv/deployment/parsoid/deploy/src$ curl -v http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page/388866 -x deployment-mediawiki-parsoid10:80
*   Trying 172.16.0.141...
* TCP_NODELAY set
* Connected to (nil) (172.16.0.141) port 80 (#0)
> GET http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page/388866 HTTP/1.1
> Host: en.wikipedia.beta.wmflabs.org
> User-Agent: curl/7.52.1
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Date: Thu, 19 Sep 2019 22:43:53 GMT
< Server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< X-Powered-By: PHP/7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1
< X-Content-Type-Options: nosniff
< Cache-control: no-cache
< P3P: CP="See https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAutoLogin/P3P for more info."
< Content-Language: en
< Vary: Accept
< Content-Revision-Id: 388866
< Backend-Timing: D=6743489 t=1568933033308810
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/HTML/2.1.0"

So, we just need to figure out the post-scap-deploy symlinking part.

mobrovac claimed this task.

Yay, thanks a lot @ssastry for your assistance! I am declaring this task done for its intended scope, we can follow up on the symlinking issue with scap.

I rejoiced too early :/. While locally it works, using the web proxy parsoid-php-beta from the outside does not:

$ curl http://parsoid-php-beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page -H'Host: en.wikipedia.beta.wmflabs.org'
< HTTP/1.1 404 Not Found
< Server: nginx/1.13.6
< Date: Fri, 20 Sep 2019 09:57:04 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive

The web proxy is correctly configured to connect to deployment-mediawiki-parsoid10:80, so I'm not quite sure I understand why we are getting a 404.

Interestingly, calling directly Parsoid on the node also produces a 404:

mobrovac@deployment-mediawiki-parsoid10:~$ curl -v http://localhost/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page/388866 -H'Host: en.wikipedia.beta.wmflabs.org'
< HTTP/1.1 404 Not Found
< Date: Fri, 20 Sep 2019 10:05:56 GMT
< Server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< Content-Length: 387
< Content-Type: text/html; charset=iso-8859-1

Interestingly, calling directly Parsoid on the node also produces a 404:

mobrovac@deployment-mediawiki-parsoid10:~$ curl -v http://localhost/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page/388866 -H'Host: en.wikipedia.beta.wmflabs.org'
< HTTP/1.1 404 Not Found
< Date: Fri, 20 Sep 2019 10:05:56 GMT
< Server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< Content-Length: 387
< Content-Type: text/html; charset=iso-8859-1

You need to use the -x proxy as in the earlier curl commands because only the deployment-mediaiwki-parsoid10 server has parsoid installed. So, all the others will fail.

As for testing it externally, MediaWiki config doesn't know about 'parsoid-php-beta.wmflabs.org' as a domain/proxy I think and maybe some config patch is required if you want to hit it from outside ... but @Pchelolo was going to look at it.

Ok, we have some (questionable) progress on this front. I modified the Varnish config locally on deployment-cache-text05 to include deployment-mediawiki-parsoid10 for the w/rest.php route, and it passes the requests through, but now we get back a 500:

$ curl https://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page -v
> GET /w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page HTTP/2
> Host: en.wikipedia.beta.wmflabs.org
> User-Agent: curl/7.66.0
> Accept: */*
> 
< HTTP/2 500 
< date: Wed, 02 Oct 2019 07:17:12 GMT
< content-type: text/html; charset=UTF-8
< content-length: 1510
< server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< x-powered-by: PHP/7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1
< x-content-type-options: nosniff
< cache-control: no-cache
< p3p: CP="See https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAutoLogin/P3P for more info."
< backend-timing: D=395356 t=1570000631706994
< vary: Accept-Encoding,X-Seven
< x-varnish: 204121456, 58039408
< age: 0
< x-cache: deployment-cache-text05 miss, deployment-cache-text05 miss
< x-cache-status: miss
< server-timing: cache;desc="miss"
< set-cookie: WMF-Last-Access=02-Oct-2019;Path=/;HttpOnly;secure;Expires=Sun, 03 Nov 2019 00:00:00 GMT
< set-cookie: WMF-Last-Access-Global=02-Oct-2019;Path=/;Domain=.wikipedia.beta.wmflabs.org;HttpOnly;secure;Expires=Sun, 03 Nov 2019 00:00:00 GMT
< x-analytics: https=1;nocookies=1
< x-client-ip: 89.172.143.92
< set-cookie: GeoIP=HR:::45.17:15.50:v4; Path=/; secure; Domain=.beta.wmflabs.org

[...]
PHP fatal error: Class 'Parsoid\Config\SiteConfig' not found

So we can see the request went through Varnish and landed on deployment-mediawiki-parsoid10 and that PHP7 is handling it, but there seems to be PHP-side misconfiguration now. I'm getting the same response when I run the same request on deployment-mediawiki-parsoid10:

$ curl http://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page -x deployment-mediawiki-parsoid10:80

Change 540379 had a related patch set uploaded (by Mobrovac; owner: Mobrovac):
[mediawiki/services/parsoid/deploy@master] Beta: Symlink the vendor directory after deploy

https://gerrit.wikimedia.org/r/540379

Change 540379 merged by Mobrovac:
[mediawiki/services/parsoid/deploy@master] Beta: Symlink the vendor directory after deploy

https://gerrit.wikimedia.org/r/540379

mobrovac removed a project: Patch-For-Review.

Alrighty, we are done here! I added the needed Varnish config to Horizon and the above patch creates a symlink for the vendor dir on every parsoid/deploy repo deploy. So now we finally have:

$ curl https://en.wikipedia.beta.wmflabs.org/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page/388866
< HTTP/2 200 
< date: Wed, 02 Oct 2019 11:15:29 GMT
< content-type: text/html; charset=utf-8; profile="https://www.mediawiki.org/wiki/Specs/HTML/2.1.0"
< server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< x-powered-by: PHP/7.2.22-1+0~20190902.26+debian9~1.gbpd64eb7+wmf1
< x-content-type-options: nosniff
< cache-control: no-cache
< p3p: CP="See https://en.wikipedia.beta.wmflabs.org/wiki/Special:CentralAutoLogin/P3P for more info."
< content-language: en
< content-revision-id: 388866
< backend-timing: D=1257687 t=1570014927979270
< vary: Accept, Accept-Encoding
< x-varnish: 205363361, 57355614
< accept-ranges: bytes
< age: 0
< x-cache: deployment-cache-text05 pass, deployment-cache-text05 pass
< x-cache-status: pass
< server-timing: cache;desc="pass"

Change 549635 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] beta: Update Parsoid port to 8001

https://gerrit.wikimedia.org/r/549635

Change 549635 merged by jenkins-bot:
[operations/mediawiki-config@master] beta: Update Parsoid port to 8001

https://gerrit.wikimedia.org/r/549635

DannyS712 subscribed.

[batch] remove patch for review tag from resolved tasks

Interestingly, calling directly Parsoid on the node also produces a 404:

mobrovac@deployment-mediawiki-parsoid10:~$ curl -v http://localhost/w/rest.php/en.wikipedia.beta.wmflabs.org/v3/page/html/Main_Page/388866 -H'Host: en.wikipedia.beta.wmflabs.org'
< HTTP/1.1 404 Not Found
< Date: Fri, 20 Sep 2019 10:05:56 GMT
< Server: deployment-mediawiki-parsoid10.deployment-prep.eqiad.wmflabs
< Content-Length: 387
< Content-Type: text/html; charset=iso-8859-1

Just a quick correction for future readers: the reason this command didn't work was because the local web server doesn't respond to 'localhost' (for some reason!). Using http://deployment-mediawiki-parsoid10/... in the curl command would have worked.