Page MenuHomePhabricator

Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing
Closed, ResolvedPublic

Description

Current porting status:

  • There are still some small bits of Parsoid/JS code / support that haven't yet fully made it over to the PHP version, we expect those to be resolved by end of July.
  • We will also probably have preliminary versions of the Parsoid REST API endpoints merged in this time period as well.
  • PHP Parser Tests script is almost ready and we anticipate we'll be quite far along to fixing test failures by the end of July.

Given all this, we should start preparing for deploying Parsoid/PHP + MediaWiki to scandium, which is our current round-trip testing host. This deployment to scandium will:

  • Since scandium is on the production cluster, Parsoid will be able to run against production wikis. This will enable us to
    • start running roundtrip tests with Parsoid/PHP and fix bugs uncovered in the process.
    • run performance tests and benchmark performance.
  • Work through deployment processes and puppetization code

Additional info / requirements:

  • scandium already has php-7 enabled
  • T213494: Installing composer modules for deployment needs resolving
  • Mediawiki should be configured in read-only mode
  • If necessary, we should not expose Parsoid API to the internet and only enable it for localhost access by the round trip testing script (which is likely going to continue be in node.js).

Event Timeline

ssastry created this task.Jul 15 2019, 3:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 15 2019, 3:17 PM
ssastry triaged this task as High priority.Jul 15 2019, 3:18 PM
ssastry updated the task description. (Show Details)Jul 15 2019, 3:21 PM
ssastry moved this task from Backlog to Deployment on the Parsoid-PHP board.Jul 16 2019, 4:09 AM

Can I ask why such parser tests will need to run in production rather than via the CI infrastructure? CI can reach the production wikis via the public interface, so I don't see a reason to keep perpetraring the off-standard thing we have with scandium, unless there are specific reasons to do so.

Can I ask why such parser tests will need to run in production rather than via the CI infrastructure? CI can reach the production wikis via the public interface, so I don't see a reason to keep perpetraring the off-standard thing we have with scandium, unless there are specific reasons to do so.

We run about 150K tests and these take upto 14 odd hours to run on the JS side. For reasonable test times, on the PHP side, these need to run in Mediawiki-integration mode where the MediaWiki instance has direct access to the production dbs. We have a mode where we can run just like JS where the Parsoid hits the public API, but that is going to extremely slow since since all the data accesses will now run sequentially without the JS async benefits. That mode only exists for one-off / short tests during development.

So, wherever we run these, we need to run on servers that have direct read-only access to the production data (or replicas of that data, doesn't have to be).

Joe added a subscriber: Mutante.EditedJul 25 2019, 10:24 AM

ok this sounds reasonable.

@Mutante I think we need to do what follows:

  • make the HHVM installation optional for profile::mediawiki::common and the rest (I'll open a subtask for that)
  • Install profile::mediawiki::webserver on scandium, with care of setting has_lvs to false
  • Add scandium in the mediawiki-installation dsh group
  • (optional) make scandium read-only in mediawiki-config

The webserver doesn't need to be publicly exposed either.

Can you take care of the steps 2-3 above once I've completed the first?

Anomie removed a subscriber: Anomie.Jul 25 2019, 2:32 PM

Change 526289 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: add mediawiki appserver profiles to role

https://gerrit.wikimedia.org/r/526289

Dzahn added a subscriber: Dzahn.Jul 30 2019, 1:04 AM

@Mutante I think we need to do what follows:

I made the mistake to create that second Phab user @Mutante when i had temp. lost access to my @Dzahn account and now i can't get rid of it. Sorry, I did not see the notification because of that.

Dzahn added a comment.EditedJul 30 2019, 1:14 AM
  • make the HHVM installation optional for profile::mediawiki::common and the rest (I'll open a subtask for that)

I amended to https://gerrit.wikimedia.org/r/c/operations/puppet/+/525584 and fixed an issue with variable names. jenkins is now happy. seems to compile fine. No change on scandium as expected. --> T228976

  • Install profile::mediawiki::webserver on scandium, with care of setting has_lvs to false

I made https://gerrit.wikimedia.org/r/c/operations/puppet/+/526289 which does a bit more but it's exactly a combination of the profiles as inlcuded in a role::mediawiki::appserver combined with the profiles from role::mediawiki::common (to avoid including role inside role).

  • Add scandium in the mediawiki-installation dsh group

That's only a conftool action nowadays and not an actual file anymore afaict.

  • (optional) make scandium read-only in mediawiki-config

I can make the patch but would let it get deployed in a SWAT window.

The webserver doesn't need to be publicly exposed either.

ACK, no ferm rules and no varnish/trafficserver config.

Can you take care of the steps 2-3 above once I've completed the first?

Yep, and we can talk about it on Thursday.

ssastry moved this task from Deployment to Testing / QA on the Parsoid-PHP board.Jul 30 2019, 4:56 PM
cscott added a subscriber: cscott.Jul 30 2019, 7:43 PM

I think we need to have scandium duplicate the PHP 7.2 configuration, since Parsoid is written to PHP 7.2 currently. This seemed to cause some trouble committing stuff to the vendor repo (https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/526262#message-53d7d95822b4bb887abccee55a041f5d665ece1b) which is a little confusing to me.

Is there some problem running PHP 7.2 on scandium?

I think we need to have scandium duplicate the PHP 7.2 configuration, since Parsoid is written to PHP 7.2 currently. This seemed to cause some trouble committing stuff to the vendor repo (https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/526262#message-53d7d95822b4bb887abccee55a041f5d665ece1b) which is a little confusing to me.
Is there some problem running PHP 7.2 on scandium?

@cscott: scandium will run php 7.2. that is not a problem. See https://phabricator.wikimedia.org/T228976.

The problem is that merging those repos to vendor repo will then push them to all application servers that don't have 7.2 now.

Change 526574 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake mcrouter keys for scandium.eqiad.wmnet

https://gerrit.wikimedia.org/r/526574

Change 526574 merged by Dzahn:
[labs/private@master] add fake mcrouter keys for scandium.eqiad.wmnet

https://gerrit.wikimedia.org/r/526574

Mutante removed a subscriber: Mutante.Jul 31 2019, 1:59 AM
Tgr added a comment.Jul 31 2019, 2:19 PM

The problem is that merging those repos to vendor repo will then push them to all application servers that don't have 7.2 now.

Strictly that shouldn't be a problem either (nothing will invoke the code). CI runs on HHVM (as well) though and it invokes composer, and that errors out since it sees impossible-to-satisfy dependencies. (Not sure why Quibble installs mediawiki/vendor, and then also runs composer...)

The concrete suggestion I'm making is in https://phabricator.wikimedia.org/T213494#5378265

If we just bundle wikipeg/zest in the Parsoid repo for a short time, then we don't need to worry about the vendor repository at all. That seems like the easiest way to avoid problems while we wait for T176370: Migrate to PHP 7 in WMF production.

Change 526289 merged by Dzahn:
[operations/puppet@production] parsoid::testing: add mediawiki appserver profiles to role

https://gerrit.wikimedia.org/r/526289

Mentioned in SAL (#wikimedia-operations) [2019-08-01T20:46:19Z] <mutante> puppetmaster: create mcrouter certs for scandium.eqiad.wmnet needed to make it an appserver (https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host) (T228069)

Change 527226 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: fix Hiera key to NOT install hhvm

https://gerrit.wikimedia.org/r/527226

Change 527226 merged by Dzahn:
[operations/puppet@production] parsoid::testing: fix Hiera key to NOT install hhvm

https://gerrit.wikimedia.org/r/527226

Mentioned in SAL (#wikimedia-operations) [2019-08-01T21:48:32Z] <mutante> scandium - apt-get remove --purge hhvm* (T228069)

Mentioned in SAL (#wikimedia-operations) [2019-08-01T22:13:21Z] <mutante> scandium apt-get remove --purge wikimedia-lvs-realserver (T228069)

Change 527232 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: temp. comment out php-restarts include

https://gerrit.wikimedia.org/r/527232

Change 527232 merged by Dzahn:
[operations/puppet@production] parsoid::testing: temp. comment out php-restarts include

https://gerrit.wikimedia.org/r/527232

Change 527291 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add scandium as an app test server to conftool data

https://gerrit.wikimedia.org/r/527291

Change 527414 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] role::parsoid::testing: remove unnecessary php additions

https://gerrit.wikimedia.org/r/527414

Change 527414 merged by Giuseppe Lavagetto:
[operations/puppet@production] role::parsoid::testing: remove unnecessary php additions

https://gerrit.wikimedia.org/r/527414

Mentioned in SAL (#wikimedia-operations) [2019-08-02T23:58:46Z] <mutante> scandium - apt-get remove --purge prometheus-hhvm-exporter - not needed here, no HHVM (T228069)

Dzahn added a comment.Aug 2 2019, 11:59 PM

@ssastry scandium should be just like a Mediawiki appserver now. There is php7.2 installed, Apache and Nginx are running. But HHVM has been removed.

Dzahn added a subscriber: Mutante.Aug 3 2019, 12:02 AM

@Mutante I think we need to do what follows:

  • make the HHVM installation optional for profile::mediawiki::common and the rest (I'll open a subtask for that)
  • Install profile::mediawiki::webserver on scandium, with care of setting has_lvs to false (via role mediawiki::appserver)
  • Add scandium in the mediawiki-installation dsh group (done by you)
  • (optional) make scandium read-only in mediawiki-config
  • The webserver doesn't need to be publicly exposed either.

Looks all done to me, except the (optional) part to make it readonly in mediawiki-config.

Change 527291 abandoned by Dzahn:
add scandium as an app test server to conftool data

https://gerrit.wikimedia.org/r/527291

Dzahn removed a subscriber: Mutante.
Dzahn assigned this task to ssastry.Aug 5 2019, 7:08 PM

Hi @ssastry i talked to Giuseppe and he made some tests of scandium's webserver and we think this is resolved now. Does it look ok to you?

Hi @ssastry i talked to Giuseppe and he made some tests of scandium's webserver and we think this is resolved now. Does it look ok to you?

Thanks! We'll pick a deployment option tomorrow (T229858) and work on that and once we have Parsoid/PHP + Mediawiki integration deployed onto scandium, I'll resolve this task.

I may potentially need help with T229363: Write and merge puppet patch to update parsoid-rt-client.config.js to talk to Parsoid/PHP and I'll ping you if I do. But, that will probably have to wait till we resolve this task. But, if you have thoughts on making that config handle both JS/PHP versions, please leave your thoughts there.

Change 529173 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/mediawiki-config@master] Make scandium a read-only appserver

https://gerrit.wikimedia.org/r/529173

Dzahn added a comment.Aug 8 2019, 9:13 PM
  • (optional) make scandium read-only in mediawiki-config

Talked to Subbu, he made https://gerrit.wikimedia.org/r/529173

The webserver doesn't need to be publicly exposed either.

Note there are existing sites https://parsoid-rt-tests.wikimedia.org/ https://parsoid-vd-tests.wikimedia.org/ on it but behind varnish/trafficserver layer. This did not change when turning it into an MW appserver.

Change 529400 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: remove parameter use_parsoid_php again

https://gerrit.wikimedia.org/r/529400

Change 529400 merged by Dzahn:
[operations/puppet@production] parsoid::testing: remove parameter use_parsoid_php again

https://gerrit.wikimedia.org/r/529400

Change 529173 abandoned by Subramanya Sastry:
Make scandium a read-only appserver enable exception logging

Reason:
Not needed. The crashers are showing up in the fatal monitor already anyway. Plus we can make this readonly by tweaking the parsoid-repo config file.

https://gerrit.wikimedia.org/r/529173