Page MenuHomePhabricator
Create Task
Maniphest T228069

Deploy Parsoid-PHP with Mediawiki to scandium for RT and performance testing
Closed, ResolvedPublic
Actions

Description

Current porting status:

  • There are still some small bits of Parsoid/JS code / support that haven't yet fully made it over to the PHP version, we expect those to be resolved by end of July.
  • We will also probably have preliminary versions of the Parsoid REST API endpoints merged in this time period as well.
  • PHP Parser Tests script is almost ready and we anticipate we'll be quite far along to fixing test failures by the end of July.

Given all this, we should start preparing for deploying Parsoid/PHP + MediaWiki to scandium, which is our current round-trip testing host. This deployment to scandium will:

  • Since scandium is on the production cluster, Parsoid will be able to run against production wikis. This will enable us to
    • start running roundtrip tests with Parsoid/PHP and fix bugs uncovered in the process.
    • run performance tests and benchmark performance.
  • Work through deployment processes and puppetization code

Additional info / requirements:

  • scandium already has php-7 enabled
  • T213494: Installing composer modules for deployment needs resolving
  • Mediawiki should be configured in read-only mode
  • If necessary, we should not expose Parsoid API to the internet and only enable it for localhost access by the round trip testing script (which is likely going to continue be in node.js).

Details

SubjectRepoBranchLines +/-
Make scandium a read-only appserver + enable exception loggingoperations/mediawiki-configmaster+3 -0
parsoid::testing: remove parameter use_parsoid_php againoperations/puppetproduction+4 -12
add scandium as an app test server to conftool dataoperations/puppetproduction+1 -0
role::parsoid::testing: remove unnecessary php additionsoperations/puppetproduction+5 -7
parsoid::testing: temp. comment out php-restarts includeoperations/puppetproduction+2 -1
parsoid::testing: fix Hiera key to NOT install hhvmoperations/puppetproduction+1 -1
parsoid::testing: add mediawiki appserver profiles to roleoperations/puppetproduction+40 -6
add fake mcrouter keys for scandium.eqiad.wmnetlabs/privatemaster+0 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

ssastry created this task.Jul 15 2019, 3:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 15 2019, 3:17 PM
ssastry triaged this task as High priority.Jul 15 2019, 3:18 PM
ssastry updated the task description. (Show Details)Jul 15 2019, 3:21 PM
dr0ptp4kt subscribed.Jul 15 2019, 6:55 PM
ssastry moved this task from Backlog to Deployment on the Parsoid-PHP board.Jul 16 2019, 4:09 AM
WDoranWMF edited projects, added Platform Engineering (Parsoid PHP (CDP2)); removed Platform Engineering, Services.Jul 18 2019, 4:04 PM
WDoranWMF edited projects, added Platform Engineering (Parsoid REST API in PHP (CDP2)); removed Platform Engineering (Parsoid PHP (CDP2)).
Joe added a project: serviceops.Jul 22 2019, 8:49 AM

Can I ask why such parser tests will need to run in production rather than via the CI infrastructure? CI can reach the production wikis via the public interface, so I don't see a reason to keep perpetraring the off-standard thing we have with scandium, unless there are specific reasons to do so.

ssastry added a comment.Jul 22 2019, 3:06 PM
In T228069#5352568, @Joe wrote:

Can I ask why such parser tests will need to run in production rather than via the CI infrastructure? CI can reach the production wikis via the public interface, so I don't see a reason to keep perpetraring the off-standard thing we have with scandium, unless there are specific reasons to do so.

We run about 150K tests and these take upto 14 odd hours to run on the JS side. For reasonable test times, on the PHP side, these need to run in Mediawiki-integration mode where the MediaWiki instance has direct access to the production dbs. We have a mode where we can run just like JS where the Parsoid hits the public API, but that is going to extremely slow since since all the data accesses will now run sequentially without the JS async benefits. That mode only exists for one-off / short tests during development.

So, wherever we run these, we need to run on servers that have direct read-only access to the production data (or replicas of that data, doesn't have to be).

Joe added a subscriber: Mutante.EditedJul 25 2019, 10:24 AM

ok this sounds reasonable.

@Mutante I think we need to do what follows:

  • make the HHVM installation optional for profile::mediawiki::common and the rest (I'll open a subtask for that)
  • Install profile::mediawiki::webserver on scandium, with care of setting has_lvs to false
  • Add scandium in the mediawiki-installation dsh group
  • (optional) make scandium read-only in mediawiki-config

The webserver doesn't need to be publicly exposed either.

Can you take care of the steps 2-3 above once I've completed the first?

Anomie unsubscribed.Jul 25 2019, 2:32 PM
ssastry added a parent task: T229015: Tracking: Direct live production traffic at Parsoid/PHP.Jul 25 2019, 4:19 PM
WDoranWMF moved this task from Parsoid REST API in PHP (CDP2) to mop on the Platform Engineering board.Jul 26 2019, 7:03 PM
WDoranWMF edited projects, added Platform Team Initiatives (Parsoid REST API in PHP (CDP2)); removed Platform Engineering (Parsoid REST API in PHP (CDP2)).
gerritbot added a comment.Jul 30 2019, 1:01 AM

Change 526289 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: add mediawiki appserver profiles to role

https://gerrit.wikimedia.org/r/526289

gerritbot added a project: Patch-For-Review.Jul 30 2019, 1:01 AM
Dzahn subscribed.Jul 30 2019, 1:04 AM
In T228069#5364513, @Joe wrote:

@Mutante I think we need to do what follows:

I made the mistake to create that second Phab user @Mutante when i had temp. lost access to my @Dzahn account and now i can't get rid of it. Sorry, I did not see the notification because of that.

Dzahn added a comment.EditedJul 30 2019, 1:14 AM
In T228069#5364513, @Joe wrote:
  • make the HHVM installation optional for profile::mediawiki::common and the rest (I'll open a subtask for that)

I amended to https://gerrit.wikimedia.org/r/c/operations/puppet/+/525584 and fixed an issue with variable names. jenkins is now happy. seems to compile fine. No change on scandium as expected. --> T228976

  • Install profile::mediawiki::webserver on scandium, with care of setting has_lvs to false

I made https://gerrit.wikimedia.org/r/c/operations/puppet/+/526289 which does a bit more but it's exactly a combination of the profiles as inlcuded in a role::mediawiki::appserver combined with the profiles from role::mediawiki::common (to avoid including role inside role).

  • Add scandium in the mediawiki-installation dsh group

That's only a conftool action nowadays and not an actual file anymore afaict.

  • (optional) make scandium read-only in mediawiki-config

I can make the patch but would let it get deployed in a SWAT window.

The webserver doesn't need to be publicly exposed either.

ACK, no ferm rules and no varnish/trafficserver config.

Can you take care of the steps 2-3 above once I've completed the first?

Yep, and we can talk about it on Thursday.

ssastry added a parent task: T229356: Run roundtrip tests with Parsoid/PHP on scandium.Jul 30 2019, 4:47 PM
ssastry added a subtask: T213494: Installing composer modules for deployment.
ssastry moved this task from Deployment to Testing / QA on the Parsoid-PHP board.Jul 30 2019, 4:56 PM
cscott subscribed.Jul 30 2019, 7:43 PM

I think we need to have scandium duplicate the PHP 7.2 configuration, since Parsoid is written to PHP 7.2 currently. This seemed to cause some trouble committing stuff to the vendor repo (https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/526262#message-53d7d95822b4bb887abccee55a041f5d665ece1b) which is a little confusing to me.

Is there some problem running PHP 7.2 on scandium?

ssastry added a comment.Jul 30 2019, 7:45 PM
In T228069#5377706, @cscott wrote:

I think we need to have scandium duplicate the PHP 7.2 configuration, since Parsoid is written to PHP 7.2 currently. This seemed to cause some trouble committing stuff to the vendor repo (https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/526262#message-53d7d95822b4bb887abccee55a041f5d665ece1b) which is a little confusing to me.

Is there some problem running PHP 7.2 on scandium?

@cscott: scandium will run php 7.2. that is not a problem. See https://phabricator.wikimedia.org/T228976.

The problem is that merging those repos to vendor repo will then push them to all application servers that don't have 7.2 now.

gerritbot added a comment.Jul 31 2019, 1:51 AM

Change 526574 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake mcrouter keys for scandium.eqiad.wmnet

https://gerrit.wikimedia.org/r/526574

gerritbot added a comment.Jul 31 2019, 1:52 AM

Change 526574 merged by Dzahn:
[labs/private@master] add fake mcrouter keys for scandium.eqiad.wmnet

https://gerrit.wikimedia.org/r/526574

Dzahn mentioned this in rLPRI792b4be494a8: add fake mcrouter keys for scandium.eqiad.wmnet.Jul 31 2019, 1:57 AM
Mutante unsubscribed.Jul 31 2019, 1:59 AM
Tgr added a comment.Jul 31 2019, 2:19 PM
In T228069#5377713, @ssastry wrote:

The problem is that merging those repos to vendor repo will then push them to all application servers that don't have 7.2 now.

Strictly that shouldn't be a problem either (nothing will invoke the code). CI runs on HHVM (as well) though and it invokes composer, and that errors out since it sees impossible-to-satisfy dependencies. (Not sure why Quibble installs mediawiki/vendor, and then also runs composer...)

WDoranWMF added a project: Platform Engineering.Jul 31 2019, 3:34 PM
WDoranWMF removed a project: Platform Engineering.
cscott added a comment.Jul 31 2019, 3:43 PM

The concrete suggestion I'm making is in https://phabricator.wikimedia.org/T213494#5378265

If we just bundle wikipeg/zest in the Parsoid repo for a short time, then we don't need to worry about the vendor repository at all. That seems like the easiest way to avoid problems while we wait for T176370: Migrate to PHP 7 in WMF production.

Joe closed subtask T228976: Allow to avoid installing HHVM from the mediawiki puppet module and profile as Resolved.Aug 1 2019, 10:11 AM
gerritbot added a comment.Aug 1 2019, 6:48 PM

Change 526289 merged by Dzahn:
[operations/puppet@production] parsoid::testing: add mediawiki appserver profiles to role

https://gerrit.wikimedia.org/r/526289

Maintenance_bot removed a project: Patch-For-Review.Aug 1 2019, 7:10 PM
Stashbot added a comment.Aug 1 2019, 8:46 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-01T20:46:19Z] <mutante> puppetmaster: create mcrouter certs for scandium.eqiad.wmnet needed to make it an appserver (https://wikitech.wikimedia.org/wiki/Mcrouter#Generate_certs_for_a_new_host) (T228069)

gerritbot added a comment.Aug 1 2019, 9:36 PM

Change 527226 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: fix Hiera key to NOT install hhvm

https://gerrit.wikimedia.org/r/527226

gerritbot added a project: Patch-For-Review.Aug 1 2019, 9:36 PM
gerritbot added a comment.Aug 1 2019, 9:47 PM

Change 527226 merged by Dzahn:
[operations/puppet@production] parsoid::testing: fix Hiera key to NOT install hhvm

https://gerrit.wikimedia.org/r/527226

Stashbot added a comment.Aug 1 2019, 9:48 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-01T21:48:32Z] <mutante> scandium - apt-get remove --purge hhvm* (T228069)

Maintenance_bot removed a project: Patch-For-Review.Aug 1 2019, 10:10 PM
Stashbot added a comment.Aug 1 2019, 10:13 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-01T22:13:21Z] <mutante> scandium apt-get remove --purge wikimedia-lvs-realserver (T228069)

gerritbot added a comment.Aug 1 2019, 11:02 PM

Change 527232 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: temp. comment out php-restarts include

https://gerrit.wikimedia.org/r/527232

gerritbot added a project: Patch-For-Review.Aug 1 2019, 11:02 PM
gerritbot added a comment.Aug 1 2019, 11:14 PM

Change 527232 merged by Dzahn:
[operations/puppet@production] parsoid::testing: temp. comment out php-restarts include

https://gerrit.wikimedia.org/r/527232

Maintenance_bot removed a project: Patch-For-Review.Aug 2 2019, 12:11 AM
gerritbot added a comment.Aug 2 2019, 12:34 AM

Change 527291 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] add scandium as an app test server to conftool data

https://gerrit.wikimedia.org/r/527291

gerritbot added a project: Patch-For-Review.Aug 2 2019, 12:34 AM
gerritbot added a comment.Aug 2 2019, 6:50 AM

Change 527414 had a related patch set uploaded (by Giuseppe Lavagetto; owner: Giuseppe Lavagetto):
[operations/puppet@production] role::parsoid::testing: remove unnecessary php additions

https://gerrit.wikimedia.org/r/527414

gerritbot added a comment.Aug 2 2019, 6:51 AM

Change 527414 merged by Giuseppe Lavagetto:
[operations/puppet@production] role::parsoid::testing: remove unnecessary php additions

https://gerrit.wikimedia.org/r/527414

Stashbot added a comment.Aug 2 2019, 11:58 PM

Mentioned in SAL (#wikimedia-operations) [2019-08-02T23:58:46Z] <mutante> scandium - apt-get remove --purge prometheus-hhvm-exporter - not needed here, no HHVM (T228069)

Dzahn added a comment.Aug 2 2019, 11:59 PM

@ssastry scandium should be just like a Mediawiki appserver now. There is php7.2 installed, Apache and Nginx are running. But HHVM has been removed.

Dzahn added a subscriber: Mutante.Aug 3 2019, 12:02 AM
In T228069#5364513, @Joe wrote:

@Mutante I think we need to do what follows:

  • make the HHVM installation optional for profile::mediawiki::common and the rest (I'll open a subtask for that)
  • Install profile::mediawiki::webserver on scandium, with care of setting has_lvs to false (via role mediawiki::appserver)
  • Add scandium in the mediawiki-installation dsh group (done by you)
  • (optional) make scandium read-only in mediawiki-config
  • The webserver doesn't need to be publicly exposed either.

Looks all done to me, except the (optional) part to make it readonly in mediawiki-config.

gerritbot added a comment.Aug 3 2019, 12:23 AM

Change 527291 abandoned by Dzahn:
add scandium as an app test server to conftool data

https://gerrit.wikimedia.org/r/527291

Dzahn removed a project: Patch-For-Review.Aug 3 2019, 12:37 AM
Dzahn removed a subscriber: Mutante.
ssastry removed a subtask: T213494: Installing composer modules for deployment.Aug 5 2019, 1:56 PM
Dzahn assigned this task to ssastry.Aug 5 2019, 7:08 PM

Hi @ssastry i talked to Giuseppe and he made some tests of scandium's webserver and we think this is resolved now. Does it look ok to you?

ssastry added a comment.Aug 5 2019, 8:13 PM
In T228069#5394158, @Dzahn wrote:

Hi @ssastry i talked to Giuseppe and he made some tests of scandium's webserver and we think this is resolved now. Does it look ok to you?

Thanks! We'll pick a deployment option tomorrow (T229858) and work on that and once we have Parsoid/PHP + Mediawiki integration deployed onto scandium, I'll resolve this task.

I may potentially need help with T229363: Write and merge puppet patch to update parsoid-rt-client.config.js to talk to Parsoid/PHP and I'll ping you if I do. But, that will probably have to wait till we resolve this task. But, if you have thoughts on making that config handle both JS/PHP versions, please leave your thoughts there.

ssastry removed a parent task: T229015: Tracking: Direct live production traffic at Parsoid/PHP.Aug 5 2019, 8:48 PM
ssastry closed this task as Resolved.Aug 8 2019, 6:27 PM
ssastry closed subtask T229858: Pick a simple (short-term) deployment option for scandium as Resolved.
gerritbot added a comment.Aug 8 2019, 9:08 PM

Change 529173 had a related patch set uploaded (by Subramanya Sastry; owner: Subramanya Sastry):
[operations/mediawiki-config@master] Make scandium a read-only appserver

https://gerrit.wikimedia.org/r/529173

gerritbot added a project: Patch-For-Review.Aug 8 2019, 9:08 PM
Dzahn added a comment.Aug 8 2019, 9:13 PM
In T228069#5364513, @Joe wrote:
  • (optional) make scandium read-only in mediawiki-config

Talked to Subbu, he made https://gerrit.wikimedia.org/r/529173

The webserver doesn't need to be publicly exposed either.

Note there are existing sites https://parsoid-rt-tests.wikimedia.org/ https://parsoid-vd-tests.wikimedia.org/ on it but behind varnish/trafficserver layer. This did not change when turning it into an MW appserver.

gerritbot added a comment.Aug 9 2019, 4:43 PM

Change 529400 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] parsoid::testing: remove parameter use_parsoid_php again

https://gerrit.wikimedia.org/r/529400

gerritbot added a comment.Aug 10 2019, 2:59 AM

Change 529400 merged by Dzahn:
[operations/puppet@production] parsoid::testing: remove parameter use_parsoid_php again

https://gerrit.wikimedia.org/r/529400

gerritbot added a comment.Sep 3 2019, 9:32 PM

Change 529173 abandoned by Subramanya Sastry:
Make scandium a read-only appserver enable exception logging

Reason:
Not needed. The crashers are showing up in the fatal monitor already anyway. Plus we can make this readonly by tweaking the parsoid-repo config file.

https://gerrit.wikimedia.org/r/529173

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2019, 10:10 PM
Stashbot added a comment.Nov 14 2019, 4:06 PM

Mentioned in SAL (#wikimedia-operations) [2019-11-14T16:06:01Z] <mutante> scandium - upgrading PHP version to 7.2.24 (fyi, @subbu T228069) (T237239)

Stashbot mentioned this in T237239: Upgrade to PHP 7.2.24.Nov 14 2019, 4:06 PM
Dzahn added a comment.Nov 14 2019, 4:07 PM

@ssastry fyi, the PHP minor version on scandium got upgraded.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL