Page MenuHomePhabricator

Remove HHVM from production
Open, Needs TriagePublic

Description

We have started migrating servers to serve via PHP7 only, this is a task to discuss and track the removal of HHVM from all production servers.

  • Migrate smaller sites (noc, portals, search etc)
  • Remove hhvm conditions - puppet 539326
  • Stop monitoring hhvm (LVS, prometheus et al) - puppet 543129
  • Stop and disable hhvm service 543131
  • Remove hhvm, service files, configs
  • Prevent hhvm from being installed - puppet
  • Remove tmpreaper (also: T151304) - puppet
  • Clean up Grafana graphs
  • Remove hhvm leftovers in puppet - puppet
  • Reimage all mw* servers

Related Objects

Event Timeline

jijiki created this task.Aug 5 2019, 9:55 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 5 2019, 9:55 AM
Paladox added a subscriber: Paladox.Aug 5 2019, 2:31 PM
Dzahn added a subscriber: Dzahn.Sep 5 2019, 5:27 AM
Fito added a subscriber: Fito.Sep 6 2019, 8:56 AM

Change 538108 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] hhvm: make it possible to let puppet remove all hhvm remnants

https://gerrit.wikimedia.org/r/538108

I uploaded a patch to allow us to clean hosts from hhvm resources by setting a Hiera key to absent. The example change applies it on mwdebug1001.eqiad.wmnet.

We can use host names first and later roles to mass-remove if it seems ok. Wanna take a look?

Here is what it would remove:

https://puppet-compiler.wmflabs.org/compiler1002/18476/mwdebug1001.eqiad.wmnet/

Change 538108 abandoned by Dzahn:
hhvm: make it possible to let puppet completely remove hhvm

Reason:
per comments above and discussion in serviceops meeting today, reimage is preferrred over this

https://gerrit.wikimedia.org/r/538108

jijiki moved this task from Backlog to Next up on the serviceops board.Fri, Sep 27, 6:29 AM
jijiki updated the task description. (Show Details)Fri, Sep 27, 6:34 AM

Change 539458 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: install php-fpm on maintenance servers

https://gerrit.wikimedia.org/r/539458

Change 539459 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] noc: switch catch_all to php-fpm

https://gerrit.wikimedia.org/r/539459

Change 539465 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: switch search.wikimedia.org to PHP7

https://gerrit.wikimedia.org/r/539465

Change 539488 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: switch wwwportals to PHP7

https://gerrit.wikimedia.org/r/539488

Change 539458 merged by Effie Mouzeli:
[operations/puppet@production] hiera: install php-fpm on maintenance servers

https://gerrit.wikimedia.org/r/539458

Change 539505 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hiera: reduce php-fpm workers on maintenance servers

https://gerrit.wikimedia.org/r/539505

Change 539505 merged by Effie Mouzeli:
[operations/puppet@production] hiera: reduce php-fpm workers on maintenance servers

https://gerrit.wikimedia.org/r/539505

Change 539459 merged by Effie Mouzeli:
[operations/puppet@production] noc: switch catch_all to php-fpm

https://gerrit.wikimedia.org/r/539459

Change 539541 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] WIP: mediawiki: remove cleanup apache configs from hhvm

https://gerrit.wikimedia.org/r/539541

Mentioned in SAL (#wikimedia-operations) [2019-09-27T14:32:58Z] <effie> Disable puppet and reload apache on mw* for 539465 and 539488 - T229792

Change 539465 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: switch search.wikimedia.org to PHP7

https://gerrit.wikimedia.org/r/539465

Change 539488 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: switch wwwportals to PHP7

https://gerrit.wikimedia.org/r/539488

Change 539541 abandoned by Effie Mouzeli:
WIP: mediawiki: remove cleanup apache configs from hhvm

https://gerrit.wikimedia.org/r/539541

jijiki updated the task description. (Show Details)Tue, Oct 8, 11:19 AM
jijiki updated the task description. (Show Details)Tue, Oct 8, 11:37 AM
jijiki updated the task description. (Show Details)Tue, Oct 8, 2:08 PM

Change 539326 had a related patch set uploaded (by Effie Mouzeli; owner: Giuseppe Lavagetto):
[operations/puppet@production] mediawiki: remove the PHP/HHVM conditionals from the code

https://gerrit.wikimedia.org/r/539326

Mentioned in SAL (#wikimedia-operations) [2019-10-14T13:18:04Z] <effie> Disable puppet on mw* to remove php72_only feature flag - T229792

Change 539326 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: remove the PHP/HHVM conditionals from the code

https://gerrit.wikimedia.org/r/539326

Mentioned in SAL (#wikimedia-operations) [2019-10-14T14:13:13Z] <effie> Enable puppet on mw* servers and reload apache - T229792

Change 539128 had a related patch set uploaded (by Effie Mouzeli; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7

https://gerrit.wikimedia.org/r/539128

jijiki updated the task description. (Show Details)Tue, Oct 15, 11:59 AM

Change 543129 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: stop monitoring hhvm

https://gerrit.wikimedia.org/r/543129

Change 543131 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: stop hhvm service from all hosts

https://gerrit.wikimedia.org/r/543131

Change 543129 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: stop monitoring hhvm

https://gerrit.wikimedia.org/r/543129

Mentioned in SAL (#wikimedia-operations) [2019-10-16T09:27:01Z] <effie> Disable puppet on all hosts running hhvm to merge 543131 - T229792

Change 543131 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: stop hhvm service from all hosts

https://gerrit.wikimedia.org/r/543131

Mentioned in SAL (#wikimedia-operations) [2019-10-16T09:40:28Z] <effie> enable puppet on all hosts running hhvm - T229792

jijiki updated the task description. (Show Details)Wed, Oct 16, 12:00 PM
Krinkle added subscribers: Joe, Krinkle.EditedThu, Oct 17, 3:04 AM

Change 539128 had a related patch set uploaded (by Effie Mouzeli; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7
https://gerrit.wikimedia.org/r/539128

@jijiki @Joe I assume this is OK to roll out now, so I've merged it. We have until next Tuesday to revert/delay if I assumed incorrectly!

Change 539128 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7

https://gerrit.wikimedia.org/r/539128

jijiki moved this task from Next up to Doing on the serviceops board.Mon, Oct 21, 9:41 AM

Change 544847 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] spec: remove hhvm references from tests

https://gerrit.wikimedia.org/r/544847

Change 544864 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: make all files and packages absent by default

https://gerrit.wikimedia.org/r/544864

Change 544847 merged by Effie Mouzeli:
[operations/puppet@production] spec: remove hhvm references from tests

https://gerrit.wikimedia.org/r/544847