Page MenuHomePhabricator

Remove HHVM from production
Open, HighPublic

Description

We have started migrating servers to serve via PHP7 only, this is a task to discuss and track the removal of HHVM from all production servers.

  • Migrate smaller sites (noc, portals, search etc)
  • Remove hhvm conditions - puppet 539326
  • Stop monitoring hhvm (LVS, prometheus et al) - puppet 543129
  • Stop and disable hhvm service 543131
  • Remove hhvm, service files, configs
  • Update cookbooks/spicerack
  • Removed references from logstash and prometheus
  • Remove hhvm leftovers in puppet - puppet
  • Prevent hhvm from being installed - puppet
  • Remove tmpreaper (also: T151304) - puppet
  • Clean up Grafana graphs
  • Reimage all mw* servers T239054

Details

Related Gerrit Patches:
operations/puppet : productionmediawiki: Remove apache systemd override for tmpreaper
operations/puppet : productionRemove apache systemd override now that tmpreaper is fixed
operations/puppet : productionmediawiki: remove all hhvm related files and hieradata
operations/puppet : productionhhvm: Remove hhvm module from puppet
operations/puppet : productionlogstash: remove HHVM references
operations/puppet : productionprometheus: Remove dead HHVM code
operations/puppet : productionmediawiki: Remove HHVM references and includes
operations/puppet : productionmediawiki: Update decommission_appserver.sh
operations/puppet : productionadmin,mediawiki: Remove hhvm related sudo privileges
operations/software/spicerack : mastermediawiki: remove hhvm from stop_cronjobs()
operations/puppet : productionlogging: remove hhvm references
operations/puppet : productionprometheus: remove hhvm stats gathering and stop exporters
operations/puppet : productionhhvm: force the removal of some directories
operations/puppet : productionhhvm: remove mwrepl and /etc/hhvm/fatal-error.php
operations/puppet : productionjobrunner: rename hhvm_jobrunner_port to jobrunner_port
operations/puppet : productionhhvm: remove hhvm leftovers from apache configs
operations/puppet : productionmediawiki::webserver: disable hhvm-needs-restart cron
operations/puppet : productionhhvm: fixes in removal
operations/puppet : productionhhvm: make all files and packages absent by default
operations/puppet : productionspec: remove hhvm references from tests
mediawiki/extensions/WikimediaEvents : masterRemove the cookie extraction for sending users to PHP7
operations/puppet : productionhhvm: stop hhvm service from all hosts
operations/puppet : productionhhvm: stop monitoring hhvm
operations/puppet : productionmediawiki: remove the PHP/HHVM conditionals from the code
operations/puppet : productionWIP: mediawiki: remove cleanup apache configs from hhvm
operations/puppet : productionmediawiki: switch wwwportals to PHP7
operations/puppet : productionmediawiki: switch search.wikimedia.org to PHP7
operations/puppet : productionnoc: switch catch_all to php-fpm
operations/puppet : productionhiera: reduce php-fpm workers on maintenance servers
operations/puppet : productionhiera: install php-fpm on maintenance servers
operations/puppet : productionhhvm: make it possible to let puppet completely remove hhvm

Related Objects

StatusAssignedTask
ResolvedMaxSem
OpenNone
OpenNone
ResolvedReedy
ResolvedDaimona
ResolvedDaimona
Resolvedaaron
ResolvedMaxSem
OpenNone
ResolvedDaimona
OpenNone
ResolvedReedy
OpenNone
OpenNone
OpenNone
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
Resolvedjijiki
Openjijiki
Resolvedawight
Openjijiki
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-operations) [2019-10-14T14:13:13Z] <effie> Enable puppet on mw* servers and reload apache - T229792

Change 539128 had a related patch set uploaded (by Effie Mouzeli; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7

https://gerrit.wikimedia.org/r/539128

jijiki updated the task description. (Show Details)Oct 15 2019, 11:59 AM

Change 543129 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: stop monitoring hhvm

https://gerrit.wikimedia.org/r/543129

Change 543131 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: stop hhvm service from all hosts

https://gerrit.wikimedia.org/r/543131

Change 543129 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: stop monitoring hhvm

https://gerrit.wikimedia.org/r/543129

Mentioned in SAL (#wikimedia-operations) [2019-10-16T09:27:01Z] <effie> Disable puppet on all hosts running hhvm to merge 543131 - T229792

Change 543131 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: stop hhvm service from all hosts

https://gerrit.wikimedia.org/r/543131

Mentioned in SAL (#wikimedia-operations) [2019-10-16T09:40:28Z] <effie> enable puppet on all hosts running hhvm - T229792

jijiki updated the task description. (Show Details)Oct 16 2019, 12:00 PM
Krinkle added subscribers: Joe, Krinkle.EditedOct 17 2019, 3:04 AM

Change 539128 had a related patch set uploaded (by Effie Mouzeli; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7
https://gerrit.wikimedia.org/r/539128

@jijiki @Joe I assume this is OK to roll out now, so I've merged it. We have until next Tuesday to revert/delay if I assumed incorrectly!

Change 539128 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7

https://gerrit.wikimedia.org/r/539128

jijiki moved this task from Next up to Doing on the serviceops board.Oct 21 2019, 9:41 AM

Change 544847 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] spec: remove hhvm references from tests

https://gerrit.wikimedia.org/r/544847

Change 544864 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: make all files and packages absent by default

https://gerrit.wikimedia.org/r/544864

Change 544847 merged by Effie Mouzeli:
[operations/puppet@production] spec: remove hhvm references from tests

https://gerrit.wikimedia.org/r/544847

Change 545652 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: remove hhvm leftovers from apache configs

https://gerrit.wikimedia.org/r/545652

Mentioned in SAL (#wikimedia-operations) [2019-10-24T12:33:08Z] <effie> Stopping puppet on all hosts including the hhvm class (C:hhvm) - 544864 - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T12:56:55Z] <effie> purge hhvm hhvm-luasandbox hhvm-tidy hhvm-wikidiff2 hhvm-dbg from mw* canaries - T229792

Change 544864 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: make all files and packages absent by default

https://gerrit.wikimedia.org/r/544864

Change 545854 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: fixes in removal

https://gerrit.wikimedia.org/r/545854

Change 545854 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: fixes in removal

https://gerrit.wikimedia.org/r/545854

Mentioned in SAL (#wikimedia-operations) [2019-10-24T14:40:09Z] <effie> Remove hhvm hhvm-luasandbox hhvm-tidy hhvm-wikidiff2 hhvm-dbg from all canaries and codfw - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T14:47:43Z] <effie> run puppet on all canaries and codfw - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:09:55Z] <effie> Remove hhvm packages and enable puppet across the fleet - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:18:22Z] <effie> Slowly reload apache across the fleet (as we are enabling puppet) - T229792

jijiki updated the task description. (Show Details)Oct 24 2019, 3:59 PM
Krinkle removed a subscriber: Krinkle.Oct 24 2019, 4:18 PM

Change 545950 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki::webserver: disable hhvm-needs-restart cron

https://gerrit.wikimedia.org/r/545950

Change 545950 merged by Dzahn:
[operations/puppet@production] mediawiki::webserver: disable hhvm-needs-restart cron

https://gerrit.wikimedia.org/r/545950

Dzahn added a comment.Oct 24 2019, 9:56 PM

merged the above because we were getting cron spam from appservers with "/usr/local/bin/hhvm-needs-restart: not found"

Mentioned in SAL (#wikimedia-operations) [2019-10-24T22:01:05Z] <mutante> mw1270 - was alerting in Icinga as degraded systemd state - reason was 'hhvm.service not-found". systemctl reset-failed cleared it. could cause monitoring spam on more servers (T229792)

jijiki added a comment.EditedOct 25 2019, 4:13 PM

merged the above because we were getting cron spam from appservers with "/usr/local/bin/hhvm-needs-restart: not found"

Thank you for taking care of it, I failed to notice it !

Mentioned in SAL (#wikimedia-operations) [2019-10-28T11:33:28Z] <effie> Disable puppet on mw* for 545652 - T229792

Change 545652 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: remove hhvm leftovers from apache configs

https://gerrit.wikimedia.org/r/545652

Change 546461 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] jobrunner: rename hhvm_jobrunner_port to jobrunner_port

https://gerrit.wikimedia.org/r/546461

Change 546461 merged by Effie Mouzeli:
[operations/puppet@production] jobrunner: rename hhvm_jobrunner_port to jobrunner_port

https://gerrit.wikimedia.org/r/546461

Mentioned in SAL (#wikimedia-operations) [2019-10-28T13:05:47Z] <effie> enable puppet on mw2* servers, depool and repool to reload apache - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-28T13:13:06Z] <effie> enable puppet on mw[1261-1265].eqiad.wmnet (mw canaries), depool and repool to reload apache - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-28T13:23:08Z] <effie> enable puppet on mw1*, depool and repool to reload apache - T229792

Change 547079 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: fremove mwrepl and /etc/hhvm/fatal-error.php

https://gerrit.wikimedia.org/r/547079

Change 547144 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] (WIP) prometheus: remove hhvm stats gathering

https://gerrit.wikimedia.org/r/547144

Change 547079 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: remove mwrepl and /etc/hhvm/fatal-error.php

https://gerrit.wikimedia.org/r/547079

Change 547227 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: force the removal of some directories

https://gerrit.wikimedia.org/r/547227

Change 547227 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: force the removal of some directories

https://gerrit.wikimedia.org/r/547227

Change 547489 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] (WIP) logging: remove hhvm references in filters

https://gerrit.wikimedia.org/r/547489

jijiki triaged this task as High priority.Oct 31 2019, 11:39 AM

Change 547714 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/software/spicerack@master] mediawiki: remove hhvm from stop_cronjobs()

https://gerrit.wikimedia.org/r/547714

jijiki updated the task description. (Show Details)Nov 1 2019, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-01T10:33:08Z] <effie> Disable puppet on mediawiki and prometheus servers to remove hhvm exporters - T229792

Change 547144 merged by Effie Mouzeli:
[operations/puppet@production] prometheus: remove hhvm stats gathering and stop exporters

https://gerrit.wikimedia.org/r/547144

Mentioned in SAL (#wikimedia-operations) [2019-11-01T10:54:39Z] <effie> remove prometheus-hhvm-exporter package from mw* servers - T229792

Change 547489 merged by Effie Mouzeli:
[operations/puppet@production] logging: remove hhvm references

https://gerrit.wikimedia.org/r/547489

jijiki updated the task description. (Show Details)Nov 4 2019, 8:43 AM

Change 547714 merged by jenkins-bot:
[operations/software/spicerack@master] mediawiki: remove hhvm from stop_cronjobs()

https://gerrit.wikimedia.org/r/547714

jijiki updated the task description. (Show Details)Nov 4 2019, 9:30 AM

Change 550483 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] admin: Remove hhvm related sudo privileges

https://gerrit.wikimedia.org/r/550483

Change 550818 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: Remove HHVM references and includes

https://gerrit.wikimedia.org/r/550818

Change 551161 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] prometheus: Remove dead HHVM code

https://gerrit.wikimedia.org/r/551161

Change 551162 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: Update decommission_appserver.sh

https://gerrit.wikimedia.org/r/551162

Change 551524 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] logstash: remove HHVM references

https://gerrit.wikimedia.org/r/551524

Change 551526 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: Remove hhvm module from puppet

https://gerrit.wikimedia.org/r/551526

Change 551527 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] (WIP) mediawiki: remove all hhvm related files and hieradata

https://gerrit.wikimedia.org/r/551527

Change 550483 merged by Effie Mouzeli:
[operations/puppet@production] admin,mediawiki: Remove hhvm related sudo privileges

https://gerrit.wikimedia.org/r/550483

Change 551162 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: Update decommission_appserver.sh

https://gerrit.wikimedia.org/r/551162

Mentioned in SAL (#wikimedia-operations) [2019-11-19T09:40:12Z] <effie> disable puppet on P:mediawiki::php - T229792

Change 550818 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: Remove HHVM references and includes

https://gerrit.wikimedia.org/r/550818

Change 551161 merged by Effie Mouzeli:
[operations/puppet@production] prometheus: Remove dead HHVM code

https://gerrit.wikimedia.org/r/551161

Change 551524 merged by Effie Mouzeli:
[operations/puppet@production] logstash: remove HHVM references

https://gerrit.wikimedia.org/r/551524

Change 551526 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: Remove hhvm module from puppet

https://gerrit.wikimedia.org/r/551526

Change 551527 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: remove all hhvm related files and hieradata

https://gerrit.wikimedia.org/r/551527

Change 489982 had a related patch set uploaded (by Effie Mouzeli; owner: Muehlenhoff):
[operations/puppet@production] Remove apache systemd override now that tmpreaper is fixed

https://gerrit.wikimedia.org/r/489982

Change 553052 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: Remove apache systemd override for tmpreaper

https://gerrit.wikimedia.org/r/553052

Change 489982 abandoned by Effie Mouzeli:
Remove apache systemd override now that tmpreaper is fixed

Reason:
Rebase hell, resubmitted
in Ie62c8c6a11fb89d5f1dbf9985a59a25d93aac65c

https://gerrit.wikimedia.org/r/489982

Change 553052 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: Remove apache systemd override for tmpreaper

https://gerrit.wikimedia.org/r/553052

Mentioned in SAL (#wikimedia-operations) [2019-11-26T13:03:46Z] <effie> Remove tmpreaper package from all mediawiki servers - T229792

jijiki updated the task description. (Show Details)Fri, Dec 6, 9:08 AM