Page MenuHomePhabricator

Remove HHVM from production
Closed, ResolvedPublic

Description

We have started migrating servers to serve via PHP7 only, this is a task to discuss and track the removal of HHVM from all production servers.

  • Migrate smaller sites (noc, portals, search etc)
  • Remove hhvm conditions - puppet 539326
  • Stop monitoring hhvm (LVS, prometheus et al) - puppet 543129
  • Stop and disable hhvm service 543131
  • Remove hhvm, service files, configs
  • Update cookbooks/spicerack
  • Removed references from logstash and prometheus
  • Remove hhvm leftovers in puppet - puppet
  • Prevent hhvm from being installed - puppet
  • Remove tmpreaper (also: T151304) - puppet
  • Clean up Grafana graphs
  • Reimage all mw* servers T239054

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+0 -71
operations/puppetproduction+0 -68
operations/puppetproduction+0 -66
operations/puppetproduction+13 -409
operations/puppetproduction+0 -2 K
operations/puppetproduction+0 -20
operations/puppetproduction+0 -62
operations/puppetproduction+3 -119
operations/puppetproduction+4 -4
operations/puppetproduction+0 -4
operations/software/spicerackmaster+1 -3
operations/puppetproduction+2 -76
operations/puppetproduction+11 -41
operations/puppetproduction+4 -1
operations/puppetproduction+9 -2
operations/puppetproduction+4 -0
operations/puppetproduction+8 -21
operations/puppetproduction+1 -1
operations/puppetproduction+4 -4
operations/puppetproduction+25 -18
operations/puppetproduction+10 -14
mediawiki/extensions/WikimediaEventsmaster+3 -54
operations/puppetproduction+2 -1
operations/puppetproduction+6 -109
operations/puppetproduction+13 -136
operations/puppetproduction+0 -27
operations/puppetproduction+2 -3
operations/puppetproduction+4 -2
operations/puppetproduction+14 -3
operations/puppetproduction+1 -1
operations/puppetproduction+2 -3
operations/puppetproduction+29 -11
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
ResolvedMaxSem
ResolvedNone
ResolvedReedy
ResolvedDaimona
ResolvedDaimona
Resolvedaaron
ResolvedMaxSem
ResolvedReedy
ResolvedDaimona
OpenNone
ResolvedReedy
ResolvedJdforrester-WMF
OpenNone
OpenNone
ResolvedJdforrester-WMF
ResolvedJdforrester-WMF
Resolvedjijiki
Resolvedjijiki
Resolvedawight
Resolvedjijiki

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
jijiki updated the task description. (Show Details)Oct 16 2019, 12:00 PM
Krinkle added subscribers: Joe, Krinkle.EditedOct 17 2019, 3:04 AM

Change 539128 had a related patch set uploaded (by Effie Mouzeli; owner: Giuseppe Lavagetto):
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7

https://gerrit.wikimedia.org/r/539128

@jijiki @Joe I assume this is OK to roll out now, so I've merged it. We have until next Tuesday to revert/delay if I assumed incorrectly!

Change 539128 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@master] Remove the cookie extraction for sending users to PHP7

https://gerrit.wikimedia.org/r/539128

jijiki moved this task from Next up 🥌 to Doing 😎 on the serviceops board.Oct 21 2019, 9:41 AM

Change 544847 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] spec: remove hhvm references from tests

https://gerrit.wikimedia.org/r/544847

Change 544864 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: make all files and packages absent by default

https://gerrit.wikimedia.org/r/544864

Change 544847 merged by Effie Mouzeli:
[operations/puppet@production] spec: remove hhvm references from tests

https://gerrit.wikimedia.org/r/544847

Change 545652 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: remove hhvm leftovers from apache configs

https://gerrit.wikimedia.org/r/545652

Mentioned in SAL (#wikimedia-operations) [2019-10-24T12:33:08Z] <effie> Stopping puppet on all hosts including the hhvm class (C:hhvm) - 544864 - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T12:56:55Z] <effie> purge hhvm hhvm-luasandbox hhvm-tidy hhvm-wikidiff2 hhvm-dbg from mw* canaries - T229792

Change 544864 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: make all files and packages absent by default

https://gerrit.wikimedia.org/r/544864

Change 545854 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: fixes in removal

https://gerrit.wikimedia.org/r/545854

Change 545854 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: fixes in removal

https://gerrit.wikimedia.org/r/545854

Mentioned in SAL (#wikimedia-operations) [2019-10-24T14:40:09Z] <effie> Remove hhvm hhvm-luasandbox hhvm-tidy hhvm-wikidiff2 hhvm-dbg from all canaries and codfw - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T14:47:43Z] <effie> run puppet on all canaries and codfw - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:09:55Z] <effie> Remove hhvm packages and enable puppet across the fleet - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:18:22Z] <effie> Slowly reload apache across the fleet (as we are enabling puppet) - T229792

jijiki updated the task description. (Show Details)Oct 24 2019, 3:59 PM
Krinkle removed a subscriber: Krinkle.Oct 24 2019, 4:18 PM

Change 545950 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] mediawiki::webserver: disable hhvm-needs-restart cron

https://gerrit.wikimedia.org/r/545950

Change 545950 merged by Dzahn:
[operations/puppet@production] mediawiki::webserver: disable hhvm-needs-restart cron

https://gerrit.wikimedia.org/r/545950

Dzahn added a comment.Oct 24 2019, 9:56 PM

merged the above because we were getting cron spam from appservers with "/usr/local/bin/hhvm-needs-restart: not found"

Mentioned in SAL (#wikimedia-operations) [2019-10-24T22:01:05Z] <mutante> mw1270 - was alerting in Icinga as degraded systemd state - reason was 'hhvm.service not-found". systemctl reset-failed cleared it. could cause monitoring spam on more servers (T229792)

jijiki added a comment.EditedOct 25 2019, 4:13 PM

merged the above because we were getting cron spam from appservers with "/usr/local/bin/hhvm-needs-restart: not found"

Thank you for taking care of it, I failed to notice it !

Mentioned in SAL (#wikimedia-operations) [2019-10-28T11:33:28Z] <effie> Disable puppet on mw* for 545652 - T229792

Change 545652 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: remove hhvm leftovers from apache configs

https://gerrit.wikimedia.org/r/545652

Change 546461 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] jobrunner: rename hhvm_jobrunner_port to jobrunner_port

https://gerrit.wikimedia.org/r/546461

Change 546461 merged by Effie Mouzeli:
[operations/puppet@production] jobrunner: rename hhvm_jobrunner_port to jobrunner_port

https://gerrit.wikimedia.org/r/546461

Mentioned in SAL (#wikimedia-operations) [2019-10-28T13:05:47Z] <effie> enable puppet on mw2* servers, depool and repool to reload apache - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-28T13:13:06Z] <effie> enable puppet on mw[1261-1265].eqiad.wmnet (mw canaries), depool and repool to reload apache - T229792

Mentioned in SAL (#wikimedia-operations) [2019-10-28T13:23:08Z] <effie> enable puppet on mw1*, depool and repool to reload apache - T229792

Change 547079 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: fremove mwrepl and /etc/hhvm/fatal-error.php

https://gerrit.wikimedia.org/r/547079

Change 547144 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] (WIP) prometheus: remove hhvm stats gathering

https://gerrit.wikimedia.org/r/547144

Change 547079 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: remove mwrepl and /etc/hhvm/fatal-error.php

https://gerrit.wikimedia.org/r/547079

Change 547227 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: force the removal of some directories

https://gerrit.wikimedia.org/r/547227

Change 547227 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: force the removal of some directories

https://gerrit.wikimedia.org/r/547227

Change 547489 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] (WIP) logging: remove hhvm references in filters

https://gerrit.wikimedia.org/r/547489

jijiki triaged this task as High priority.Oct 31 2019, 11:39 AM

Change 547714 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/software/spicerack@master] mediawiki: remove hhvm from stop_cronjobs()

https://gerrit.wikimedia.org/r/547714

jijiki updated the task description. (Show Details)Nov 1 2019, 9:43 AM

Mentioned in SAL (#wikimedia-operations) [2019-11-01T10:33:08Z] <effie> Disable puppet on mediawiki and prometheus servers to remove hhvm exporters - T229792

Change 547144 merged by Effie Mouzeli:
[operations/puppet@production] prometheus: remove hhvm stats gathering and stop exporters

https://gerrit.wikimedia.org/r/547144

Mentioned in SAL (#wikimedia-operations) [2019-11-01T10:54:39Z] <effie> remove prometheus-hhvm-exporter package from mw* servers - T229792

Change 547489 merged by Effie Mouzeli:
[operations/puppet@production] logging: remove hhvm references

https://gerrit.wikimedia.org/r/547489

jijiki updated the task description. (Show Details)Nov 4 2019, 8:43 AM

Change 547714 merged by jenkins-bot:
[operations/software/spicerack@master] mediawiki: remove hhvm from stop_cronjobs()

https://gerrit.wikimedia.org/r/547714

jijiki updated the task description. (Show Details)Nov 4 2019, 9:30 AM

Change 550483 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] admin: Remove hhvm related sudo privileges

https://gerrit.wikimedia.org/r/550483

Change 550818 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: Remove HHVM references and includes

https://gerrit.wikimedia.org/r/550818

Change 551161 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] prometheus: Remove dead HHVM code

https://gerrit.wikimedia.org/r/551161

Change 551162 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: Update decommission_appserver.sh

https://gerrit.wikimedia.org/r/551162

Change 551524 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] logstash: remove HHVM references

https://gerrit.wikimedia.org/r/551524

Change 551526 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] hhvm: Remove hhvm module from puppet

https://gerrit.wikimedia.org/r/551526

Change 551527 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] (WIP) mediawiki: remove all hhvm related files and hieradata

https://gerrit.wikimedia.org/r/551527

Change 550483 merged by Effie Mouzeli:
[operations/puppet@production] admin,mediawiki: Remove hhvm related sudo privileges

https://gerrit.wikimedia.org/r/550483

Change 551162 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: Update decommission_appserver.sh

https://gerrit.wikimedia.org/r/551162

Mentioned in SAL (#wikimedia-operations) [2019-11-19T09:40:12Z] <effie> disable puppet on P:mediawiki::php - T229792

Change 550818 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: Remove HHVM references and includes

https://gerrit.wikimedia.org/r/550818

Change 551161 merged by Effie Mouzeli:
[operations/puppet@production] prometheus: Remove dead HHVM code

https://gerrit.wikimedia.org/r/551161

Change 551524 merged by Effie Mouzeli:
[operations/puppet@production] logstash: remove HHVM references

https://gerrit.wikimedia.org/r/551524

Change 551526 merged by Effie Mouzeli:
[operations/puppet@production] hhvm: Remove hhvm module from puppet

https://gerrit.wikimedia.org/r/551526

Change 551527 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: remove all hhvm related files and hieradata

https://gerrit.wikimedia.org/r/551527

Change 489982 had a related patch set uploaded (by Effie Mouzeli; owner: Muehlenhoff):
[operations/puppet@production] Remove apache systemd override now that tmpreaper is fixed

https://gerrit.wikimedia.org/r/489982

Change 553052 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] mediawiki: Remove apache systemd override for tmpreaper

https://gerrit.wikimedia.org/r/553052

Change 489982 abandoned by Effie Mouzeli:
Remove apache systemd override now that tmpreaper is fixed

Reason:
Rebase hell, resubmitted
in Ie62c8c6a11fb89d5f1dbf9985a59a25d93aac65c

https://gerrit.wikimedia.org/r/489982

Change 553052 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: Remove apache systemd override for tmpreaper

https://gerrit.wikimedia.org/r/553052

Mentioned in SAL (#wikimedia-operations) [2019-11-26T13:03:46Z] <effie> Remove tmpreaper package from all mediawiki servers - T229792

jijiki updated the task description. (Show Details)Dec 6 2019, 9:08 AM

Change 556282 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[operations/puppet@production] mediawiki: Remove unused HHVM files

https://gerrit.wikimedia.org/r/556282

Change 556282 merged by Effie Mouzeli:
[operations/puppet@production] mediawiki: Remove unused HHVM files

https://gerrit.wikimedia.org/r/556282

jijiki updated the task description. (Show Details)
jijiki closed this task as Resolved.Jan 5 2020, 8:45 PM
jijiki updated the task description. (Show Details)
Dzahn awarded a token.Jan 6 2020, 8:58 PM
Dzahn rescinded a token.Jan 6 2020, 9:02 PM
Dzahn awarded a token.