[x] announce migration window to: ops list, wikitech-l list, Slack
[x] schedule downtime via cookbook for phab1001 and all services on it, via cookbook:
[cumin2002:~] $ sudo cookbook sre.hosts.downtime -D 14 -r 'T322250' phab1001.eqiad.wmnet
[x] confirm downtime is active in Icinga web UI (https://icinga.wikimedia.org)
[x] disable puppet on phab1001: sudo disable-puppet 'T280597'
[x] stop Apache, PHP-FPM and phd on phab1001
[phab1001:~] sudo systemctl stop apache2
[phab1001:~] sudo systemctl stop php7.3-fpm
[phab1001:~] sudo systemctl stop phd
[x] confirm there are no more PHP processes running
[phab1001:~] sudo ps aux | grep php
[x] rsync /srv/repos diff by pulling on phab1004 from phab1001:
[phab1004:/] (as root) rsync -avp --bwlimit=2m --delete rsync://phab1001.eqiad.wmnet/srv-repos/ /srv/repos/
[x] check on phab1004 if any files under /srv/repos owned by UID 497 (vcs). if so, give them to user phd
[phab1004:/] find /srv/repos -uid 497
[phab1004:/] find /srv/repos -uid 497 -exec chown phd {} \;
- find proved far too slow on a fresh rsync of the repos data. We used chmod -R phd:phd instead, accepting that everything is phd:phd and not some mix of phd:phd and phd:www-data
[x] check on phab1004 if any files under /srv/repos owned by GID 498 (aphlict). if so, give them to group phd
[phab1004:/] find /srv/repos -gid 498
[phab1004:/] find /srv/repos -gid 498 -exec chgrp phd {} \;
- find proved far too slow on a fresh rsync of the repos data. We used chmod -R phd:phd instead, accepting that everything is phd:phd and not some mix of phd:phd and phd:www-data
[x] check on phab1004 if any files under /srv/repos are owned by a user that is NOT phd
[phab1004:/] find /srv/repos ! -user phd
[x] expect this to show the PHEX repo but nothing else. decide what to do with PHEX (root-owned)
- Decision here: Only some stuff under here was root-owned, that seems likely to have been an artifact of some manual operation on phab1001
[x] output the full tree of /srv/repos and compare number of directories / files between both servers
[phab1001:/] tree -upfg > /root/repos-tree (this file will be just under 500MB of text)
[phab1001:/] tail /root/repos-tree
[phab1004:/] tree -upfg > /root/repos-tree
[phab1004:/] tail /root/repos-tree
[] optional: if not satisfied yet: copy result file from old server to new server (scp -3 ...) and run an actual diff between them
[x] set mysql ports for master and slave, specifically for eqiad (currently this happens in codfw but not in common hiera)
merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/859145 run-puppet-agent, check what happens on phab1004
[x] merge re-revert of the phabricator server name in common Hiera, run puppet, watch the changes on phab1004 and phab2002
https://gerrit.wikimedia.org/r/c/operations/puppet/+/860031
[x] run a scap deploy to phab1004
(insert command, deployment server name)
[x] enable phd service on phab1004
merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/859628 and run-puppet-agent
[x] wait a couple minutes and check phd is still running (how long?)
(if killed by puppet for any reason, it'll be every puppet run...)
[x] merge re-revert of the DNS/SPF change
https://gerrit.wikimedia.org/r/c/operations/dns/+/860032 and run "authdns-update" on ns0.wikimedia.org, syncs to other DNS servers
[x] wait about a minute and optionally use "dig phabricator.discovery.wmnet @ns0.wikimedia.org" to see it change from alias for phab1001 to an alias for phab1004
[x] informational: dumps don't need to switch, they are already on phab1004, this has happened before
[x] informational: stats emails don't need to switch, they are already on phab1004, this has happened before
testing
[x] check https://phabricator.wikimedia.org works, watch out for yellow exclamation marks / warnings for admins
[x] test aphlict works by moving something on a workboard while someone else watches
[x] test if a ticket update shows up on IRC
[x] test if email from a ticket update arrives (by a user who has email notifications)
[x] check phabricator logs for exceptions (that aren't usual noise)
(insert command / pathes)
[x] test if CI works / "recheck" on a change in Gerrit
finalizing
[] merge patch to disable phd (and apache and php-fpm) on phab1001?
[] verify proper monitoring downtime on phab1001
[] reply to list emails and Slack that migration is done succesfully, link to ticket in case they see any issues
[] publish fingerprints on wikitech page
after migration is done and grace period (how long?):
[] double check which settings can move to common Hiera, remove setting from hosts files in Hiera
[] merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/859631 and check puppet run
[] remove phab1001 from mysql grants, coordinate with DBA on merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/858419
[] create decom ticket for phab1001 - https://phabricator.wikimedia.org/T323418
[] remove production puppet role from phab1001, merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/824804
[] run decom cookbook from a cumin host on phab1001
[cumin2002:~] $ sudo cookbook sre.hosts.decommission phab1001.eqiad.wmnet -t T323418
[] remove phab1001 from site.pp https://gerrit.wikimedia.org/r/c/operations/puppet/+/858421
[] check all the SRE boxes on decom ticket, assign to dcops in eqiad, add dcops tag
[] resolve https://phabricator.wikimedia.org/T280597
[] set OKR to 100% in Betterworks, profit