In order to rebuild puppetmaster1001 with stretch we will need to first failover the puppet ca service to puppetmaster2001. Creating a task to prepare for this.
Puppet CA failover process for review
# Disable puppet across the fleet
## `neodymium:~$ sudo cumin -p 95 -b 100 '*' "disable-puppet 'temporarily disabled for puppet ca relocation - T189891 - godog'"`
# Ensure rsync/git (ca, private and volatile) destinations are up to date on puppetmaster2001
## /var/lib/puppet/server/ssl/ca
## /var/lib/puppet/volatile
## /srv/private/
# Make backup copies of puppetmaster[12]001:/var/lib/puppet to neodymium/sarin
# Merge change updating `puppetmaster::ca_server: puppetmaster2001.codfw.wmnet` in hiera (https://gerrit.wikimedia.org/r/c/420721/) in order to...
## Repoint puppet agents ca_server to puppetmaster2001.codfw.wmnet
## Repoint apache frontend proxypass entries to puppetmaster2001.codfw.wmnet
## Reverse the direction of the puppetmaster rsync to puppetmaster2001 -> puppetmaster1001
# Enable and run puppet on puppetmaster1001
# Enable and run puppet on puppetmaster2001
# Enable and run puppet on a few canary hosts (puppet agents)
# Enable and force puppet agent run across fleet
## open a screen/tmux on neodymium or sarin and run:
## `sudo cumin -p 70 -b 15 '*' "run-puppet-agent -q -e 'temporarily disabled for puppet ca relocation - T189891 - godog'"`