In order to rebuild puppetmaster1001 with stretch we will need to first failover the puppet ca service to puppetmaster2001. Creating a task to prepare for this.
Puppet CA failover process for review
- Disable puppet across the fleet
- neodymium:~$ sudo cumin -p 95 -b 100 '*' "disable-puppet 'temporarily disabled for puppet ca relocation - T189891 - godog'"
- Ensure rsync/git (ca, private and volatile) destinations are up to date on puppetmaster2001
- Make backup copies of puppetmaster001:/var/lib/puppet to neodymium/sarin
- Merge change updating puppetmaster::ca_server: puppetmaster2001.codfw.wmnet in hiera (https://gerrit.wikimedia.org/r/c/420721/) in order to...
- Repoint puppet agents ca_server to puppetmaster2001.codfw.wmnet
- Repoint apache frontend proxypass entries to puppetmaster2001.codfw.wmnet
- Reverse the direction of the puppetmaster rsync to puppetmaster2001 -> puppetmaster1001
- Enable and run puppet on puppetmaster1001
- Enable and run puppet on puppetmaster2001
- Enable and run puppet on a few canary hosts (puppet agents)
- Enable and force puppet agent run across fleet
- open a screen/tmux on neodymium or sarin and run:
- sudo cumin -p 70 -b 15 '*' "run-puppet-agent -q -e 'temporarily disabled for puppet ca relocation - T189891 - godog'"