[x] build and test labspuppetmaster1001 and 1002
[x] understand and test the frontend/backend/balancing arrangement between the two
[x] apply local puppetmaster rename patch to both
[x] security audit, update
[x] switch base images/new VMs to new puppetmaster
[x] switch designate cert-cleaner to clean certs on the new puppetmaster
[x] move existing instances to new puppetmasters, update certs
[x] commit hiera patches switching puppetmaster
[x] switch default enc address in /etc/puppet/hiera.yaml to labspuppetmaster1001
[] search/replace uses of old labs-puppetmaster-eqiad name
[] depuppetize/turn off puppetmaster on labcontrol1001
[] ??? should the salt master also be moved ???
I'm going to build two new labs puppetmasters:
labspuppetmaster1001 (frontend, worker)
labspuppetmaster1002 (worker only)
These will be built using the normal production puppetmaster profiles (but will use the labs private repo and no puppet repo.) labspuppetmaster1001 will use the service name 'labs-puppetmaster.wikimedia.org'.
Once they're up and running, I'll apply local patches to both puppetmasters, like this:
```
--- a/hieradata/eqiad.yaml
+++ b/hieradata/eqiad.yaml
@@ -78,7 +78,7 @@ labs_nova_controller: &labsnovacontroller "labcontrol1001.wikimedia.org"
labs_nova_controller_spare: &labsnovacontrollerspare "labcontrol1002.wikimedia.org"
labs_glance_controller: &labsglancecontroller "labcontrol1001.wikimedia.org"
-labs_puppet_master: &labspuppetmaster "labs-puppetmaster-eqiad.wikimedia.org"
+labs_puppet_master: &labspuppetmaster "labs-puppetmaster.wikimedia.org"
labs_keystone_host: &labskeystonehost "labcontrol1001.wikimedia.org"
labs_osm_host: "wikitech.wikimedia.org"
```
That will make the new puppetmaster 'sticky' such that once a given instance is moved to the new master it'll be committed to that master ever after.
Then, I can move new hosts over to the new puppetmaster with:
```
$ grep 'server = labs-puppetmaster-eqiad.wikimedia.org' /etc/puppet/puppet.conf && sed -i 's/labs-puppetmaster-eqiad.wikimedia.org/labs-puppetmaster.wikimedia.org/g' /etc/puppet/puppet.conf && rm -rf /var/lib/puppet/ssl && puppet agent --enable && puppet agent --onetime --verbose --no-daemonize --no-splay --show_diff --waitforcert=10 —certname=`hostname -f` —server=labs-puppetmaster.wikimedia.org
```
I've tested this in labtest and it should work fine. My only real concern is that we need to make VERY sure that vms can never access the private repo, and that puppet doesn't accidentally copy the private repo onto these new puppetmasters. I'm nervous about this on account of using the same profiles as production...