We have two pdns hosts for labs: labservices1001 (labs-ns2) and holmium (labs-ns3). Both use a common database on m5-master. Now that each is writing to the database, we may have race conditions. Better would be to have each server use a database running on the same server as the pdns service.
Roll-out steps:
Proposed steps:
- Get mysql running on holmium and labservices1001
- Create replicated pdns dbs on holmium and labservices, each replicating from the pdns db on m5-master
- disable puppet on labnet1002 and turn off the nova API
- disable puppet on labservices1001 and stop all designate services
- Now we should only be reading from pdns dbs, not writing to them.
- merge https://gerrit.wikimedia.org/r/#/c/286670/1 and apply on Holmium
- dig tests @ labs-ns1.wikimedia.org and labs-recursor1.wikimedia.org
- enable puppet on labservices1001, apply, immediately kill designate services again and disable puppet
- dig tests @ labs-ns0.wikimedia.org and labs-recursor0.wikimedia.org
- Powerdns db-reading and resolving work! Next, stop, Designate and writing
- disable db replication, make pdns dbs read/write on both holmium and labservices1001
- enable and run puppet on labservices1001 (thus turning designate on)
- create a new record via Horizon, dig test @ labs-ns0.wikimedia.org and labs-recursor0.wikimedia.org and labs-ns1.wikimedia.org and labs-recursor1.wikimedia.org
- create a new domain (via designate commandline), create a new record on that domain, dig test
- enable nova API, create new instance and dig test
- Declare victory
- (much later) drop the pdns database on m5-master