We have two pdns hosts for labs: labservices1001 (labs-ns2) and holmium (labs-ns3). Both use a common database on m5-master. Now that each is writing to the database, we may have race conditions. Better would be to have each server use a database running on the same server as the pdns service.
Roll-out steps:
Proposed steps:
[x] Get mysql running on holmium and labservices1001
[x] Create replicated pdns dbs on holmium and labservices, each replicating from the pdns db on m5-master
[x] disable puppet on labnet1002 and turn off the nova API
[x] disable puppet on labservices1001 and stop all designate services
** Now we should only be reading from pdns dbs, not writing to them.
[x] merge https://gerrit.wikimedia.org/r/#/c/286670/1 and apply on Holmium
[x] dig tests @ labs-ns1.wikimedia.org and labs-recursor1.wikimedia.org
[] enable puppet on labservices1001, apply, immediately kill designate services again and disable puppet
[] dig tests @ labs-ns0.wikimedia.org and labs-recursor0.wikimedia.org
** Powerdns db-reading and resolving work! Next, stop, Designate and writing
[] disable db replication, make pdns dbs read/write on both holmium and labservices1001
[] enable and run puppet on labservices1001 (thus turning designate on)
[] create a new record via Horizon, dig test @ labs-ns0.wikimedia.org and labs-recursor0.wikimedia.org and labs-ns1.wikimedia.org and labs-recursor1.wikimedia.org
[] create a new domain (via designate commandline), create a new record on that domain, dig test
[] enable nova API, create new instance and dig test
** Declare victory
[] (much later) drop the pdns database on m5-master