We have a test host (db1077), and we're planning to get another one, so we can at least have 1 master + 1 slave, which is useful for load-testing, debugging, and some very basic testing. But we currently have no good way of testing certain kinds of things:
- master swaps
- puppet changes
- automated reimaging
- automated provisioning
- a new tool like Orchestrator (for anything which involves changing state in our clusters)
Proposal:
Create a Cloud VPS project for testing database automation. This would use Pontoon to minimize differences to the production puppet environment.
This would contain a puppetmaster (for pontoon), a zarcillo instance, maybe a tendril instance, and a bunch of small db VMs. We'd use some dummy data sets for the databases, and have replication set up in a way that's similar to production. This would allow testing (almost) all of the above in a similar environment to how it would work in production. The exception would be auto-provisioning (which we don't currently have, in any case), where we might need to put one of our fake databases into the backup infra so we can restore it from there.
Ideally this would be fairly lightweight to run, and ~fully automated, so that if necessary the entire env can be torn down and brought back up fresh without much manual time or intervention needed.