As we work on making the switchover process as easy as possible it appears pretty clear to me that everything is going to be easier if we implement something like https://gerrit.wikimedia.org/r/#/c/266509 to work across all applications, and not just for mediawiki.
I'm unsure this should be a blocker, but it would definitely make things way easier for ops when doing the switchover, and would standardize definitions across different applications.
In short, we need some sort of simple, even rudimentary discovery system, and maybe integrate it into our applications.
Simple forms of this could be:
- Add dns records for all services in the various DCs, plus a set for the master
- Create a json/yaml file containing all the definitions and distribute it via puppet
- Add a series of entries to conftool and store the data into etcd. It can then be polled directly from applications, or just create a json file via confd