The only real way to do service discovery on labs right now is to query the wikitech API, which allows us to list instances per project. This, however, is going away at some point as we try to get rid of OpenStackManager from wikitech. It also has some load concerns, since every hit queries the nova API.
I propose the following instead:
Have a cron job that runs every 5-10mins, generates all the info we could possibly need in a nice JSON format and dump it somewhere it could be server as a static file. Code would then just hit this to get the data it needs. This would run on silver. This would also allow us to do things that we can't do in code that can be called arbitrary number of times.
- Prometheus service discovery (hits wikitech API now)
- Shinken service discovery (hits wikitech API now)
- clush service discovery (hits wikitech API now)