In the long term view, we want all of our cache PoPs to host, locally:
- Recursive DNS (for use by local machines only)
- NTP (ditto, peered with other pops + upstream)
- AuthDNS (for future Anycast work, see: T98006
- Possibly Prometheus
- Possibly kafka brokers + zookeeper
- Possibly etcd hosts as well?
- Possibly install-server stuff, apt mirrors, and/or webproxy
This seems like a good match for using ganeti over 3 physical hosts as an "infra" cluster within each PoP. We can also do this with 3x traditional hosts using blended roles to host some of these lightweight things together. Should sort this out in ulsfo first (as it lacks *everything* on that list) to get working configurations, then upgrade esams as well (which has some of this, but not in a ganeti cluster), and then use the same basic configuration for future PoPs as well.