All of our cache sites should ideally have local recdns and NTP servicesIn the long term view, we want all of our cache PoPs to host, and ulsfo currently does not (it relies on eqiad + codfw remotely).locally:
Plans are still being formulated about the generic shape this will take - could either be just a pair of physical boxes,1. Recursive DNS (for use by local machines only)
2. NTP (ditto, peered with other pops + upstream)
3. AuthDNS (for future Anycast work, see: T98006
4. Possibly kafka brokers + zookeeper
5. or part of a small ganeti cluster at each PoP for other misc infra Possibly etcd hosts as well?
6. Possibly install-services? If we virtualizeer stuff, we have to keep in mind that NTP servers inside virtualization don't generally work wellapt mirrors, so those might have to live outside of virtualization (but could be on the virt's physical hosts)and/or webproxy
This seems like a good match for using ganeti over 3 physical hosts as an "infra" cluster within each PoP. Either wayShould sort this out in ulsfo first (as it lacks *everything* on that list) to get working configurations, then upgrade esams as well (which has some of this, but not in a ganeti cluster), we'll probably end up ordering a little hardwareand then use the same basic configuration for future PoPs as well.