Added dns AAAA record to cloudcephosd10[01-15]

Added AAAA record for coludbockup and cloudcephmon, the rest will need more careful checking

@Andrew Can you give it a look too? may be related to the issues you were seeing with auth timeouts?

Clouddb: I know little to nothing about these servers. Most sql grants are only configured for ipv4, so it's possible that adding AAAA records will cause access failures. Note, though, that this is not a mixed cluster: the clouddb hosts in codfw server an entirely different purpose from those in eqiad. Bad naming :(

Cloudnet100[34]: It's definitely true that the service running on this is ipv4 only, but since the counterparts in codfw1dev work it should be safe to add.

It should be safe to add the cloudvirt records.

In T312557#8204286, @Andrew wrote:

What's involved in adding these records? Is it just filling in a field in netbox or are there separate dns steps needed?

As mentioned in the task description, it's all outlined in the wikitech page: https://wikitech.wikimedia.org/wiki/DNS/Netbox#Mixed_clusters that would point you to https://wikitech.wikimedia.org/wiki/DNS/Netbox#Add_missing_DNS_name_to_the_primary_IPv6_address where, based if you need to add to few hosts or many hosts, there are two different approaches.

@Andrew the outlined process required 2 steps. You just performed step 1 (adding the DNS names in Netbox) and not step 2 (running the sre.dns.netbox cookbook).
The lack of running step 2 is causing various issues:

1. The changes have not been deployed to the DNS so they are not live and we don't know if they are ok or might cause issues to the services on those hosts
2. It caused Icinga to alert:

Thu 22:25:33   icinga-wm| PROBLEM - Uncommitted DNS changes in Netbox on netbox1002 is CRITICAL: Netbox has uncommitted DNS changes https://wikitech.wikimedia.org/wiki/Monitoring/Netbox_DNS_uncommitted_changes

3. It's currently de-facto blocking any other Netbox changes to be propagated via the sre.dns.netbox cookbook without incurring in the risk of breaking the services on those hosts
   • In particular when people run the provision or decommissioning workflows expect a DNS diff when running the cookbook to just have the affected hosts and not spurious changes. In those cases usually the operator stops and have to go around and ask to know if it's safe to proceed or not, blocking them.

I'm raising this to the WMCS folks in EU timezones to see if we can merge those changes and unblock other changes.

FYI the current diff is P33742

@Volans I've now run the sre.dns.netbox cookbook, which completed successfully.

Just so I'm up to speed -- did you remove the AAAA records that I added before my break?