I've re-imaged quite a lot of cloudcephosd hosts over the last few days: cloudcephosd1016 through 1024, not including (for some reason) 1021.
After the reimages, a diff has appeared in netbox for several private addresses:
2025-06-14 22:05:28,412 [INFO] Commit details: {'insertions': 0, 'deletions': 16, 'lines': 16, 'files': 5}
commit 1058996d79cf215e13b2f41748357c9be3c3f0af
Author: generate-dns-snippets <noc@wikimedia.org>
Date: Sat Jun 14 22:05:28 2025 +0000
root@cumin1002: test
diff --git a/1.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa b/1.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa
index ec2efd53..86aeb714 100644
--- a/1.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa
+++ b/1.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa
@@ -8,6 +8,5 @@
0.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1008.private.eqiad.wikimedia.cloud.
1.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1009.private.eqiad.wikimedia.cloud.
5.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1021.private.eqiad.wikimedia.cloud.
-6.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1022.private.eqiad.wikimedia.cloud.
7.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudrabbit1001.private.eqiad.wikimedia.cloud.
5.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcontrol1011.private.eqiad.wikimedia.cloud.
diff --git a/1.20.172.in-addr.arpa b/1.20.172.in-addr.arpa
index e137327c..465917e8 100644
--- a/1.20.172.in-addr.arpa
+++ b/1.20.172.in-addr.arpa
@@ -8,6 +8,5 @@
10 1H IN PTR cloudcephosd1008.private.eqiad.wikimedia.cloud.
11 1H IN PTR cloudcephosd1009.private.eqiad.wikimedia.cloud.
15 1H IN PTR cloudcephosd1021.private.eqiad.wikimedia.cloud.
-16 1H IN PTR cloudcephosd1022.private.eqiad.wikimedia.cloud.
17 1H IN PTR cloudrabbit1001.private.eqiad.wikimedia.cloud.
25 1H IN PTR cloudcontrol1011.private.eqiad.wikimedia.cloud.
diff --git a/2.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa b/2.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa
index a899a02d..99ad618f 100644
--- a/2.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa
+++ b/2.0.2.0.0.0.0.a.0.8.c.e.2.0.a.2.ip6.arpa
@@ -6,9 +6,6 @@
7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1011.private.eqiad.wikimedia.cloud.
8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1012.private.eqiad.wikimedia.cloud.
9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1013.private.eqiad.wikimedia.cloud.
-3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1020.private.eqiad.wikimedia.cloud.
-4.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1023.private.eqiad.wikimedia.cloud.
-5.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudcephosd1024.private.eqiad.wikimedia.cloud.
3.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudvirt1040.private.eqiad.wikimedia.cloud.
4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudvirt1041.private.eqiad.wikimedia.cloud.
5.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0 1H IN PTR cloudvirt1042.private.eqiad.wikimedia.cloud.
diff --git a/2.20.172.in-addr.arpa b/2.20.172.in-addr.arpa
index 16a9cb9a..9bacdffd 100644
--- a/2.20.172.in-addr.arpa
+++ b/2.20.172.in-addr.arpa
@@ -6,9 +6,6 @@
7 1H IN PTR cloudcephosd1011.private.eqiad.wikimedia.cloud.
8 1H IN PTR cloudcephosd1012.private.eqiad.wikimedia.cloud.
9 1H IN PTR cloudcephosd1013.private.eqiad.wikimedia.cloud.
-13 1H IN PTR cloudcephosd1020.private.eqiad.wikimedia.cloud.
-14 1H IN PTR cloudcephosd1023.private.eqiad.wikimedia.cloud.
-15 1H IN PTR cloudcephosd1024.private.eqiad.wikimedia.cloud.
23 1H IN PTR cloudvirt1040.private.eqiad.wikimedia.cloud.
24 1H IN PTR cloudvirt1041.private.eqiad.wikimedia.cloud.
25 1H IN PTR cloudvirt1042.private.eqiad.wikimedia.cloud.
diff --git a/wikimedia.cloud-eqiad b/wikimedia.cloud-eqiad
index e6847359..76f9d6d7 100644
--- a/wikimedia.cloud-eqiad
+++ b/wikimedia.cloud-eqiad
@@ -26,16 +26,8 @@ cloudcephosd1012.private.eqiad 1H IN A 172.20.2.8
cloudcephosd1012.private.eqiad 1H IN AAAA 2a02:ec80:a000:202::8
cloudcephosd1013.private.eqiad 1H IN A 172.20.2.9
cloudcephosd1013.private.eqiad 1H IN AAAA 2a02:ec80:a000:202::9
-cloudcephosd1020.private.eqiad 1H IN A 172.20.2.13
-cloudcephosd1020.private.eqiad 1H IN AAAA 2a02:ec80:a000:202::13
cloudcephosd1021.private.eqiad 1H IN A 172.20.1.15
cloudcephosd1021.private.eqiad 1H IN AAAA 2a02:ec80:a000:201::15
-cloudcephosd1022.private.eqiad 1H IN A 172.20.1.16
-cloudcephosd1022.private.eqiad 1H IN AAAA 2a02:ec80:a000:201::16
-cloudcephosd1023.private.eqiad 1H IN A 172.20.2.14
-cloudcephosd1023.private.eqiad 1H IN AAAA 2a02:ec80:a000:202::14
-cloudcephosd1024.private.eqiad 1H IN A 172.20.2.15
-cloudcephosd1024.private.eqiad 1H IN AAAA 2a02:ec80:a000:202::15
cloudcephosd1025.private.eqiad 1H IN A 172.20.3.5
cloudcephosd1025.private.eqiad 1H IN AAAA 2a02:ec80:a000:203::5
cloudcephosd1026.private.eqiad 1H IN A 172.20.3.6
METADATA: {"path": "/tmp/dns-c25pcHBldHM-tqgmrjor", "sha1": "1058996d79cf215e13b2f41748357c9be3c3f0af", "insertions": 0, "deletions": 16, "lines": 16, "files": 5}
================ This raises several mysteries:
- Why is reimaging messing with those addresses at all? @Volans says that it's because of syncing with puppetdb, but I don't see evidence that those addresses were ever referenced in puppet.
- Why just those four hosts, when I re-imaged more than four? Why isn't there a diff for e.g. cloudcephosd1019 as well?
- What even are those .private.eqiad.wmcloud addresses in this context? I know we use addresses like that for routing within the cloud private network, but as far as I can tell they don't actually route within the ceph network. I can't ping any of them, for example, not even the ones that still have dns refs to other osd nodes, like cloudcephosd1034. And not all ceph osd nodes even have them:
root@cloudnet1005:~# dig +short cloudcephosd1034.private.eqiad.wikimedia.cloud 172.20.4.9 root@cloudnet1005:~# ping 172.20.4.9 PING 172.20.4.9 (172.20.4.9) 56(84) bytes of data. ^C --- 172.20.4.9 ping statistics --- 6 packets transmitted, 0 received, 100% packet loss, time 5108ms root@cloudnet1005:~# dig +short cloudcephosd1035.private.eqiad.wikimedia.cloud
- What will happen when those records are removed? I'm about to find out the answer to that one, for a few hosts at least :(