I have run the decommission cookbook for es1015 (T268810) but it fails on:
Generating the DNS records from Netbox data. It will take a couple of minutes.
2020-11-30 07:19:24,355 [INFO] Gathering devices, interfaces, addresses and prefixes from Netbox
2020-11-30 07:25:00,031 [INFO] Gathered 2192 devices from Netbox
2020-11-30 07:25:00,031 [INFO] Generating DNS records
2020-11-30 07:25:07,488 [INFO] Generated 12058 direct and reverse records (6029 each) in 25 direct zones and 168 reverse zones
2020-11-30 07:25:07,489 [INFO] Cloning /srv/netbox-exports/dns.git/ to /tmp/dns-c25pcHBldHM-6wpw1nde ...
2020-11-30 07:25:07,669 [INFO] Generating zonefile snippets to directory /tmp/dns-c25pcHBldHM-6wpw1nde
2020-11-30 07:25:08,340 [INFO] Committed changes: c8f31a4dd26fadd3c40127c331b5f84260ecc48f
2020-11-30 07:25:08,359 [INFO] Validating generated data
2020-11-30 07:25:08,359 [INFO] Commit details: {'insertions': 0, 'deletions': 2, 'lines': 2, 'files': 2}
commit c8f31a4dd26fadd3c40127c331b5f84260ecc48f
Author: generate-dns-snippets <noc@wikimedia.org>
Date: Mon Nov 30 07:25:08 2020 +0000
marostegui@cumin1001: es1015.eqiad.wmnet decommissioned, removing all IPs except the asset tag one
diff --git a/4.65.10.in-addr.arpa b/4.65.10.in-addr.arpa
index 1e6f271..ff06baf 100644
--- a/4.65.10.in-addr.arpa
+++ b/4.65.10.in-addr.arpa
@@ -21,7 +21,6 @@
35 1H IN PTR wmf4703.mgmt.eqiad.wmnet.
38 1H IN PTR es1013.mgmt.eqiad.wmnet.
38 1H IN PTR wmf4706.mgmt.eqiad.wmnet.
-40 1H IN PTR es1015.mgmt.eqiad.wmnet.
40 1H IN PTR wmf4708.mgmt.eqiad.wmnet.
41 1H IN PTR wmf4709.mgmt.eqiad.wmnet.
42 1H IN PTR es1017.mgmt.eqiad.wmnet.
diff --git a/mgmt.eqiad.wmnet b/mgmt.eqiad.wmnet
index 139b5b9..a55def5 100644
--- a/mgmt.eqiad.wmnet
+++ b/mgmt.eqiad.wmnet
@@ -348,7 +348,6 @@ elastic1065 1H IN A 10.65.7.106
elastic1066 1H IN A 10.65.7.107
elastic1067 1H IN A 10.65.7.108
es1013 1H IN A 10.65.4.38
-es1015 1H IN A 10.65.4.40
es1017 1H IN A 10.65.4.42
es1018 1H IN A 10.65.4.43
es1019 1H IN A 10.65.4.44
METADATA: {"path": "/tmp/dns-c25pcHBldHM-6wpw1nde", "sha1": "c8f31a4dd26fadd3c40127c331b5f84260ecc48f", "insertions": 0, "deletions": 2, "lines": 2, "files": 2}
Have you checked that the diff is OK?
Type "done" to proceed
> done
2020-11-30 07:25:13,336 [INFO] Pushed with bitflags 256: 218dc28..c8f31a4
2020-11-30 07:25:13,403 [INFO] Temporary directory /tmp/dns-c25pcHBldHM-6wpw1nde removed.
Updating the Netbox passive copies of the repository on netbox2001.wikimedia.org
Updating the authdns copies of the repository on authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
Deploying the updated zonefiles on authdns[1001,2001].wikimedia.org,dns[1001-1002,2001-2002,3001-3002,4001-4002,5001-5002].wikimedia.org
Failed to run the sre.dns.netbox cookbook
Traceback (most recent call last):
File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 351, in run
dns_netbox_run(dns_netbox_args, spicerack)
File "/srv/deployment/spicerack/cookbooks/sre/dns/netbox.py", line 132, in run
git=AUTHDNS_DNS_CHECKOUT_PATH, netbox=AUTHDNS_NETBOX_CHECKOUT_PATH))
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 476, in run_sync
batch_sleep=batch_sleep, is_safe=is_safe)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 646, in _execute
raise RemoteExecutionError(ret, 'Cumin execution failed')
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
**Failed to run the sre.dns.netbox cookbook**: Cumin execution failed (exit_code=2)
ERROR: some step failed, check the task updates.
Updated Phabricator task T268810
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)After chatting with @elukey - it looks like any of the dns servers get updated. Tried running it again to see if it was a one time thing, but it failed on the same step.
All the steps before the DNS part were ok.