While using the decom cookbook, trying to remove a ganeti VM, spicerack errored making requests to the Ganeti RAPI, and exited with return code 1.
The VM that was to be deleted was planet1002.eqiad.wmnet.
Traceback (most recent call last):
File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 452, in run
self._decommission_host(fqdn)
File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 329, in _decommission_host
virtual_machine = ganeti.instance(fqdn, cluster=ganeti_cluster)
File "/usr/lib/python3/dist-packages/spicerack/ganeti.py", line 400, in instance
master = self.rapi(cluster).master
File "/usr/lib/python3/dist-packages/spicerack/ganeti.py", line 118, in master
return self.info.get("master")
File "/usr/lib/python3/dist-packages/spicerack/ganeti.py", line 108, in info
return self._api_get_request("info")
File "/usr/lib/python3/dist-packages/spicerack/ganeti.py", line 93, in _api_get_request
raise GanetiError("Error while performing request to RAPI") from ex
spicerack.ganeti.GanetiError: Error while performing request to RAPI
**Host steps raised exception**: Error while performing request to RAPI
Released lock for key /spicerack/locks/custom/sre.hosts.decommission:planet1002: {'concurrency': 1, 'created': '2023-12-13 20:28:49.763518', 'owner': 'dzahn@cumin1001 [2452329]', 'ttl': 600}
Sleeping for 3 minutes to get netbox caches in sync
Acquired lock for key /spicerack/locks/cookbooks/sre.dns.netbox: {'concurrency': 1, 'created': '2023-12-13 20:32:12.394279', 'owner': 'dzahn@cumin1001 [2452329]', 'ttl': 300}
START - Cookbook sre.dns.netbox
...
ERROR: some step failed, check the task updates.
Updated Phabricator task T348392
Released lock for key /spicerack/locks/cookbooks/sre.hosts.decommission: {'concurrency': 20, 'created': '2023-12-13 20:28:41.472721', 'owner': 'dzahn@cumin1001 [2452329]', 'ttl': 1800}
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts planet1002.eqiad.wmnet