Page MenuHomePhabricator
Paste P12817

cookbook decomission mendelevium
ActivePublic
Actions

Authored by akosiaris on Sep 28 2020, 1:05 PM.
Tags
None
Referenced Files
F32366480: cookbook decomission mendelevium
Sep 28 2020, 1:05 PM
Subscribers
None
> done
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: ['mendelevium.eqiad.wmnet']
**Failed downtime host on Icinga (likely already removed)**
Found Ganeti VM
Shutting down VM mendelevium.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet
VM shutdown
Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
Sleeping for 20s to avoid race conditions...
Removed host mendelevium.eqiad.wmnet from Debmonitor
Removed from DebMonitor
Removed from Puppet master and PuppetDB
Issuing Ganeti remove command, it can take up to 15 minutes...
Removing VM mendelevium.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet. This may take a few minutes.
VM removed
Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
Generating the DNS records from Netbox data. It will take a couple of minutes.
Failed to run the sre.dns.netbox cookbook
Traceback (most recent call last):
File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 291, in run
dns_netbox_run(dns_netbox_args, spicerack)
File "/srv/deployment/spicerack/cookbooks/sre/dns/netbox.py", line 61, in run
results = netbox_host.run_sync(command, is_safe=True)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 476, in run_sync
batch_sleep=batch_sleep, is_safe=is_safe)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 646, in _execute
raise RemoteExecutionError(ret, 'Cumin execution failed')
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)
**Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required**
ERROR: some step failed, check the task updates.
Updated Phabricator task T263993
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL