Page Menu
Home
Phabricator
Search
Configure Global Search
Log In
Files
F32369709
raw-paste-data.txt
No One
Actions
View File
Edit File
Delete File
View Transforms
Subscribe
Mute Notifications
Award Token
Flag For Later
Authored By
akosiaris
Oct 1 2020, 8:25 AM
2020-10-01 08:25:51 (UTC+0)
Size
2 KB
Referenced Files
None
Subscribers
None
raw-paste-data.txt
View Options
START - Cookbook sre.hosts.decommission
ATTENTION: destructive action for 1 hosts: proton1001.eqiad.wmnet
Are you sure to proceed?
Type "done" to proceed
> done
Looking for matches in puppetmaster1001.eqiad.wmnet:/var/lib/git/operations/puppet
hieradata/common.yaml:- 10.64.0.200 # kafka-main1001.eqiad.wmnet
hieradata/role/common/restbase/production.yaml: listen_address: 10.64.0.209
Looking for matches in puppetmaster1001.eqiad.wmnet:/srv/private
Looking for matches in deploy1001.eqiad.wmnet:/srv/mediawiki-staging
Found match(es) in the Puppet or mediawiki-config repositories (see above), proceed anyway?
Type "done" to proceed
> done
Scheduling downtime on Icinga server alert1001.wikimedia.org for hosts: ['proton1001.eqiad.wmnet']
Downtimed host on Icinga
Found Ganeti VM
Shutting down VM proton1001.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet
VM shutdown
Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
Sleeping for 20s to avoid race conditions...
Removed host proton1001.eqiad.wmnet from Debmonitor
Removed from DebMonitor
Removed from Puppet master and PuppetDB
Issuing Ganeti remove command, it can take up to 15 minutes...
Removing VM proton1001.eqiad.wmnet in cluster ganeti01.svc.eqiad.wmnet. This may take a few minutes.
VM removed
Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
Generating the DNS records from Netbox data. It will take a couple of minutes.
Failed to run the sre.dns.netbox cookbook
Traceback (most recent call last):
File "/srv/deployment/spicerack/cookbooks/sre/hosts/decommission.py", line 296, in run
dns_netbox_run(dns_netbox_args, spicerack)
File "/srv/deployment/spicerack/cookbooks/sre/dns/netbox.py", line 61, in run
results = netbox_host.run_sync(command, is_safe=True)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 476, in run_sync
batch_sleep=batch_sleep, is_safe=is_safe)
File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 646, in _execute
raise RemoteExecutionError(ret, 'Cumin execution failed')
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)
**Not all affected DC(s) have been migrated to automatic DNS, a manual patch to the operations/dns repository is required**
ERROR: some step failed, check the task updates.
Updated Phabricator task T255877
END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1)
File Metadata
Details
Attached
Mime Type
text/plain; charset=utf-8
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
8579347
Default Alt Text
raw-paste-data.txt (2 KB)
Attached To
Mode
P12876 (An Untitled Masterwork)
Attached
Detach File
Event Timeline
Log In to Comment