cookbooks.sre.hosts.decommission executed by andrew@cumin1001 for hosts: cloudvirt1015.eqiad.wmnet

• cloudvirt1015.eqiad.wmnet (PASS)
  • Downtimed host on Icinga
  • Found physical host
  • Downtimed management interface on Icinga
  • Wiped bootloaders
  • Powered off
  • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by andrew@cumin1001 for hosts: cloudvirt[1001-1009].eqiad.wmnet

• cloudvirt1001.eqiad.wmnet (PASS)
  • Downtimed host on Icinga
  • Found physical host
  • Downtimed management interface on Icinga
  • Wiped bootloaders
  • Powered off
  • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB

cookbooks.sre.hosts.decommission executed by jayme@cumin1001 for hosts: etcd[1001-1003].eqiad.wmnet

• etcd1001.eqiad.wmnet (WARN)
  • Downtimed host on Icinga
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Site eqiad DNS records not yet migrated to the automatic system, manual patch required

Completed auto-reimage of hosts:

['maps2005.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

maps2005.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009151634_pt1979_20492_maps2005_codfw_wmnet.log.

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009142120_volans_11195_sretest1001_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009142051_volans_3184_sretest1001_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009142024_volans_29184_sretest1001_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['sretest1001.eqiad.wmnet']

Script wmf-auto-reimage was launched by volans on cumin1001.eqiad.wmnet for hosts:

sretest1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009141834_volans_6008_sretest1001_eqiad_wmnet.log.

cookbooks.sre.hosts.decommission executed by volans@cumin1001 for hosts: debmonitor2001.codfw.wmnet

• debmonitor2001.codfw.wmnet (FAIL)
  • Failed downtime host on Icinga (likely already removed)
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Site codfw DNS records not yet migrated to the automatic system, manual patch required

cookbooks.sre.hosts.decommission executed by jmm@cumin1001 for hosts: debmonitor1001.eqiad.wmnet

• debmonitor1001.eqiad.wmnet (WARN)
  • Downtimed host on Icinga
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.eqiad.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Site eqiad DNS records not yet migrated to the automatic system, manual patch required

cookbooks.sre.hosts.decommission executed by volans@cumin1001 for hosts: debmonitor2001.codfw.wmnet

• debmonitor2001.codfw.wmnet (WARN)
  • Downtimed host on Icinga
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster ganeti01.svc.codfw.wmnet to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Site codfw DNS records not yet migrated to the automatic system, manual patch required

Icinga downtime for 7 days, 0:00:00 set by jayme@cumin1001 on 3 host(s) and their services with reason: shutting down, host sheduled for decommission

etcd[1001-1003].eqiad.wmnet

Completed auto-reimage of hosts:

['es2034.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2034.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009102344_pt1979_25423_es2034_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2033.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2033.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009102255_pt1979_15332_es2033_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2032.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2032.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009102213_pt1979_8632_es2032_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2031.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2031.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009102133_pt1979_32214_es2031_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2030.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2030.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009102051_pt1979_23945_es2030_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2029.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2029.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009102005_pt1979_13731_es2029_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2028.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2028.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009101807_pt1979_25637_es2028_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2027.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2027.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009101624_pt1979_8604_es2027_codfw_wmnet.log.

Completed auto-reimage of hosts:

['es2027.codfw.wmnet']

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

es2027.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009101604_pt1979_4360_es2027_codfw_wmnet.log.

Completed auto-reimage of hosts:

['db2141.codfw.wmnet']

Script wmf-auto-reimage was launched by jynus on cumin2001.codfw.wmnet for hosts:

['db2141.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009100831_jynus_19743.log.

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

db2141.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009091924_pt1979_17781_db2141_codfw_wmnet.log.

Completed auto-reimage of hosts:

['db1133.eqiad.wmnet']

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['db1133.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009081101_jynus_9028.log.

Icinga downtime for 4:00:00 set by jayme@cumin1001 on 18 host(s) and their services with reason: Reinitialize eqiad k8s cluster with new etcd

argon.eqiad.wmnet,chlorine.eqiad.wmnet,kubernetes[1001-1016].eqiad.wmnet

Completed auto-reimage of hosts:

['es2026.codfw.wmnet']

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es2026.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009080515_marostegui_32677.log.

Completed auto-reimage of hosts:

['an-worker1110.eqiad.wmnet']

Completed auto-reimage of hosts:

['es2026.codfw.wmnet']

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1110.eqiad.wmnet', 'an-worker1101.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071557_elukey_6414.log.

Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts:

['es2026.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071502_marostegui_20879.log.

Completed auto-reimage of hosts:

['relforge1003.eqiad.wmnet', 'relforge1004.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['relforge1003.eqiad.wmnet', 'relforge1004.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071412_elukey_7772.log.

Completed auto-reimage of hosts:

['an-worker1097.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1097.eqiad.wmnet', 'an-worker1098.eqiad.wmnet', 'an-worker1099.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071300_elukey_2391.log.

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1102.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1096.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009071014_elukey_16133.log.

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1102.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009070947_elukey_7498.log.

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1096.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009070938_elukey_6717.log.

Completed auto-reimage of hosts:

['an-worker1096.eqiad.wmnet']

Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts:

['an-worker1096.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/202009070853_elukey_28208.log.

Completed auto-reimage of hosts:

['an-worker1117.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1114.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1111.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1113.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1116.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1115.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1104.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1109.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1108.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1110.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1112.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1105.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1103.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1106.eqiad.wmnet']

Completed auto-reimage of hosts:

['an-worker1107.eqiad.wmnet']

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1117.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042025_cmjohnson_22173_an-worker1117_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1116.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042025_cmjohnson_21677_an-worker1116_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1115.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042024_cmjohnson_21114_an-worker1115_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1114.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042024_cmjohnson_20549_an-worker1114_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1104.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042023_cmjohnson_19530_an-worker1104_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['an-worker1102.eqiad.wmnet']

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1108.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042017_cmjohnson_12076_an-worker1108_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1109.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042017_cmjohnson_11996_an-worker1109_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1110.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042017_cmjohnson_11037_an-worker1110_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1111.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042017_cmjohnson_10966_an-worker1111_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1112.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042016_cmjohnson_9840_an-worker1112_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1113.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042017_cmjohnson_10472_an-worker1113_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['an-worker1104.eqiad.wmnet']

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1104.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009042003_cmjohnson_26831_an-worker1104_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['an-worker1104.eqiad.wmnet']

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1107.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009041947_cmjohnson_12164_an-worker1107_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1106.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009041946_cmjohnson_11244_an-worker1106_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1105.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009041945_cmjohnson_10537_an-worker1105_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1104.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009041945_cmjohnson_10008_an-worker1104_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1103.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009041944_cmjohnson_9313_an-worker1103_eqiad_wmnet.log.

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1102.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009041943_cmjohnson_8670_an-worker1102_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['an-worker1101.eqiad.wmnet']

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1101.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009032022_cmjohnson_31526_an-worker1101_eqiad_wmnet.log.

Completed auto-reimage of hosts:

['an-worker1101.eqiad.wmnet']

Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts:

an-worker1101.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/202009031959_cmjohnson_10210_an-worker1101_eqiad_wmnet.log.