Page MenuHomePhabricator
Create Task
Maniphest T344355

Replacement of esams VMs in knams Ganeti clusters
Closed, ResolvedPublic
Actions

Description

Tracking task for decom and recreation of ganeti VMs in esams

Old VM nameNew VM namenew cluster (esams01 or esams02)old VM decommed
bast3006.wikimedia.orgbast3007.wikimedia.orgesams01Done
doh3001.wikimedia.orgdoh3003esams01Done
doh3002.wikimedia.orgdoh3004esams02Done
durum3001.esams.wmnetdurum3003.esams.wmnetesams01Done
durum3002.esams.wmnetdurum3004.esams.wmnetesams02Done
install3002.wikimedia.orginstall3003.wikimedia.orgesams02Done
ncredir3001.esams.wmnetncredir3003.esams.wmnetesams01Done
ncredir3002.esams.wmnetncredir3004.esams.wmnetesams02Done
netflow3002.esams.wmnetnetflow3002.esams.wmnetesams02Done
ping3003.esams.wmnetNot needed for nown/aDone
prometheus3002.esams.wmnetprometheus3003.esams.wmnetesams02Done

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

MoritzMuehlenhoff created this task.Aug 16 2023, 2:49 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptAug 16 2023, 2:49 PM
ayounsi updated the task description. (Show Details)Aug 16 2023, 2:51 PM
gerritbot added a comment.Aug 16 2023, 3:02 PM

Change 949537 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add netflow3003 to Ferm rules for Kafka jumbo

https://gerrit.wikimedia.org/r/949537

gerritbot added a project: Patch-For-Review.Aug 16 2023, 3:02 PM
gerritbot added a comment.Aug 16 2023, 3:05 PM

Change 949537 abandoned by Muehlenhoff:

[operations/puppet@production] Add netflow3003 to Ferm rules for Kafka jumbo

Reason:

Abandoned in favour of 949534

https://gerrit.wikimedia.org/r/949537

ops-monitoring-bot added a comment.Aug 16 2023, 3:09 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: bast3006.wikimedia.org

  • bast3006.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
gerritbot added a comment.Aug 16 2023, 3:13 PM

Change 949541 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove bast3006/ping3003 from site.pp

https://gerrit.wikimedia.org/r/949541

MoritzMuehlenhoff updated the task description. (Show Details)Aug 16 2023, 3:18 PM
gerritbot added a comment.Aug 16 2023, 3:19 PM

Change 949541 merged by Muehlenhoff:

[operations/puppet@production] Remove bast3006/ping3003 from site.pp

https://gerrit.wikimedia.org/r/949541

ops-monitoring-bot added a comment.Aug 16 2023, 3:20 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ping3003.esams.wmnet

  • ping3003.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
MoritzMuehlenhoff updated the task description. (Show Details)Aug 16 2023, 3:21 PM
gerritbot added a comment.Aug 16 2023, 3:25 PM

Change 949543 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] New install server for new esams

https://gerrit.wikimedia.org/r/949543

gerritbot added a comment.Aug 16 2023, 3:27 PM

Change 949543 merged by Muehlenhoff:

[operations/puppet@production] New install server for new esams

https://gerrit.wikimedia.org/r/949543

ops-monitoring-bot added a comment.Aug 16 2023, 3:46 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: netflow3002.esams.wmnet

  • netflow3002.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
MoritzMuehlenhoff updated the task description. (Show Details)Aug 16 2023, 3:47 PM
gerritbot added a comment.Aug 16 2023, 3:57 PM

Change 949552 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make install3003 the new install server for esams

https://gerrit.wikimedia.org/r/949552

MoritzMuehlenhoff updated the task description. (Show Details)Aug 16 2023, 4:06 PM
ops-monitoring-bot added a comment.Aug 16 2023, 4:22 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: doh[3001-3002].wikimedia.org

  • doh3001.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
  • doh3002.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
ops-monitoring-bot added a comment.Aug 16 2023, 4:31 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: durum[3001-3002].esams.wmnet

  • durum3001.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
  • durum3002.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
ssingh updated the task description. (Show Details)Aug 16 2023, 4:37 PM
gerritbot added a comment.Aug 16 2023, 4:54 PM

Change 949558 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] ncredir300x: decommission hosts in esams

https://gerrit.wikimedia.org/r/949558

ops-monitoring-bot added a comment.Aug 16 2023, 6:10 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: lvs[3005-3007].esams.wmnet

  • lvs3005.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • lvs3006.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • lvs3007.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Aug 17 2023, 6:39 AM

Change 949552 merged by Muehlenhoff:

[operations/puppet@production] Make install3003 the new install server for esams

https://gerrit.wikimedia.org/r/949552

gerritbot added a comment.Aug 17 2023, 7:05 AM

Change 949628 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Point the esams webproxy to install3003

https://gerrit.wikimedia.org/r/949628

gerritbot added a comment.Aug 17 2023, 7:07 AM

Change 949628 merged by Muehlenhoff:

[operations/dns@master] Point the esams webproxy to install3003

https://gerrit.wikimedia.org/r/949628

ops-monitoring-bot added a comment.Aug 17 2023, 7:12 AM

Icinga downtime and Alertmanager silence (ID=756bda9d-0fe5-407f-8e34-35d788d9ab8c) set by jmm@cumin2002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: decom in progress

install3002.wikimedia.org
ops-monitoring-bot added a comment.Aug 17 2023, 7:48 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: install3002.wikimedia.org

  • install3002.wikimedia.org (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • Failed to shutdown VM, manually run gnt-instance remove on the Ganeti master for the esams cluster: Cumin execution failed (exit_code=2)
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • Failed to remove VM, manually run gnt-instance remove on the Ganeti master for the esams cluster: Cumin execution failed (exit_code=2)
    • Started forced sync of VMs in Ganeti cluster esams to Netbox

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Aug 17 2023, 8:05 AM

Change 949558 merged by Muehlenhoff:

[operations/puppet@production] ncredir300x: decommission hosts in esams

https://gerrit.wikimedia.org/r/949558

ops-monitoring-bot added a comment.Aug 17 2023, 8:16 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ncredir3002.esams.wmnet

  • ncredir3002.esams.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • Failed to remove VM, manually run gnt-instance remove on the Ganeti master for the esams cluster: Cumin execution failed (exit_code=2)
    • Started forced sync of VMs in Ganeti cluster esams to Netbox

ERROR: some step on some host failed, check the bolded items above

MoritzMuehlenhoff updated the task description. (Show Details)Aug 17 2023, 8:18 AM
ops-monitoring-bot added a comment.Aug 17 2023, 8:25 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ncredir3001.esams.wmnet

  • ncredir3001.esams.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
ops-monitoring-bot added a comment.Aug 17 2023, 8:29 AM

cookbooks.sre.hosts.decommission executed by filippo@cumin1001 for hosts: prometheus3002.esams.wmnet

  • prometheus3002.esams.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • Failed to shutdown VM, manually run gnt-instance remove on the Ganeti master for the esams cluster: Cumin execution failed (exit_code=2)
    • Started forced sync of VMs in Ganeti cluster esams to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • Failed to remove VM, manually run gnt-instance remove on the Ganeti master for the esams cluster: Cumin execution failed (exit_code=2)
    • Started forced sync of VMs in Ganeti cluster esams to Netbox

ERROR: some step on some host failed, check the bolded items above

fgiunchedi updated the task description. (Show Details)Aug 17 2023, 8:32 AM
gerritbot added a comment.Aug 17 2023, 8:43 AM

Change 949837 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] Out with prometheus3002, in with prometheus3003

https://gerrit.wikimedia.org/r/949837

MoritzMuehlenhoff updated the task description. (Show Details)Aug 17 2023, 8:45 AM
ops-monitoring-bot added a comment.Aug 17 2023, 8:45 AM

Cookbook cookbooks.sre.hosts.reimage was started by filippo@cumin1001 for host prometheus3003.esams.wmnet with OS bullseye

gerritbot added a comment.Aug 17 2023, 8:47 AM

Change 949838 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/dns@master] wmnet: use prometheus3003 in esams

https://gerrit.wikimedia.org/r/949838

gerritbot added a comment.Aug 17 2023, 9:02 AM

Change 949837 merged by Filippo Giunchedi:

[operations/puppet@production] Out with prometheus3002, in with prometheus3003

https://gerrit.wikimedia.org/r/949837

gerritbot added a comment.Aug 17 2023, 9:05 AM

Change 949838 merged by Filippo Giunchedi:

[operations/dns@master] wmnet: use prometheus3003 in esams

https://gerrit.wikimedia.org/r/949838

ops-monitoring-bot added a comment.Aug 17 2023, 9:36 AM

Cookbook cookbooks.sre.hosts.reimage started by filippo@cumin1001 for host prometheus3003.esams.wmnet with OS bullseye completed:

  • prometheus3003 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed and logged in /var/log/spicerack/sre/hosts/reimage/202308170922_filippo_3583585_prometheus3003.out, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308170927_filippo_3583585_prometheus3003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Aug 17 2023, 11:02 AM

Change 949941 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add ncredir300[34]

https://gerrit.wikimedia.org/r/949941

gerritbot added a comment.Aug 17 2023, 1:01 PM

Change 949941 merged by Muehlenhoff:

[operations/puppet@production] Add ncredir300[34]

https://gerrit.wikimedia.org/r/949941

MoritzMuehlenhoff updated the task description. (Show Details)Aug 17 2023, 1:47 PM
gerritbot added a comment.Aug 17 2023, 1:53 PM

Change 949977 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add durum300[34] to site.pp

https://gerrit.wikimedia.org/r/949977

gerritbot added a comment.Aug 17 2023, 2:01 PM

Change 949977 merged by Muehlenhoff:

[operations/puppet@production] Add durum300[34] to site.pp

https://gerrit.wikimedia.org/r/949977

MoritzMuehlenhoff updated the task description. (Show Details)Aug 17 2023, 2:47 PM
gerritbot added a comment.Aug 17 2023, 2:50 PM

Change 949987 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add doh300[34]

https://gerrit.wikimedia.org/r/949987

gerritbot added a comment.Aug 17 2023, 3:01 PM

Change 949987 merged by Muehlenhoff:

[operations/puppet@production] Add doh300[34]

https://gerrit.wikimedia.org/r/949987

RobH unsubscribed.Aug 17 2023, 3:07 PM
MoritzMuehlenhoff updated the task description. (Show Details)Aug 17 2023, 3:36 PM
MoritzMuehlenhoff updated the task description. (Show Details)
gerritbot added a comment.Aug 17 2023, 5:49 PM

Change 950000 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: reimage ncredir300[34] to proper role

https://gerrit.wikimedia.org/r/950000

gerritbot added a comment.Aug 17 2023, 5:56 PM

Change 950000 merged by Ssingh:

[operations/puppet@production] site: reimage ncredir300[34] to proper role

https://gerrit.wikimedia.org/r/950000

gerritbot added a comment.Aug 17 2023, 6:56 PM

Change 950026 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] conf-tool/esams: add ncredir300[34]

https://gerrit.wikimedia.org/r/950026

gerritbot added a comment.Aug 17 2023, 8:38 PM

Change 950026 merged by Ssingh:

[operations/puppet@production] conf-tool/esams: add ncredir300[34]

https://gerrit.wikimedia.org/r/950026

gerritbot added a comment.Aug 18 2023, 11:00 AM

Change 950159 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make bast3007 a bastion

https://gerrit.wikimedia.org/r/950159

MoritzMuehlenhoff updated the task description. (Show Details)Aug 18 2023, 11:00 AM
gerritbot added a comment.Aug 18 2023, 11:29 AM

Change 950159 merged by Muehlenhoff:

[operations/puppet@production] Make bast3007 a bastion

https://gerrit.wikimedia.org/r/950159

gerritbot added a comment.Aug 22 2023, 3:30 PM

Change 951528 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: add wikidough VMs for esams

https://gerrit.wikimedia.org/r/951528

gerritbot added a comment.Aug 22 2023, 3:30 PM

Change 951528 merged by Ssingh:

[operations/puppet@production] site: add wikidough VMs for esams

https://gerrit.wikimedia.org/r/951528

gerritbot added a comment.Aug 22 2023, 3:41 PM

Change 951531 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site: remove older references to doh300[34]

https://gerrit.wikimedia.org/r/951531

gerritbot added a comment.Aug 22 2023, 3:42 PM

Change 951531 merged by Ssingh:

[operations/puppet@production] site: remove older references to doh300[34]

https://gerrit.wikimedia.org/r/951531

gerritbot added a comment.Aug 22 2023, 3:47 PM

Change 951532 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: update authorized_hosts for acme_chief for WDNS

https://gerrit.wikimedia.org/r/951532

Stashbot added a comment.Aug 22 2023, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2023-08-22T15:58:29Z] <sukhe> sudo cookbook sre.ganeti.makevm --vcpus 2 --memory 8 --disk 15 --network public --os bullseye --cluster esams01 --group BY27 -t T344355 doh3003

gerritbot added a comment.Aug 22 2023, 4:03 PM

Change 951532 merged by Ssingh:

[operations/puppet@production] hiera: update authorized_hosts for acme_chief for WDNS

https://gerrit.wikimedia.org/r/951532

ops-monitoring-bot added a comment.Aug 22 2023, 5:13 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: doh3003.wikimedia.org

  • doh3003.wikimedia.org (WARN)
    • Host not found on Icinga, unable to downtime it
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams01 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams01 to Netbox
ops-monitoring-bot added a comment.Aug 22 2023, 5:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host doh3003.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2023, 5:56 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host doh3003.wikimedia.org with OS bullseye completed:

  • doh3003 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308221742_sukhe_584678_doh3003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Aug 22 2023, 6:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host doh3004.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2023, 6:37 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host doh3004.wikimedia.org with OS bullseye completed:

  • doh3004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308221821_sukhe_632198_doh3004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Aug 22 2023, 6:48 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: doh3004.wikimedia.org

  • doh3004.wikimedia.org (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster esams01 to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster esams01 to Netbox
ops-monitoring-bot added a comment.Aug 22 2023, 6:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host doh3004.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Aug 22 2023, 7:32 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host doh3004.wikimedia.org with OS bullseye completed:

  • doh3004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via gnt-instance
    • Host up (Debian installer)
    • Set boot media to disk
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202308221916_sukhe_688765_doh3004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
gerritbot added a comment.Aug 22 2023, 9:07 PM

Change 951581 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] devices: add doh300[34] to asw1-b*27-esams

https://gerrit.wikimedia.org/r/951581

ssingh updated the task description. (Show Details)Aug 22 2023, 9:07 PM
MoritzMuehlenhoff closed this task as Resolved.Aug 23 2023, 6:54 AM

This is complete

gerritbot added a comment.Aug 24 2023, 12:57 PM

Change 951581 merged by Ssingh:

[operations/homer/public@master] devices: add doh300[34] to asw1-b*27-esams

https://gerrit.wikimedia.org/r/951581

fgiunchedi mentioned this in T345743: reprovision ping VM in esams.Sep 6 2023, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits