Page MenuHomePhabricator
Create Task
Maniphest T380307

installation tracking for hosts affected by magru re-shuffle
Closed, ResolvedPublic
Actions

Description

Due to the issues noted on T376737, we're going to have to reimage a large number of ops-magru hosts. This task will have the per host checklist for each item and what needs to be checked for each to return to service.

Migration Step Checklist

Rob has compiled the following order of operation after day 1 server swaps. Completing these in any other order may not result in two hosts being ready for reimage:

  • Support Swaps each host pair around, Rob remotely confirms hardware is correct and remotely accessible.
  • Take screen shots / note the network port and cable ID info for both host A and B.
  • Run decommission cookbook for both hosts A and B.
  • Pull up both A and B in netbox, swap around A and B hostname + rack location, set both from Decommissioning to Planned state.
  • Pull up host A's mgmt ip and enter udpated fqdn under dns name for the mgmt ip entry.
  • Run Provision Host Network details for host A, use host B's netowkr port and cable info recorded earlier.
  • Pull up host B's mgmt ip and enter udpated fqdn under dns name for the mgmt ip entry.
  • Run Provision Host Network details for host B, use host B's netowkr port and cable info recorded earlier.
  • Run sre.dns.netbox to push updated dns for hosts A and B.
  • Run sre.network.configure-switch-interfaces for host A and host B

Per host setup checklist

dns7001
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • NIC doesn't detect, remote hands task updated with info, don't check this line off until NIC is reseated and detects - remote hands reseated nic and now it works and shows link to port 8 via idrac interface
  • Run the sre.hosts.reimage cookbook
dns7002
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
ganeti7001
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
  • create and link in service implementation task for Mortiz for ganeti700[1-4]
ganeti7002
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
  • create and link in service implementation task for Mortiz for ganeti700[1-4]
ganeti7003
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
ganeti7004
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
lvs7001
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook

lvs7001 network connections
nic port 1: 70101 asw1-b3-magru (WMF12033) et-0/0/11 (black DAC)
nic port 2: 70114 asw1-b4-magru (WMF12034) et-0/0/12 (yellow singlemode)

lvs7003
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook

lvs7003 network info:
nic port 0: cable 70100 : asw1-b3-magru:et-0/0/8
nic port 1: cable 70113 : asw2-b4-magru:et-0/0/13

cp7001
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
cp7002
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
cp7003
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
cp7004
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
cp7006
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
cp7008
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
cp7010
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook
cp7015
  • Ensure host is fully migrated per T376737
  • Update netbox with host's new location and service tag, copy down old hostname cable IDs, delete off old hostname, and install via script onto this host.
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
hiera: fix magru dns7001 ip address during migrationoperations/puppetproduction+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Restricted Task
 ResolvedssinghT380307 installation tracking for hosts affected by magru re-shuffle

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 19 2024, 6:05 PM
RobH added a parent task: Restricted Task.Nov 19 2024, 6:05 PM
RobH moved this task from Backlog to Racking Tasks on the ops-magru board.
RobH updated the task description. (Show Details)Nov 25 2024, 2:16 PM
ops-monitoring-bot added a comment.Nov 25 2024, 4:58 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: ganeti7003.magru.wmnet

  • ganeti7003.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Nov 25 2024, 5:10 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: cp7006.magru.wmnet

  • cp7006.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

RobH updated the task description. (Show Details)Nov 25 2024, 5:20 PM
RobH updated the task description. (Show Details)Nov 25 2024, 5:23 PM
ops-monitoring-bot added a comment.Nov 25 2024, 5:39 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: ganeti7004.magru.wmnet

  • ganeti7004.magru.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Nov 25 2024, 5:41 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: cp7008.magru.wmnet

  • cp7008.magru.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
RobH updated the task description. (Show Details)Nov 25 2024, 5:59 PM
ops-monitoring-bot added a comment.Nov 25 2024, 6:17 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: lvs7003.magru.wmnet

  • lvs7003.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Failed to wipe swraid, partition-table and filesystem signatures, manual intervention required to make it unbootable: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Nov 25 2024, 6:27 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: cp7015.magru.wmnet

  • cp7015.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

RobH updated the task description. (Show Details)Nov 25 2024, 6:43 PM
Fabfur updated the task description. (Show Details)Nov 25 2024, 6:49 PM
RobH updated the task description. (Show Details)Nov 25 2024, 6:54 PM
ops-monitoring-bot added a comment.Nov 25 2024, 6:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti7003.magru.wmnet with OS bookworm

BCornwall updated the task description. (Show Details)Nov 25 2024, 7:03 PM
BCornwall updated the task description. (Show Details)Nov 25 2024, 7:05 PM
ops-monitoring-bot added a comment.Nov 25 2024, 7:58 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti7003.magru.wmnet with OS bookworm completed:

  • ganeti7003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411251935_robh_3696544_ganeti7003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
RobH updated the task description. (Show Details)Nov 25 2024, 7:59 PM
ops-monitoring-bot added a comment.Nov 25 2024, 8:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti7004.magru.wmnet with OS bookworm

RobH updated the task description. (Show Details)Nov 25 2024, 8:15 PM
RobH updated the task description. (Show Details)Nov 25 2024, 8:38 PM
RobH updated the task description. (Show Details)Nov 25 2024, 8:51 PM
ops-monitoring-bot added a comment.Nov 26 2024, 12:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host cp7015.magru.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 26 2024, 1:04 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host cp7015.magru.wmnet with OS bullseye executed with errors:

  • cp7015 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp7015.magru.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 26 2024, 1:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host dns7001.wikimedia.org with OS bookworm

BCornwall updated the task description. (Show Details)Nov 26 2024, 2:31 AM
ops-monitoring-bot added a comment.Nov 26 2024, 12:30 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host cp7015.magru.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 26 2024, 12:48 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host cp7015.magru.wmnet with OS bullseye executed with errors:

  • cp7015 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console cp7015.magru.wmnet" to get a root shell, but depending on the failure this may not work.
RobH updated the task description. (Show Details)Nov 26 2024, 12:52 PM
RobH updated the task description. (Show Details)
ops-monitoring-bot added a comment.Nov 26 2024, 12:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host lvs7003.magru.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 26 2024, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1002 for host dns7001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Nov 26 2024, 1:07 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1002 for host dns7001.wikimedia.org with OS bookworm executed with errors:

  • dns7001 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dns7001.wikimedia.org" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Nov 26 2024, 1:11 PM

Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host cp7015.magru.wmnet with OS bullseye

ops-monitoring-bot added a comment.Nov 26 2024, 1:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host dns7001.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Nov 26 2024, 1:49 PM

Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host lvs7003.magru.wmnet with OS bullseye completed:

  • lvs7003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261320_fabfur_575669_lvs7003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Nov 26 2024, 2:01 PM

Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host cp7015.magru.wmnet with OS bullseye completed:

  • cp7015 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261335_fabfur_580242_cp7015.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Nov 26 2024, 2:43 PM

Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host dns7001.wikimedia.org with OS bullseye completed:

  • dns7001 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261409_fabfur_587599_dns7001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Nov 26 2024, 2:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host dns7001.wikimedia.org with OS bookworm

ops-monitoring-bot added a comment.Nov 26 2024, 3:42 PM

Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host dns7001.wikimedia.org with OS bookworm completed:

  • dns7001 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411261511_fabfur_601294_dns7001.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 26 2024, 7:23 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: ganeti7001.magru.wmnet

  • ganeti7001.magru.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Nov 26 2024, 7:43 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: cp7003.magru.wmnet

  • cp7003.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

RobH updated the task description. (Show Details)Nov 26 2024, 8:00 PM
RobH updated the task description. (Show Details)
ops-monitoring-bot added a comment.Nov 26 2024, 8:13 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: ganeti7002.magru.wmnet

  • ganeti7002.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Host steps raised exception: No non-mgmt connected interfaces found for ganeti7002. Please check Netbox.

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Nov 26 2024, 8:16 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: cp7004.magru.wmnet

  • cp7004.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Host steps raised exception: No non-mgmt connected interfaces found for cp7004. Please check Netbox.

ERROR: some step on some host failed, check the bolded items above

BCornwall updated the task description. (Show Details)Nov 26 2024, 8:25 PM
RobH updated the task description. (Show Details)Nov 26 2024, 8:27 PM
RobH updated the task description. (Show Details)Nov 26 2024, 8:34 PM
RobH updated the task description. (Show Details)
ops-monitoring-bot added a comment.Nov 26 2024, 8:47 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: dns7002.wikimedia.org

  • dns7002.wikimedia.org (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Nov 26 2024, 8:50 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: cp7002.magru.wmnet

  • cp7002.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

RobH updated the task description. (Show Details)Nov 26 2024, 9:03 PM
RobH updated the task description. (Show Details)Nov 26 2024, 9:05 PM
ops-monitoring-bot added a comment.Nov 26 2024, 9:20 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: cp7010.magru.wmnet

  • cp7010.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Nov 26 2024, 9:22 PM

cookbooks.sre.hosts.decommission executed by robh@cumin2002 for hosts: lvs7001.magru.wmnet

  • lvs7001.magru.wmnet (FAIL)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Alertmanager
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

RobH updated the task description. (Show Details)Nov 26 2024, 9:35 PM
BCornwall updated the task description. (Show Details)Nov 26 2024, 9:37 PM
RobH added a subscriber: MoritzMuehlenhoff.Nov 26 2024, 9:39 PM

@MoritzMuehlenhoff : ganeti700[12] are ready for reimage but I've just run out of steam for today. If you don't get to their reimage on Wednesday I'll do so on my Wednesday AM.

BCornwall updated the task description. (Show Details)Nov 26 2024, 9:59 PM
BCornwall updated the task description. (Show Details)Nov 26 2024, 10:38 PM
BCornwall updated the task description. (Show Details)Nov 26 2024, 11:36 PM
MoritzMuehlenhoff updated the task description. (Show Details)Nov 27 2024, 1:37 PM
gerritbot added a comment.Nov 27 2024, 4:13 PM

Change #1098554 had a related patch set uploaded (by Fabfur; author: Fabfur):

[operations/puppet@production] hiera: fix magru ip addresses during migration

https://gerrit.wikimedia.org/r/1098554

gerritbot added a project: Patch-For-Review.Nov 27 2024, 4:13 PM
Fabfur updated the task description. (Show Details)Nov 27 2024, 4:19 PM
gerritbot added a comment.Nov 27 2024, 4:25 PM

Change #1098554 merged by Fabfur:

[operations/puppet@production] hiera: fix magru dns7001 ip address during migration

https://gerrit.wikimedia.org/r/1098554

ops-monitoring-bot added a comment.Nov 27 2024, 4:26 PM

Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1002 for host cp7010.magru.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Nov 27 2024, 4:31 PM
ops-monitoring-bot added a comment.Nov 27 2024, 5:17 PM

Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1002 for host cp7010.magru.wmnet with OS bullseye completed:

  • cp7010 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411271651_fabfur_818716_cp7010.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
Fabfur subscribed.Nov 27 2024, 5:51 PM

Reverted https://gerrit.wikimedia.org/r/c/operations/puppet/+/1098573 and ran puppet agent on A:cp-magru: NOOP as expected

Fabfur added a comment.Nov 27 2024, 5:51 PM

lvs7003 has been restarted after cable swap, all fine

Fabfur added a comment.EditedNov 27 2024, 6:06 PM

BGP flag enabled on NetBox for lvs700[1-3] and dns700[12] and BGP enabled on TOR switches for those hosts

Stashbot added a comment.Nov 27 2024, 6:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-27T18:37:20Z] <fabfur@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on dns7001.wikimedia.org with reason: T380307

Stashbot added a comment.Nov 27 2024, 6:37 PM

Mentioned in SAL (#wikimedia-operations) [2024-11-27T18:37:24Z] <fabfur@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dns7001.wikimedia.org with reason: T380307

Fabfur added a comment.Nov 27 2024, 6:39 PM

Removed downtime from all lvs, dns and cp hosts in magru

Fabfur added a comment.Nov 27 2024, 6:49 PM

Repooled dnsbox cluster and run authdns-update

Fabfur added a comment.Nov 27 2024, 7:09 PM

ran puppet-agent on A:magru

Fabfur added a comment.Nov 27 2024, 7:12 PM

Repooled all depooled cp hosts before repooling whole DC

Fabfur added a comment.Nov 27 2024, 7:21 PM

Repooled magru DC

ops-monitoring-bot added a comment.Dec 3 2024, 10:16 AM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti7004.magru.wmnet with OS bookworm completed:

  • ganeti7004 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202411252034_robh_3708720_ganeti7004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Volans subscribed.Dec 3 2024, 10:25 AM

FYI I have aborted the last reimage execution that was at the last step waiting for use input for the netbox-hiera integration sync. Those changes have showed up to others and got already merged anyway. In general please don't leave cookbooks hanging for user input for days.

RobH added a comment.Dec 3 2024, 1:39 PM
In T380307#10375032, @Volans wrote:

FYI I have aborted the last reimage execution that was at the last step waiting for use input for the netbox-hiera integration sync. Those changes have showed up to others and got already merged anyway. In general please don't leave cookbooks hanging for user input for days.

Accidental, sorry about that! I had too many screen sessions and seemed to have left one abandoned!

ssingh closed this task as Resolved.Dec 4 2024, 2:31 PM
ssingh claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits