Page MenuHomePhabricator
Create Task
Maniphest T317247

Q1:rack/setup/install ulsfo misc class hosts
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of the 9 misc class hosts ordered for ulsfo refresh.

Hostname / Racking / Installation Details

Hostnames: lvs40(0[89]|10), ganeti400[5678], dns400[34]
Racking Proposal: odd numbered hostnames in rack 22, even numbered hostnames in rack 23
Networking Setup: # of Connections:1 for everything except LVS which need 2, Speed:10G. Vlan:Match existing hosts AAAA records:Y, Additional IP records (Cassandra)?
Partitioning/Raid: HW Raid: N, Partman recipe and/or desired Raid Level:
OS Distro: Bullseye
Sub-team Technical Contact:@BBlack

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

lvs4008:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - racked with power and mgmt, idrac not yet setup, allowing for automation to troubleshoot the auto provisioning script in ulsfo
  • - LVS hosts must be wired in to both their own ASW and the adjacent rack ASW
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller) - downgraded nic from 22. to 21.85.21.92
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
lvs4009:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller) NIC downgraded from 22.00.07.60 to 21.85.21.92
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
lvs4010:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller) NIC downgraded from 22.00.07.60 to 21.85.21.92
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ganeti4005:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps

& Automatic BIOS setup details - fails the provision script for redfish connection, idrac manually setup for remote connectivity so @Volans can troubleshoot whats up

  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ganeti4006:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ganeti4007:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller) ganeti4007 (NETWORK): now at version: 21.85.21.92
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ganeti4008:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
dns4003:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller) - bios newest, idrac capped, network rolled back to working version 21.85 from 22.x
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
dns4004:
  • - receive in system on procurement task T311749 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
site.pp: update LVS hosts in ulsfooperations/puppetproduction+1 -1
Make ganeti4007 a Ganeti nodeoperations/puppetproduction+2 -5
ganeti4007 insetup roleoperations/puppetproduction+4 -0
hiera: unify ulsfo LVS configurationoperations/puppetproduction+1 -11
lvs4010: set as secondary LVS and remove lvs4007 (decom)operations/puppetproduction+4 -6
sites.yaml: remove decommissioned host lvs4007operations/homer/publicmaster+0 -1
sites.yaml: add lvs4010 (ulsfo hardware refresh)operations/homer/publicmaster+1 -0
lvs4010: commission new LVS host (ulsfo hardware refresh)operations/puppetproduction+8 -3
sites.yaml: remove decommissioned host lvs4006operations/homer/publicmaster+0 -1
lvs4009: set as high-traffic2 primary LVS and remove lvs4006 (decomm)operations/puppetproduction+1 -6
lvs4006: set profile::pybal::bgp to nooperations/puppetproduction+1 -0
sites.yaml: add lvs4009 (ulsfo hardware refresh)operations/homer/publicmaster+2 -1
lvs4009: commission new LVS host (ulsfo hardware refresh)operations/puppetproduction+9 -6
sites.yaml: remove decommissioned host lvs4005operations/homer/publicmaster+0 -1
lvs4008: commission new LVS host (ulsfo hardware refresh)operations/puppetproduction+3 -1
sites.yaml: add lvs4008 (ulsfo hardware refresh)operations/homer/publicmaster+1 -0
Depool ulsfo for cp hosts hardware refreshoperations/dnsmaster+2 -0
Remove ganeti4003 from Puppetoperations/puppetproduction+1 -2
Make ganeti4006 a Ganeti nodeoperations/puppetproduction+1 -0
ganeti4006operations/puppetproduction+1 -1
Remove ganeti4002 from Puppet for decomoperations/puppetproduction+1 -2
Swap ganeti4002/ganeti4003 for blackbox smoke testsoperations/puppetproduction+2 -2
Make ganeti4005 a Ganeti nodeoperations/puppetproduction+2 -5
ulsfo ganeti4005 lvs4008operations/puppetproduction+8 -0
sites.yaml: add dns4004 to anycast_neighborsoperations/homer/publicmaster+1 -0
dns4004: add Puppet role and DNS/NTP configsoperations/puppetproduction+9 -5
Make ganeti4008 a Ganeti nodeoperations/puppetproduction+2 -5
dns4004operations/puppetproduction+4 -0
ganeti4008 setupoperations/puppetproduction+4 -0
wikimedia.org: update CNAME for ntp.ulsfo to dns4003operations/dnsmaster+1 -1
sites.yaml: add dns4003 to anycast_neighborsoperations/homer/publicmaster+1 -0
hiera: rename dns4001.yaml (decomm) to dns4003.yaml (active)operations/puppetproduction+0 -0
dns4003: add Puppet role and DNS/NTP configsoperations/puppetproduction+5 -5
adding new dns4003operations/puppetproduction+4 -1
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedRobHT317247 Q1:rack/setup/install ulsfo misc class hosts
 ResolvedRequestssinghT320440 decommission dns4002

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Papaul added a comment.Oct 20 2022, 10:09 PM

So the issue with ganeti4005 was that the bios boot setting was set to UEFI that is the reason RedFish was failing so after I changed it to BIOS i had no issues

END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti4005.mgmt.ulsfo.wmnet with reboot policy FORCED
pt1979@cumin2002:~$
gerritbot added a comment.Oct 20 2022, 11:21 PM

Change 845070 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] ulsfo ganeti4005 lvs4008

https://gerrit.wikimedia.org/r/845070

gerritbot added a comment.Oct 20 2022, 11:22 PM

Change 845070 merged by RobH:

[operations/puppet@production] ulsfo ganeti4005 lvs4008

https://gerrit.wikimedia.org/r/845070

RobH updated the task description. (Show Details)Oct 20 2022, 11:29 PM
ops-monitoring-bot added a comment.Oct 20 2022, 11:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti4005.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 20 2022, 11:38 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti4005.ulsfo.wmnet with OS bullseye executed with errors:

  • ganeti4005 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 20 2022, 11:39 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti4005.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 21 2022, 12:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host lvs4008.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 21 2022, 12:16 AM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti4005.ulsfo.wmnet with OS bullseye completed:

  • ganeti4005 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210202338_robh_1954717_ganeti4005.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Oct 21 2022, 12:47 AM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host lvs4008.ulsfo.wmnet with OS bullseye completed:

  • lvs4008 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210210007_robh_1959939_lvs4008.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
RobH updated the task description. (Show Details)Oct 21 2022, 12:59 AM
gerritbot added a comment.Oct 25 2022, 6:42 AM

Change 849010 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make ganeti4005 a Ganeti node

https://gerrit.wikimedia.org/r/849010

gerritbot added a comment.Oct 25 2022, 6:51 AM

Change 849010 merged by Muehlenhoff:

[operations/puppet@production] Make ganeti4005 a Ganeti node

https://gerrit.wikimedia.org/r/849010

gerritbot added a comment.Oct 25 2022, 8:55 AM

Change 849023 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Swap ganeti4003 with ganeti4005 for blackbox smoke tests

https://gerrit.wikimedia.org/r/849023

gerritbot added a comment.Oct 25 2022, 9:29 AM

Change 849023 merged by Muehlenhoff:

[operations/puppet@production] Swap ganeti4002/ganeti4003 for blackbox smoke tests

https://gerrit.wikimedia.org/r/849023

Stashbot added a comment.Oct 25 2022, 9:36 AM

Mentioned in SAL (#wikimedia-operations) [2022-10-25T09:36:43Z] <moritzm> drain ganeti4002 for eventual decom T317247

gerritbot added a comment.Oct 25 2022, 11:46 AM

Change 849054 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti4002 from Puppet for decom

https://gerrit.wikimedia.org/r/849054

gerritbot added a comment.Oct 25 2022, 11:49 AM

Change 849054 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti4002 from Puppet for decom

https://gerrit.wikimedia.org/r/849054

ops-monitoring-bot added a comment.Oct 25 2022, 12:09 PM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti4002.ulsfo.wmnet

  • ganeti4002.ulsfo.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Icinga/Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
• MoritzMuehlenhoff added a comment.Oct 25 2022, 12:16 PM

I have setup ganeti4005 as a node in the ulsfo Ganeti cluster and moved a VM to it to confirm it works as expected.

@RobH : I've also decomissioned ganeti4002, you can unrack it the next time you're going to ulsfo.

gerritbot added a comment.Oct 25 2022, 2:16 PM

Change 849105 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] Depool ulsfo for cp hosts hardware refresh

https://gerrit.wikimedia.org/r/849105

gerritbot added a comment.Oct 25 2022, 2:53 PM

Change 849105 merged by Ssingh:

[operations/dns@master] Depool ulsfo for cp hosts hardware refresh

https://gerrit.wikimedia.org/r/849105

RobH updated the task description. (Show Details)Oct 25 2022, 10:25 PM
Stashbot added a comment.Oct 27 2022, 9:17 AM

Mentioned in SAL (#wikimedia-operations) [2022-10-27T09:17:39Z] <moritzm> failover ganeti master in ulsfo to ganeti4008, unblocking future decom of ganeti4003 T317247

RobH updated the task description. (Show Details)Oct 27 2022, 5:57 PM
gerritbot added a comment.Oct 27 2022, 7:30 PM

Change 850260 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] ganeti4006

https://gerrit.wikimedia.org/r/850260

RobH updated the task description. (Show Details)Oct 27 2022, 7:31 PM
gerritbot added a comment.Oct 27 2022, 7:31 PM

Change 850260 merged by RobH:

[operations/puppet@production] ganeti4006

https://gerrit.wikimedia.org/r/850260

ops-monitoring-bot added a comment.Oct 27 2022, 8:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti4006.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 27 2022, 8:35 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti4006.ulsfo.wmnet with OS buster executed with errors:

  • ganeti4006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 27 2022, 8:36 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti4006.ulsfo.wmnet with OS bullseye

ops-monitoring-bot added a comment.Oct 27 2022, 9:13 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti4006.ulsfo.wmnet with OS bullseye completed:

  • ganeti4006 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210272035_robh_3631110_ganeti4006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
RobH updated the task description. (Show Details)Oct 27 2022, 9:14 PM
gerritbot added a comment.Oct 28 2022, 8:42 AM

Change 850413 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make ganeti4006 a Ganeti node

https://gerrit.wikimedia.org/r/850413

gerritbot added a comment.Oct 28 2022, 8:44 AM

Change 850413 merged by Muehlenhoff:

[operations/puppet@production] Make ganeti4006 a Ganeti node

https://gerrit.wikimedia.org/r/850413

Stashbot added a comment.Oct 28 2022, 9:53 AM

Mentioned in SAL (#wikimedia-operations) [2022-10-28T09:53:36Z] <moritzm> drain ganeti4003 for eventual decom T317247

gerritbot added a comment.Oct 28 2022, 11:00 AM

Change 850451 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Remove ganeti4003 from Puppet

https://gerrit.wikimedia.org/r/850451

gerritbot added a comment.Oct 28 2022, 11:04 AM

Change 850451 merged by Muehlenhoff:

[operations/puppet@production] Remove ganeti4003 from Puppet

https://gerrit.wikimedia.org/r/850451

ops-monitoring-bot added a comment.Oct 28 2022, 11:26 AM

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: ganeti4003.ulsfo.wmnet

  • ganeti4003.ulsfo.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Downtimed management interface on Icinga/Alertmanager
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ops-monitoring-bot added a comment.Nov 7 2022, 12:05 PM

cookbooks.sre.hosts.decommission executed by volans@cumin1001 for hosts: ganeti4003.ulsfo.wmnet

  • ganeti4003.ulsfo.wmnet (FAIL)
    • Unable to find/resolve the mgmt DNS record, using the IP instead: 10.128.129.19
    • Host not found on Icinga, unable to downtime it
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Unable to connect to the host, wipe of swraid, partition-table and filesystem signatures will not be performed: Cumin execution failed (exit_code=2)
    • Failed to power off, manual intervention required: Remote IPMI for 10.128.129.19 failed (exit=1): b''
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Nov 10 2022, 2:55 PM

Change 855583 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: add lvs4008, replacing lvs4005 (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/855583

gerritbot added a comment.Nov 10 2022, 3:13 PM

Change 855607 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs4008: commission new LVS host (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/855607

gerritbot added a comment.Nov 10 2022, 3:16 PM

Change 855583 merged by Ssingh:

[operations/homer/public@master] sites.yaml: add lvs4008 (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/855583

gerritbot added a comment.Nov 10 2022, 3:31 PM

Change 855607 merged by Ssingh:

[operations/puppet@production] lvs4008: commission new LVS host (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/855607

RobH mentioned this in T322861: ulsfo next visit checklist.Nov 10 2022, 3:58 PM
ops-monitoring-bot added a comment.Nov 14 2022, 5:09 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: lvs4005.ulsfo.wmnet

  • lvs4005.ulsfo.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
ayounsi added a comment.Nov 15 2022, 9:07 AM

@ssingh
We have those 2 active alerts: https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=cr3-ulsfo&service=BGP+status https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=cr4-ulsfo&service=BGP+status
I guess https://github.com/wikimedia/operations-homer-public/blob/master/config/sites.yaml#L238 needs to be removed now that lvs4008 is online.

gerritbot added a comment.Nov 15 2022, 12:18 PM

Change 856946 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs4005

https://gerrit.wikimedia.org/r/856946

gerritbot added a comment.Nov 15 2022, 1:20 PM

Change 856946 merged by Ssingh:

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs4005

https://gerrit.wikimedia.org/r/856946

gerritbot added a comment.Nov 17 2022, 1:15 PM

Change 858336 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs4009: commission new LVS host (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/858336

gerritbot added a comment.Nov 21 2022, 2:52 PM

Change 859065 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: add lvs4009 (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/859065

gerritbot added a comment.Nov 21 2022, 4:08 PM

Change 859086 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs4006: set profile::pybal::bgp to no

https://gerrit.wikimedia.org/r/859086

RobH updated the task description. (Show Details)Nov 21 2022, 5:22 PM
RobH updated the task description. (Show Details)Nov 21 2022, 5:30 PM
gerritbot added a comment.Nov 22 2022, 2:39 PM

Change 858336 merged by Ssingh:

[operations/puppet@production] lvs4009: commission new LVS host (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/858336

gerritbot added a comment.Nov 22 2022, 5:21 PM

Change 859065 merged by Ssingh:

[operations/homer/public@master] sites.yaml: add lvs4009 (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/859065

Stashbot added a comment.Nov 22 2022, 6:46 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-22T18:46:29Z] <sukhe> cr[34]-ulsfo: set routing-options static route 198.35.26.112/28 next-hop 10.128.0.9: T317247

Stashbot added a comment.Nov 22 2022, 6:48 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-22T18:48:27Z] <sukhe> decommissioning lvs4006: T317247

gerritbot added a comment.Nov 22 2022, 6:48 PM

Change 859086 merged by Ssingh:

[operations/puppet@production] lvs4006: set profile::pybal::bgp to no

https://gerrit.wikimedia.org/r/859086

gerritbot added a comment.Nov 22 2022, 6:58 PM

Change 859598 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs4009: set as high-traffic2 primary LVS and remove lvs4006 (decomm)

https://gerrit.wikimedia.org/r/859598

gerritbot added a comment.Nov 22 2022, 7:00 PM

Change 859600 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs4006

https://gerrit.wikimedia.org/r/859600

ops-monitoring-bot added a comment.Nov 22 2022, 7:19 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: lvs4006.ulsfo.wmnet

  • lvs4006.ulsfo.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Nov 22 2022, 7:21 PM

Change 859598 merged by Ssingh:

[operations/puppet@production] lvs4009: set as high-traffic2 primary LVS and remove lvs4006 (decomm)

https://gerrit.wikimedia.org/r/859598

gerritbot added a comment.Nov 22 2022, 7:24 PM

Change 859600 merged by jenkins-bot:

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs4006

https://gerrit.wikimedia.org/r/859600

gerritbot added a comment.Nov 23 2022, 3:35 PM

Change 860067 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs4010: commission new LVS host (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/860067

gerritbot added a comment.Nov 23 2022, 6:43 PM

Change 860089 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: add lvs4010 (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/860089

gerritbot added a comment.Nov 23 2022, 6:44 PM

Change 860067 merged by Ssingh:

[operations/puppet@production] lvs4010: commission new LVS host (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/860067

ops-monitoring-bot added a comment.Nov 23 2022, 6:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs4010.ulsfo.wmnet with OS buster

gerritbot added a comment.Nov 23 2022, 7:03 PM

Change 860094 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] lvs4010: set as secondary LVS and remove lvs4007 (decom)

https://gerrit.wikimedia.org/r/860094

ops-monitoring-bot added a comment.Nov 23 2022, 7:26 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs4010.ulsfo.wmnet with OS buster completed:

  • lvs4010 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202211231845_sukhe_397048_lvs4010.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
gerritbot added a comment.Nov 23 2022, 7:35 PM

Change 860089 merged by Ssingh:

[operations/homer/public@master] sites.yaml: add lvs4010 (ulsfo hardware refresh)

https://gerrit.wikimedia.org/r/860089

Stashbot added a comment.Nov 23 2022, 7:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-11-23T19:41:08Z] <sukhe> decommission lvs4007: T317247

gerritbot added a comment.Nov 23 2022, 7:46 PM

Change 860103 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs4007

https://gerrit.wikimedia.org/r/860103

ops-monitoring-bot added a comment.Nov 23 2022, 7:59 PM

cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: lvs4007.ulsfo.wmnet

  • lvs4007.ulsfo.wmnet (WARN)
    • Downtimed host on Icinga/Alertmanager
    • Found physical host
    • Management interface not found on Icinga, unable to downtime it
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • [Netbox] Set status to Decommissioning, deleted all non-mgmt IPs, updated switch interfaces (disabled, removed vlans, etc)
    • Configured the linked switch interface(s)
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
gerritbot added a comment.Nov 23 2022, 8:00 PM

Change 860103 merged by Ssingh:

[operations/homer/public@master] sites.yaml: remove decommissioned host lvs4007

https://gerrit.wikimedia.org/r/860103

gerritbot added a comment.Nov 23 2022, 8:05 PM

Change 860094 merged by Ssingh:

[operations/puppet@production] lvs4010: set as secondary LVS and remove lvs4007 (decom)

https://gerrit.wikimedia.org/r/860094

ssingh updated the task description. (Show Details)Nov 23 2022, 8:06 PM
gerritbot added a comment.Nov 25 2022, 6:06 PM

Change 860930 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: unify ulsfo LVS configuration

https://gerrit.wikimedia.org/r/860930

gerritbot added a comment.Nov 28 2022, 2:33 PM

Change 860930 merged by Ssingh:

[operations/puppet@production] hiera: unify ulsfo LVS configuration

https://gerrit.wikimedia.org/r/860930

gerritbot added a comment.Dec 8 2022, 4:39 PM

Change 866441 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] site.pp: update LVS hosts in ulsfo

https://gerrit.wikimedia.org/r/866441

RobH updated the task description. (Show Details)Dec 16 2022, 5:52 PM
RobH removed a project: Patch-For-Review.Dec 16 2022, 6:09 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Dec 16 2022, 6:17 PM

Change 868733 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] ganeti4007 insetup role

https://gerrit.wikimedia.org/r/868733

gerritbot added a project: Patch-For-Review.Dec 16 2022, 6:17 PM

Change 868733 merged by RobH:

[operations/puppet@production] ganeti4007 insetup role

https://gerrit.wikimedia.org/r/868733

ops-monitoring-bot added a comment.Dec 16 2022, 6:19 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti4007.ulsfo.wmnet with OS bullseye

RobH removed a project: Patch-For-Review.Dec 16 2022, 6:26 PM
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)
ops-monitoring-bot added a comment.Dec 16 2022, 7:55 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti4007.ulsfo.wmnet with OS bullseye completed:

  • ganeti4007 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202212161819_robh_1894479_ganeti4007.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
RobH closed this task as Resolved.Dec 16 2022, 8:04 PM
RobH claimed this task.

@MoritzMuehlenhoff,

ganeti4007 is all yours and this resolves all pending misc ulsfo installs =]

gerritbot added a comment.Dec 19 2022, 2:01 PM

Change 869220 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Make ganeti4007 a Ganeti node

https://gerrit.wikimedia.org/r/869220

gerritbot added a project: Patch-For-Review.Dec 19 2022, 2:01 PM
gerritbot added a comment.Dec 20 2022, 8:16 AM

Change 869220 merged by Muehlenhoff:

[operations/puppet@production] Make ganeti4007 a Ganeti node

https://gerrit.wikimedia.org/r/869220

• MoritzMuehlenhoff added a comment.Dec 20 2022, 10:03 AM

ganeti4007 has been added to the ulsfo Ganeti cluster.

Stashbot added a comment.Dec 20 2022, 10:16 AM

Mentioned in SAL (#wikimedia-operations) [2022-12-20T10:16:40Z] <moritzm> rebalance ganeti cluster in ulsfo after adding new node and decom of the old hardware T317247

gerritbot added a comment.Jan 19 2023, 2:30 PM

Change 866441 merged by Ssingh:

[operations/puppet@production] site.pp: update LVS hosts in ulsfo

https://gerrit.wikimedia.org/r/866441

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits