Page MenuHomePhabricator
Create Task
Maniphest T317244

Q1:rack/setup/install cp40[37-51]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of 16 new cp hosts to replace the hosts in ulsfo.

Please note half of these have dual 6.4TB NVMe SSDs and half only have a single 6.4TB NVMe SSD.

Hostname / Racking / Installation Details

Hostnames: cp4037 and upwards
Racking Proposal: Odd hostnames in rack 22, even hostnames in rack 23.
Networking Setup: # of Connections:1, Speed:10G. Vlan:Private AAAA records:Y
Partitioning/Raid:
OS Distro: Bullseye
Sub-team Technical Contact: @BBlack

CP Nodes' mapping for replacements and where NVMEs go:

cp nodesCurrentReplacementDisks
text21-26, 33, 3437-44Single NVME
upload27-32, 35, 3645-52Dual NVME

Firmware Notes

These shipped with the following firmwares:
bios is newest 1.6.5 - no change needed
10G nic is 22.00.07.60 , - must downgrade to 21.85.21.92 or PXE fails
idrac is 5.10.30.00, cap at this and won't upgrade to 6.x which breaks https mgmt access - no change needed

Link to correct firmware: https://www.dell.com/support/home/en-us/drivers/driversdetails?driverid=rxp80&oscode=naa&productcode=poweredge-r450

Per host setup checklist

cp4037:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - had to steal the optic for xconnection to sv1, dac cable wrong type, production network not attached yet. - fully cabled netbox updated
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller) - downgraded nic to 21.85.21.92
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4038:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4039:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4040:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4041:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4042:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4043:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4044:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4045:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit has to run homer not new sre.network.configure-switch-interfaces
  • - bios/drac/serial setup/testing,- auto provision fails on new hosts, had to set manually
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4046:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4047:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4048:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4049:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4050:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4051:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
cp4052:
  • - receive in system on procurement task T311861/T3118622 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
hieradata: unify cp@ulsfo definitionsVgutierrezoperations/puppetproduction+3 -118
cp4052: update site.pp and related configs for cp (upload) roleSsinghoperations/puppetproduction+14 -1
cp4048: update site.pp and related configs for cp (upload) roleSsinghoperations/puppetproduction+14 -1
cp4040: update site.pp and related configs for cp (text) roleSsinghoperations/puppetproduction+12 -1
cp4052: update site.pp and related configs for cp (upload) roleSsinghoperations/puppetproduction+14 -1
cp4051: update site.pp and related configs for cp (upload) roleSsinghoperations/puppetproduction+14 -1
cp4050: update site.pp and related configs for cp (upload) roleSsinghoperations/puppetproduction+14 -1
cp4041: update site.pp and related configs for cp (text) roleSsinghoperations/puppetproduction+12 -1
cp4042: update site.pp and related configs for cp (text) roleSsinghoperations/puppetproduction+12 -1
cp4043: update site.pp and related configs for cp (text) roleSsinghoperations/puppetproduction+12 -1
cp4044: update site.pp and related configs for cp (text) roleSsinghoperations/puppetproduction+12 -1
cp4048: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+14 -1
cp4046: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+14 -1
cp4040: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+12 -1
ulsfo: single_backend for all new cache nodesBBlackoperations/puppetproduction+10 -0
cp4039: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+10 -1
cp4046: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+12 -1
cp403[35]: decommission hosts as part of ulsfo refreshSsinghoperations/puppetproduction+2 -16
cp402[2468], cp403[0246]: decommission hosts for ulsfo hardware refreshSsinghoperations/puppetproduction+2 -40
cp4025: decommission host as part of the ulsfo hardware refreshSsinghoperations/puppetproduction+1 -5
cp4023: decommission host as part of the ulsfo hardware refreshSsinghoperations/puppetproduction+1 -5
cp4049: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+12 -1
cp4049: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+12 -1
cp4047: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+12 -1
cp4037: update site.pp and related configs for cp roleSsinghoperations/puppetproduction+12 -1
install_server: add cp4037 to buster Linux 5.10 image installSsinghoperations/puppetproduction+1 -1
cache node disk layout p11n for F4 configBBlackoperations/puppetproduction+12 -10
cp4045 spare role setRobHoperations/puppetproduction+4 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
Unknown Object (Task)
 ResolvedRobHT317244 Q1:rack/setup/install cp40[37-51]
 ResolvedRequestRobHT318963 decommission cp4021 &n cp4027
 ResolvedMoritzMuehlenhoffT319067 add HBA355i support to installer
 ResolvedRequestRobHT321340 decommission cp4029
 ResolvedRequestRobHT321596 ulsfo unified decom task

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a project: Patch-For-Review.Oct 20 2022, 6:37 PM
gerritbot added a comment.Oct 20 2022, 6:43 PM

Change 845039 merged by Ssingh:

[operations/puppet@production] install_server: add cp4037 to buster Linux 5.10 image install

https://gerrit.wikimedia.org/r/845039

RobH added a subtask: T321340: decommission cp4029.Oct 20 2022, 6:46 PM
gerritbot added a comment.Oct 20 2022, 6:52 PM

Change 845040 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4037: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/845040

gerritbot added a comment.Oct 20 2022, 7:00 PM

Change 845040 merged by Ssingh:

[operations/puppet@production] cp4037: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/845040

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2022, 7:31 PM
RobH updated the task description. (Show Details)Oct 20 2022, 7:58 PM
gerritbot added a comment.Oct 20 2022, 11:33 PM

Change 845073 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4047: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/845073

gerritbot added a project: Patch-For-Review.Oct 20 2022, 11:33 PM
gerritbot added a comment.Oct 20 2022, 11:41 PM

Change 845073 merged by Ssingh:

[operations/puppet@production] cp4047: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/845073

gerritbot added a comment.Oct 21 2022, 12:16 AM

Change 845075 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4049: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/845075

gerritbot added a comment.Oct 21 2022, 2:05 PM

Change 845550 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4049: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/845550

gerritbot added a comment.Oct 21 2022, 2:07 PM

Change 845550 merged by Ssingh:

[operations/puppet@production] cp4049: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/845550

Stashbot added a comment.Oct 21 2022, 2:21 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-21T14:21:29Z] <sukhe> pool new host cp4037: T317244

ssingh updated the task description. (Show Details)Oct 21 2022, 2:22 PM
ssingh updated the task description. (Show Details)Oct 21 2022, 3:26 PM
Stashbot added a comment.Oct 21 2022, 6:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-21T18:38:47Z] <sukhe> pool new host cp4047: T317244

Stashbot added a comment.Oct 21 2022, 6:38 PM

Mentioned in SAL (#wikimedia-operations) [2022-10-21T18:38:52Z] <sukhe> pool new host cp4049: T317244

ssingh updated the task description. (Show Details)Oct 24 2022, 2:26 PM
gerritbot added a comment.Oct 24 2022, 4:15 PM

Change 848423 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4023: decommission host as part of the ulsfo hardware refresh

https://gerrit.wikimedia.org/r/848423

gerritbot added a comment.Oct 24 2022, 4:17 PM

Change 845075 abandoned by Ssingh:

[operations/puppet@production] cp4049: update site.pp and related configs for cp role

Reason:

merged in 46148efdc3830

https://gerrit.wikimedia.org/r/845075

RobH added a subtask: T321596: ulsfo unified decom task.Oct 25 2022, 6:13 PM
RobH added a comment.Oct 25 2022, 6:19 PM

failure of provision script against cp4039

[1/30, retrying in 30.00s] Polling task: JID_667217070909 not completed yet: status=OK, state=Running, completed=10%
Raised while handling: Not all changes were applied successfully, see the ones reported above that starts with "Updated value..."
Enabling PXE boot on NIC NIC.Integrated.1-1-1
Skipped set of attribute BIOS.Setup.1-1 -> BootMode, has already the correct value: Bios
Updated value for attribute BIOS.Setup.1-1 -> CpuInterconnectBusLinkPower (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> EnergyPerformanceBias (marked Set On Import to True): MaxPower => BalancedPerformance
Created attribute BIOS.Setup.1-1 -> InternalUsb (with Set On Import True) with value Off
Updated value for attribute BIOS.Setup.1-1 -> PcieAspmL1 (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcC1E (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcCStates (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcPwrPerf (marked Set On Import to True): MaxPerf => OsDbpm
Updated value for attribute BIOS.Setup.1-1 -> ProcVirtualization: Enabled => Disabled
Updated value for attribute BIOS.Setup.1-1 -> ProcX2Apic: Enabled => Disabled
Updated value for attribute BIOS.Setup.1-1 -> SerialComm: OnNoConRedir => OnConRedirCom2
Updated value for attribute BIOS.Setup.1-1 -> SerialPortAddress: Com1 => Serial1Com1Serial2Com2
Updated value for attribute BIOS.Setup.1-1 -> SysProfile: PerfOptimized => PerfPerWattOptimizedOs
Updated value for attribute BIOS.Setup.1-1 -> UncoreFrequency (marked Set On Import to True): MaxUFS => DynamicUFS
Updated value for attribute BIOS.Setup.1-1 -> UsbPorts: AllOn => OnlyBackPortsOn
Updated value for attribute BIOS.Setup.1-1 -> SetBootOrderEn: HardDisk.List.1-1,NIC.Embedded.1-1-1,NIC.Embedded.2-1-1,NIC.Integrated.1-1-1,NIC.Integrated.1-2-1 => HardDisk.List.1-1,NIC.Integrated.1-1-1
Updated value for attribute BIOS.Setup.1-1 -> BiosBootSeq (marked Set On Import to True): HardDisk.List.1-1, NIC.Embedded.1-1-1, NIC.Embedded.2-1-1, NIC.Integrated.1-1-1, NIC.Integrated.1-2-1 => HardDisk.List.1-1, NIC.Integrated.1-1-1
Skipped set of attribute iDRAC.Embedded.1 -> IPMILan.1#Enable, has already the correct value: Enabled
Skipped set of attribute iDRAC.Embedded.1 -> IPv4.1#DHCPEnable, has already the correct value: Disabled
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#Address, has already the correct value: 10.128.128.24
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#DNS1, has already the correct value: 10.3.0.1
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#Gateway, has already the correct value: 10.128.128.1
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#Netmask, has already the correct value: 255.255.128.0
Skipped set of attribute iDRAC.Embedded.1 -> NICStatic.1#DNSDomainFromDHCP, has already the correct value: Disabled
Skipped set of attribute System.Embedded.1 -> ServerPwr.1#PSRapidOn, has already the correct value: Disabled
Skipped set of attribute NIC.Embedded.1-1-1 -> LegacyBootProto, has already the correct value: NONE
Skipped set of attribute NIC.Embedded.2-1-1 -> LegacyBootProto, has already the correct value: NONE
Updated value for attribute NIC.Integrated.1-1-1 -> LegacyBootProto: NONE => PXE
Skipped set of attribute NIC.Integrated.1-2-1 -> LegacyBootProto, has already the correct value: NONE
RobH updated the task description. (Show Details)Oct 25 2022, 6:35 PM
gerritbot added a comment.Oct 25 2022, 7:15 PM

Change 848423 merged by Ssingh:

[operations/puppet@production] cp4023: decommission host as part of the ulsfo hardware refresh

https://gerrit.wikimedia.org/r/848423

gerritbot added a comment.Oct 25 2022, 7:23 PM

Change 849140 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4025: decommission host as part of the ulsfo hardware refresh

https://gerrit.wikimedia.org/r/849140

gerritbot added a comment.Oct 25 2022, 7:27 PM

Change 849140 merged by Ssingh:

[operations/puppet@production] cp4025: decommission host as part of the ulsfo hardware refresh

https://gerrit.wikimedia.org/r/849140

gerritbot added a comment.Oct 25 2022, 7:34 PM

Change 849141 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp402[2468]: decommission hosts as part of the ulsfo hardware refresh

https://gerrit.wikimedia.org/r/849141

RobH updated the task description. (Show Details)Oct 25 2022, 10:07 PM
RobH renamed this task from Q1:rack/setup/install cp40[37-52] to Q1:rack/setup/install cp40[37-51].Oct 25 2022, 11:07 PM
RobH removed a project: Patch-For-Review.
RobH updated the task description. (Show Details)
gerritbot added a comment.Oct 26 2022, 12:02 AM

Change 849141 merged by Ssingh:

[operations/puppet@production] cp402[2468], cp403[0246]: decommission hosts for ulsfo hardware refresh

https://gerrit.wikimedia.org/r/849141

gerritbot added a comment.Oct 26 2022, 12:51 AM

Change 849200 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp403[35]: decommission hosts as part of ulsfo refresh

https://gerrit.wikimedia.org/r/849200

gerritbot added a project: Patch-For-Review.Oct 26 2022, 12:51 AM
gerritbot added a comment.Oct 26 2022, 1:15 AM

Change 849200 merged by Ssingh:

[operations/puppet@production] cp403[35]: decommission hosts as part of ulsfo refresh

https://gerrit.wikimedia.org/r/849200

RobH updated the task description. (Show Details)Oct 26 2022, 1:25 AM
RobH updated the task description. (Show Details)Oct 26 2022, 1:33 AM
gerritbot added a comment.Oct 26 2022, 1:47 AM

Change 849207 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4046: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849207

ssingh updated the task description. (Show Details)Oct 26 2022, 11:06 AM
gerritbot added a comment.Oct 26 2022, 11:09 AM

Change 849207 merged by Ssingh:

[operations/puppet@production] cp4046: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849207

ops-monitoring-bot added a comment.Oct 26 2022, 11:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4046.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 26 2022, 11:24 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4046.ulsfo.wmnet with OS buster executed with errors:

  • cp4046 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Oct 26 2022, 11:53 AM

Change 849534 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4039: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849534

gerritbot added a comment.Oct 26 2022, 12:04 PM

Change 849534 merged by Ssingh:

[operations/puppet@production] cp4039: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849534

gerritbot added a comment.Oct 26 2022, 3:12 PM

Change 849598 had a related patch set uploaded (by BBlack; author: BBlack):

[operations/puppet@production] ulsfo: single_backend for all new cache nodes

https://gerrit.wikimedia.org/r/849598

gerritbot added a comment.Oct 26 2022, 3:13 PM

Change 849598 merged by BBlack:

[operations/puppet@production] ulsfo: single_backend for all new cache nodes

https://gerrit.wikimedia.org/r/849598

gerritbot added a comment.Oct 26 2022, 3:40 PM

Change 849602 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4040: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849602

gerritbot added a comment.Oct 26 2022, 3:41 PM

Change 849602 merged by Ssingh:

[operations/puppet@production] cp4040: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849602

ops-monitoring-bot added a comment.Oct 26 2022, 3:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4040.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 26 2022, 3:54 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4040.ulsfo.wmnet with OS buster executed with errors:

  • cp4040 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Oct 26 2022, 5:50 PM

Change 849634 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4046: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849634

gerritbot added a comment.Oct 26 2022, 5:52 PM

Change 849634 merged by Ssingh:

[operations/puppet@production] cp4046: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849634

gerritbot added a comment.Oct 26 2022, 10:37 PM

Change 849702 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4048: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849702

gerritbot added a comment.Oct 26 2022, 10:38 PM

Change 849702 merged by Ssingh:

[operations/puppet@production] cp4048: update site.pp and related configs for cp role

https://gerrit.wikimedia.org/r/849702

ops-monitoring-bot added a comment.Oct 26 2022, 10:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4048.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 26 2022, 11:23 PM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4048.ulsfo.wmnet with OS buster executed with errors:

  • cp4048 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Oct 26 2022, 11:37 PM

Change 849713 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4044: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/849713

gerritbot added a comment.Oct 26 2022, 11:39 PM

Change 849713 merged by Ssingh:

[operations/puppet@production] cp4044: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/849713

gerritbot added a comment.Oct 27 2022, 12:54 AM

Change 849715 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4043: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/849715

gerritbot added a comment.Oct 27 2022, 12:57 AM

Change 849715 merged by Ssingh:

[operations/puppet@production] cp4043: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/849715

ops-monitoring-bot added a comment.Oct 27 2022, 12:59 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4043.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 27 2022, 1:11 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4043.ulsfo.wmnet with OS buster executed with errors:

  • cp4043 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 27 2022, 1:15 AM

Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp4043.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 27 2022, 2:13 AM

Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp4043.ulsfo.wmnet with OS buster completed:

  • cp4043 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202210270115_sukhe_3420882_cp4043.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
gerritbot added a comment.Oct 27 2022, 1:36 PM

Change 850158 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4042: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/850158

gerritbot added a comment.Oct 27 2022, 1:38 PM

Change 850158 merged by Ssingh:

[operations/puppet@production] cp4042: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/850158

gerritbot added a comment.Oct 27 2022, 2:43 PM

Change 850176 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4041: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/850176

gerritbot added a comment.Oct 27 2022, 2:45 PM

Change 850176 merged by Ssingh:

[operations/puppet@production] cp4041: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/850176

gerritbot added a comment.Oct 27 2022, 3:32 PM

Change 850186 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4050: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850186

gerritbot added a comment.Oct 27 2022, 3:44 PM

Change 850186 merged by Ssingh:

[operations/puppet@production] cp4050: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850186

gerritbot added a comment.Oct 27 2022, 4:01 PM

Change 850192 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4051: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850192

gerritbot added a comment.Oct 27 2022, 4:02 PM

Change 850193 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4052: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850193

ssingh updated the task description. (Show Details)Oct 27 2022, 4:07 PM
ssingh updated the task description. (Show Details)
gerritbot added a comment.Oct 27 2022, 4:43 PM

Change 850192 merged by Ssingh:

[operations/puppet@production] cp4051: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850192

RobH added a comment.EditedOct 27 2022, 5:12 PM

cp4040 and cp4048 had no link lights and when investigated had the DAC cable clicked in on the NIC, but not pressed in quite all the way. Reseated and the link lights came up immediately.

ssingh updated the task description. (Show Details)Oct 27 2022, 6:00 PM
RobH updated the task description. (Show Details)Oct 27 2022, 6:02 PM
gerritbot added a comment.Oct 27 2022, 8:48 PM

Change 850193 merged by Ssingh:

[operations/puppet@production] cp4052: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850193

gerritbot added a comment.Oct 27 2022, 8:52 PM

Change 850273 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4040: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/850273

RobH added a comment.Oct 27 2022, 9:18 PM

When trying to run the sre.hosts.provision script on cp4052, I get the following issue:

[1/30, retrying in 30.00s] Polling task: JID_669057087428 not completed yet: status=OK, state=Running, completed=10%
Skipped set of attribute BIOS.Setup.1-1 -> BootMode, has already the correct value: Bios
Updated value for attribute BIOS.Setup.1-1 -> CpuInterconnectBusLinkPower (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> EnergyPerformanceBias (marked Set On Import to True): MaxPower => BalancedPerformance
Updated value for attribute BIOS.Setup.1-1 -> PcieAspmL1 (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcC1E (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcCStates (marked Set On Import to True): Disabled => Enabled
Updated value for attribute BIOS.Setup.1-1 -> ProcPwrPerf (marked Set On Import to True): MaxPerf => OsDbpm
Updated value for attribute BIOS.Setup.1-1 -> ProcVirtualization: Enabled => Disabled
Updated value for attribute BIOS.Setup.1-1 -> ProcX2Apic: Enabled => Disabled
Updated value for attribute BIOS.Setup.1-1 -> SysProfile: PerfOptimized => PerfPerWattOptimizedOs
Updated value for attribute BIOS.Setup.1-1 -> UncoreFrequency (marked Set On Import to True): MaxUFS => DynamicUFS
Updated value for attribute BIOS.Setup.1-1 -> UsbPorts: AllOn => OnlyBackPortsOn
Updated value for attribute BIOS.Setup.1-1 -> SerialComm: OnNoConRedir => OnConRedir
Updated value for attribute BIOS.Setup.1-1 -> SerialPortAddress: Com1 => Com2
Updated value for attribute BIOS.Setup.1-1 -> SetBootOrderEn: HardDisk.List.1-1 => HardDisk.List.1-1,NIC.Integrated.1-1-1
Updated value for attribute BIOS.Setup.1-1 -> BiosBootSeq (marked Set On Import to True): HardDisk.List.1-1 => HardDisk.List.1-1, NIC.Integrated.1-1-1
Skipped set of attribute iDRAC.Embedded.1 -> IPMILan.1#Enable, has already the correct value: Enabled
Skipped set of attribute iDRAC.Embedded.1 -> IPv4.1#DHCPEnable, has already the correct value: Disabled
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#Address, has already the correct value: 10.128.128.10
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#DNS1, has already the correct value: 10.3.0.1
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#Gateway, has already the correct value: 10.128.128.1
Skipped set of attribute iDRAC.Embedded.1 -> IPv4Static.1#Netmask, has already the correct value: 255.255.128.0
Skipped set of attribute iDRAC.Embedded.1 -> NICStatic.1#DNSDomainFromDHCP, has already the correct value: Disabled
Skipped set of attribute System.Embedded.1 -> ServerPwr.1#PSRapidOn, has already the correct value: Disabled
Skipped set of attribute NIC.Embedded.1-1-1 -> LegacyBootProto, has already the correct value: NONE
Skipped set of attribute NIC.Embedded.2-1-1 -> LegacyBootProto, has already the correct value: NONE
Skipped set of attribute NIC.Integrated.1-1-1 -> LegacyBootProto, has already the correct value: PXE
Skipped set of attribute NIC.Integrated.1-2-1 -> LegacyBootProto, has already the correct value: NONE
First attempt to load the new configuration failed, auto-retrying once
Testing Redfish API connection to cp4052 (10.128.128.10)
[IDRAC.2.5.SYS057] Exporting Server Configuration Profile.
[1/30, retrying in 30.00s] Polling task: JID_669057399361 not completed yet: status=OK, state=Running, completed=10%
Raised while handling: Not all changes were applied successfully, see the ones reported above that starts with "Updated value..."
gerritbot added a comment.Oct 27 2022, 11:59 PM

Change 850273 merged by Ssingh:

[operations/puppet@production] cp4040: update site.pp and related configs for cp (text) role

https://gerrit.wikimedia.org/r/850273

gerritbot added a comment.Oct 28 2022, 12:48 AM

Change 850307 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4048: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850307

gerritbot added a comment.Oct 28 2022, 12:58 AM

Change 850307 merged by Ssingh:

[operations/puppet@production] cp4048: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850307

gerritbot added a comment.Oct 28 2022, 1:58 PM

Change 850483 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] cp4052: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850483

ops-monitoring-bot added a comment.Oct 28 2022, 3:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host cp4052.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 28 2022, 3:50 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host cp4052.ulsfo.wmnet with OS buster executed with errors:

  • cp4052 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
RobH updated the task description. (Show Details)Oct 28 2022, 4:35 PM
gerritbot added a comment.Oct 28 2022, 4:40 PM

Change 850483 merged by Ssingh:

[operations/puppet@production] cp4052: update site.pp and related configs for cp (upload) role

https://gerrit.wikimedia.org/r/850483

ssingh updated the task description. (Show Details)Oct 28 2022, 5:32 PM
ssingh updated the task description. (Show Details)
ssingh added subscribers: cmooney, Volans, Papaul.Oct 28 2022, 5:35 PM

Traffic update: all the new cp hosts in ulsfo are marked active and pooled. Rob: Feel free to mark this as resolved.

Thanks to @RobH, @Papaul, @BBlack, @cmooney, @Volans for their help in getting this done.

RobH closed this task as Resolved.Oct 28 2022, 5:36 PM
gerritbot added a comment.Nov 11 2022, 9:39 AM

Change 855962 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hieradata: unify ulsfo definitions

https://gerrit.wikimedia.org/r/855962

gerritbot added a comment.Nov 11 2022, 9:54 AM

Change 855962 merged by Vgutierrez:

[operations/puppet@production] hieradata: unify cp@ulsfo definitions

https://gerrit.wikimedia.org/r/855962

RobH closed subtask T321340: decommission cp4029 as Resolved.Jan 3 2023, 7:35 PM
RobH closed subtask T321596: ulsfo unified decom task as Resolved.Jan 12 2023, 4:54 PM
ssingh closed subtask T319067: add HBA355i support to installer as Resolved.Jan 12 2023, 5:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits