Page MenuHomePhabricator
Create Task
Maniphest T326848

Q3:rack/setup/install ms-fe2013 - ms-fe2014, thanos-fe2004
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ms-fe2013 - ms-fe2014, thanos-fe2004

Hostname / Racking / Installation Details

Hostnames: ms-fe2013 - ms-fe2014, thanos-fe2004
Racking Proposal:
ms-fe: One host per row
thanos-fe: Row C as it doesn't have a thanos-fe host yet
Partitioning/Raid: Same as existing ms-fe hosts
OS Distro: Bullseye
Sub-team Technical Contact: @MatthewVernon

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ms-fe2013: Rack: A2 - U32 - Port 31
  • - receive in system on procurement task T324220 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ms-fe2014: Rack: B2 - U33 - Port 32
  • - receive in system on procurement task T324220 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
thanos-fe2004: Rack: C7 - U7 - Port 6
  • - receive in system on procurement task T324220 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role::insetup::serviceops.
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
sre.hardware.upgrade-firmware: incorrect error statusoperations/cookbooksmaster+11 -2
Ad new ms-fe and thanos-fe node to site.ppoperations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Jan 12 2023, 6:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 12 2023, 6:45 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.Jan 12 2023, 6:45 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH mentioned this in T326849: serviceops implementation tracking for ms-fe2013 - ms-fe2014, thanos-fe2004.
RobH unsubscribed.
Maintenance_bot added a project: SRE.Jan 12 2023, 7:29 PM
RobH closed subtask T326849: serviceops implementation tracking for ms-fe2013 - ms-fe2014, thanos-fe2004 as Invalid.Jan 12 2023, 8:06 PM
LSobanski subscribed.Feb 16 2023, 2:52 PM
Jhancock.wm assigned this task to Papaul.Feb 16 2023, 8:43 PM
Jhancock.wm updated the task description. (Show Details)
MatthewVernon added a comment.Feb 23 2023, 4:49 PM

Hi @Papaul could you give me an update on timescales for getting this hardware ready to go, please? From an operational perspective, it'd be great to have these nodes in production ASAP. Thanks :)

Papaul added a subscriber: jbond.Feb 24 2023, 7:29 PM

@jbond

poweredge-r450: picking DellDriverCategory.BIOS update file
We have found multiple entries please pick from the list below:
0: /srv/firmware/poweredge-r450/BIOS/BIOS_G7K8G_WN64_1.8.2_01.EXE
1: /srv/firmware/poweredge-r450/BIOS/BIOS_DHRG5_WN64_1.7.5.EXE
2: Download new file
==> Please select the entry you want
> 0
User input is: "0"
ms-fe2013.codfw.wmnet (BIOS): target_version: 1.8.2, current_version: 1.8.2
ms-fe2013.codfw.wmnet (BIOS): Skipping already at target version 1.8.2
ms-fe2013.codfw.wmnet: no job_id for BIOS update
Resetting chassis power status for ms-fe2013 to ForceOff
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-fe2013']
pt1979@cumin2002:~$
Papaul updated the task description. (Show Details)Feb 24 2023, 7:36 PM
gerritbot added a comment.Feb 24 2023, 9:24 PM

Change 891914 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Ad new ms-fe and thanos-fe node to site.pp

https://gerrit.wikimedia.org/r/891914

gerritbot added a project: Patch-For-Review.Feb 24 2023, 9:24 PM
gerritbot added a comment.Feb 24 2023, 9:34 PM

Change 891914 merged by Papaul:

[operations/puppet@production] Ad new ms-fe and thanos-fe node to site.pp

https://gerrit.wikimedia.org/r/891914

ops-monitoring-bot added a comment.Feb 24 2023, 9:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host ms-fe2013.codfw.wmnet with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Feb 24 2023, 10:10 PM
ops-monitoring-bot added a comment.Feb 24 2023, 10:49 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host ms-fe2013.codfw.wmnet with OS bullseye completed:

  • ms-fe2013 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302242153_pt1979_990154_ms-fe2013.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
Papaul updated the task description. (Show Details)Feb 24 2023, 10:52 PM
jbond added a comment.Feb 27 2023, 11:14 AM
In T326848#8645012, @Papaul wrote:

@jbond

poweredge-r450: picking DellDriverCategory.BIOS update file
We have found multiple entries please pick from the list below:
0: /srv/firmware/poweredge-r450/BIOS/BIOS_G7K8G_WN64_1.8.2_01.EXE
1: /srv/firmware/poweredge-r450/BIOS/BIOS_DHRG5_WN64_1.7.5.EXE
2: Download new file
==> Please select the entry you want
> 0
User input is: "0"
ms-fe2013.codfw.wmnet (BIOS): target_version: 1.8.2, current_version: 1.8.2
ms-fe2013.codfw.wmnet (BIOS): Skipping already at target version 1.8.2
ms-fe2013.codfw.wmnet: no job_id for BIOS update
Resetting chassis power status for ms-fe2013 to ForceOff
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['ms-fe2013']
pt1979@cumin2002:~$

From the output and the logs, specifically the following line it looks like the upgrade works successfully however there is some issue with handling the command that powers the server back down.

ms-fe2013.codfw.wmnet (BIOS): target_version: 1.8.2, current_version: 1.8.2

if you see this its safe to assume that the firmware has been upgraded and there probably some other minor issue with the cookbook

Stashbot added a comment.Feb 27 2023, 11:28 AM

Mentioned in SAL (#wikimedia-operations) [2023-02-27T11:28:10Z] <jbond@cumin2002> START - Cookbook sre.hosts.downtime for 4:00:00 on ms-fe2013.codfw.wmnet with reason: testing redfish T326848

Stashbot added a comment.Feb 27 2023, 11:28 AM

Mentioned in SAL (#wikimedia-operations) [2023-02-27T11:28:25Z] <jbond@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on ms-fe2013.codfw.wmnet with reason: testing redfish T326848

gerritbot added a comment.Feb 27 2023, 11:55 AM

Change 892393 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/cookbooks@master] sre.hardware.upgrade-firmware: incorrect error status

https://gerrit.wikimedia.org/r/892393

gerritbot added a project: Patch-For-Review.Feb 27 2023, 11:55 AM
jbond added a comment.Feb 27 2023, 11:56 AM

@MatthewVernon the firmware, bios and network have all been upgraded so should be good to procead

gerritbot added a comment.Feb 27 2023, 2:29 PM

Change 892393 merged by jenkins-bot:

[operations/cookbooks@master] sre.hardware.upgrade-firmware: incorrect error status

https://gerrit.wikimedia.org/r/892393

Maintenance_bot removed a project: Patch-For-Review.Feb 27 2023, 2:29 PM
ops-monitoring-bot added a comment.Mar 2 2023, 7:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host ms-fe2014.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 2 2023, 7:59 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host ms-fe2014.codfw.wmnet with OS bullseye completed:

  • ms-fe2014 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303021905_pt1979_2640155_ms-fe2014.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
ops-monitoring-bot added a comment.Mar 2 2023, 8:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host thanos-fe2004.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Mar 2 2023, 9:01 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host thanos-fe2004.codfw.wmnet with OS bullseye completed:

  • thanos-fe2004 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202303022012_pt1979_2653067_thanos-fe2004.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
Papaul closed this task as Resolved.Mar 2 2023, 9:21 PM
Papaul updated the task description. (Show Details)

This is complete

MatthewVernon added a comment.Mar 3 2023, 10:10 AM

Thanks!

Eevans subscribed.Mar 3 2023, 4:28 PM
Eevans mentioned this in T331178: Bring ms-fe201[3-4] into service.Mar 3 2023, 10:40 PM
fgiunchedi mentioned this in T336348: Put in service thanos-fe[12]004.May 10 2023, 8:31 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL