Page MenuHomePhabricator
Create Task
Maniphest T349839

Q2:rack/setup/install ms-be refresh
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of X

Hostname / Racking / Installation Details

hostnames: ms-be20[74-80]
racking: not more than 1 per rack, please, though they can share with existing nodes (e.g. you could put them in the racks the old systems are coming out of, which I think are A2,A7,B2,B7,C2,C7,D7)
networking: 10G private VLAN like existing ms-be* nodes
Partitioning/Raid: JBOD, please unlike previous ms-be* nodes, we now want everything non-RAID (cf T308677)
Point of Contact: @MatthewVernon

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ms-be2074:
  • - receive in system on procurement task T348059 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ms-be2075:
  • - receive in system on procurement task T348059 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ms-be2076:
  • - receive in system on procurement task T348059 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ms-be2077:
  • - receive in system on procurement task T348059 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ms-be2078:
  • - receive in system on procurement task T348059 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ms-be2079:
  • - receive in system on procurement task T348059 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ms-be2080:
  • - receive in system on procurement task T348059 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer from an active cumin host to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT349839 Q2:rack/setup/install ms-be refresh

Event Timeline

RobH created this task.Oct 26 2023, 4:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 26 2023, 4:14 PM
RobH mentioned this in Unknown Object (Task).Oct 26 2023, 4:14 PM
RobH added a parent task: Unknown Object (Task).
RobH assigned this task to MatthewVernon.Oct 26 2023, 4:17 PM
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH added subscribers: KOfori, MatthewVernon.

@MatthewVernon,

Per @KOfori's recommendation, you would be the best person to provide the hostname and racking details for these 7 new ms-be hosts slated to arrive in codfw this quarter.

Please update the racking details section and assign back to me for followup. Thanks!

Maintenance_bot added a project: SRE.Oct 26 2023, 4:29 PM
MatthewVernon reassigned this task from MatthewVernon to RobH.Nov 17 2023, 3:33 PM

Hi. I think:
hostnames: ms-be20[74-80]
racking: not more than 1 per rack, please, though they can share with existing nodes (e.g. you could put them in the racks the old systems are coming out of, which I think are A2,A7,B2,B7,C2,C7,D7)
networking: 10G private VLAN like existing ms-be* nodes
Partitioning/Raid: JBOD, please unlike previous ms-be* nodes, we now want everything non-RAID (cf T308677)

Is that all OK?

RobH removed RobH as the assignee of this task.Nov 17 2023, 6:04 PM
RobH updated the task description. (Show Details)
RobH unsubscribed.

Updated task description with updated racking details and removing myself as assignee. Once these arrive on-site, one of our ops-codfw engineers will rack and update this task.

MatthewVernon mentioned this in T353149: Q3 ms backend refresh work.Dec 11 2023, 2:18 PM
MatthewVernon added a project: SRE-swift-storage.Dec 11 2023, 2:21 PM
Jhancock.wm updated the task description. (Show Details)Dec 15 2023, 7:36 PM
Jhancock.wm claimed this task.Dec 15 2023, 7:46 PM
Jhancock.wm updated the task description. (Show Details)
Jhancock.wm updated the task description. (Show Details)Dec 18 2023, 4:48 PM
ops-monitoring-bot added a comment.Dec 18 2023, 5:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2074.codfw.wmnet with OS bullseye

Jhancock.wm updated the task description. (Show Details)Dec 20 2023, 2:20 PM
ops-monitoring-bot added a comment.Dec 20 2023, 3:22 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2074.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2074 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 20 2023, 4:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host ms-be2080.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 8:37 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2074.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 9:24 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2074.codfw.wmnet with OS bullseye completed:

  • ms-be2074 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202056_jhancock_589712_ms-be2074.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 20 2023, 9:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2075.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 9:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2076.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 9:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2077.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 9:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2078.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 9:33 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2079.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 9:34 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2080.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 10:16 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2076.codfw.wmnet with OS bullseye completed:

  • ms-be2076 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202152_jhancock_618123_ms-be2076.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 20 2023, 10:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2078.codfw.wmnet with OS bullseye completed:

  • ms-be2078 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202148_jhancock_618433_ms-be2078.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.Dec 20 2023, 10:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2077.codfw.wmnet with OS bullseye completed:

  • ms-be2077 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202154_jhancock_618142_ms-be2077.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 20 2023, 10:23 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2079.codfw.wmnet with OS bullseye completed:

  • ms-be2079 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202157_jhancock_618479_ms-be2079.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 20 2023, 10:24 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2080.codfw.wmnet with OS bullseye completed:

  • ms-be2080 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Unable to downtime the new host on Icinga/Alertmanager, the sre.hosts.downtime cookbook returned 99
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312202159_jhancock_618532_ms-be2080.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Dec 20 2023, 11:00 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2075.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Dec 20 2023, 11:44 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2075.codfw.wmnet with OS bullseye executed with errors:

  • ms-be2075 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Dec 20 2023, 11:47 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host ms-be2075.codfw.wmnet with OS bullseye

Jhancock.wm updated the task description. (Show Details)Dec 20 2023, 11:58 PM
ops-monitoring-bot added a comment.Dec 21 2023, 1:13 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host ms-be2075.codfw.wmnet with OS bullseye completed:

  • ms-be2075 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202312210027_jhancock_691831_ms-be2075.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Dec 21 2023, 1:15 AM
Jhancock.wm updated the task description. (Show Details)

@MatthewVernon apologies for the wait on this one. all good now.

MatthewVernon added a comment.Dec 21 2023, 10:25 AM

Thanks :)

MatthewVernon mentioned this in T415189: DHCP failing for at least 2 ms-be servers in codfw.Wed, Jan 21, 12:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits