Page MenuHomePhabricator
Create Task
Maniphest T299433

Q3:(Need By: TBD) rack/setup/install ml-cache200[1-3]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of ml-cache200[1-3]

Hostname / Racking / Installation Details

Hostnames: ml-cache200[1-3]
Racking Proposal: Spread in rows as much as possible, and bonus point if they don't share anything with the ml-serve200* nodes.
Networking/Subnet/VLAN/IP: 10G if possible would be good, but otherwise we can use 1G as well. Private VLAN.
Partitioning/Raid: We'll likely need a root partition + a srv partition, so if these nodes have already a standard recipe to use it will be fine, otherwise we'll find one.
OS Distro: Bullseye (default unless otherwise specified)

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ml-cache2001: A2 U10
  • - receive in system on procurement task T297640 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ml-cache2002: B2 U10
  • - receive in system on procurement task T297640 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.
ml-cache2003:C2 U20
  • - receive in system on procurement task T297640 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - bios/drac/serial setup/testing, see Lifecycle Steps & Automatic BIOS setup details
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - this should include updates to netboot.pp, and site.pp role(insetup) or cp systems use role(insetup::nofirm).
  • - OS installation & initital puppet run via sre.hosts.reimage cookbook.

Details

SubjectRepoBranchLines +/-
partman: add missing .cfg file extension to recipe used by ml-cacheoperations/puppetproduction+1 -1
Add ml-cache200[1-3] to site.ppoperations/puppetproduction+5 -0
install_server: add partman recipe for ml-cache nodesoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedPapaulT299433 Q3:(Need By: TBD) rack/setup/install ml-cache200[1-3]

Event Timeline

RobH created this task.Jan 18 2022, 5:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 18 2022, 5:45 PM
RobH mentioned this in Unknown Object (Task).Jan 18 2022, 5:45 PM
Maintenance_bot added a project: SRE.Jan 18 2022, 5:45 PM
RobH assigned this task to Papaul.Jan 18 2022, 5:45 PM
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.
RobH added subscribers: calbon, elukey.Jan 18 2022, 5:57 PM
elukey moved this task from Unsorted to Backlog/SRE on the Machine-Learning-Team board.Jan 19 2022, 6:12 PM
Papaul added a comment.Feb 17 2022, 2:43 PM

@elukey can you please get me the Partitioning/Raid information?

Thanks

Papaul updated the task description. (Show Details)Feb 17 2022, 2:45 PM
elukey added a comment.Feb 17 2022, 2:45 PM

@Papaul Hi! IIRC these nodes have two 2TB disks, so I'd go for the standard raid1 recipe: echo partman/standard.cfg partman/raid1-2dev

Lemme know if it makes sense.

gerritbot added a comment.Feb 17 2022, 2:49 PM

Change 763518 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: add partman recipe for ml-cache nodes

https://gerrit.wikimedia.org/r/763518

gerritbot added a project: Patch-For-Review.Feb 17 2022, 2:49 PM

Change 763518 merged by Elukey:

[operations/puppet@production] install_server: add partman recipe for ml-cache nodes

https://gerrit.wikimedia.org/r/763518

elukey added a comment.Feb 17 2022, 2:52 PM

Went ahead and merged the change, I've also ran puppet across install nodes, so you can install the os whenever you want :)

Papaul added a comment.Feb 17 2022, 3:07 PM

@elukey thanks

Maintenance_bot removed a project: Patch-For-Review.Feb 17 2022, 3:11 PM
Papaul updated the task description. (Show Details)Feb 17 2022, 9:19 PM
Papaul updated the task description. (Show Details)Feb 18 2022, 3:13 PM
Papaul updated the task description. (Show Details)Feb 18 2022, 4:28 PM
gerritbot added a comment.Feb 18 2022, 8:50 PM

Change 763818 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Add ml-cache200[1-3] to site.pp

https://gerrit.wikimedia.org/r/763818

gerritbot added a project: Patch-For-Review.Feb 18 2022, 8:50 PM
gerritbot added a comment.Feb 18 2022, 8:52 PM

Change 763818 merged by Papaul:

[operations/puppet@production] Add ml-cache200[1-3] to site.pp

https://gerrit.wikimedia.org/r/763818

ops-monitoring-bot added a comment.Feb 18 2022, 8:57 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host ml-cache2002.codfw.wmnet with OS bullseye

Papaul added a comment.Feb 18 2022, 9:05 PM

@elukey can you please double check again the partman i am getting the error below

Failed to retrieve the preconfiguration file            │
   │ The file needed for preconfiguration could not be retrieved from  │
   │ http://apt.wikimedia.org/autoinstall/partman/raid1-2dev. The      │
   │ installation will proceed in non-automated mode.
Dzahn subscribed.Feb 18 2022, 9:08 PM

@Papaul It's just missing the ".cfg" file ending.

https://apt.wikimedia.org/autoinstall/partman/

gerritbot added a comment.Feb 18 2022, 9:09 PM

Change 763837 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] partman: add missing .cfg file extension to recipe used by ml-cache

https://gerrit.wikimedia.org/r/763837

gerritbot added a comment.Feb 18 2022, 9:12 PM

Change 763837 merged by Dzahn:

[operations/puppet@production] partman: add missing .cfg file extension to recipe used by ml-cache

https://gerrit.wikimedia.org/r/763837

Dzahn added a comment.Feb 18 2022, 9:14 PM

@Papaul deployed fix and ran puppet on apt1001. try again now

Papaul added a comment.Feb 18 2022, 9:14 PM

thanks

ops-monitoring-bot added a comment.Feb 18 2022, 9:22 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host ml-cache2002.codfw.wmnet with OS bullseye executed with errors:

  • ml-cache2002 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 18 2022, 9:24 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host ml-cache2002.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 18 2022, 9:56 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host ml-cache2002.codfw.wmnet with OS bullseye completed:

  • ml-cache2002 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202182124_pt1979_4095899_ml-cache2002.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 18 2022, 9:59 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host ml-cache2003.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 18 2022, 10:36 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host ml-cache2003.codfw.wmnet with OS bullseye completed:

  • ml-cache2003 (WARN)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202182159_pt1979_4100767_ml-cache2003.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Feb 18 2022, 10:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host ml-cache2001.codfw.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 18 2022, 11:18 PM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host ml-cache2001.codfw.wmnet with OS bullseye completed:

  • ml-cache2001 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202182246_pt1979_4108914_ml-cache2001.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
Papaul closed this task as Resolved.Feb 18 2022, 11:23 PM
Papaul updated the task description. (Show Details)
Papaul updated the task description. (Show Details)

@elukey complete

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits