Page MenuHomePhabricator
Create Task
Maniphest T405406

Q2:rack/setup/install dse-k8s-worker200[45]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of dse-k8s-worker200[45]

Hostname / Racking / Installation Details

Hostnames: dse-k8s-worker200[45]
Racking Proposal: We already have three workers in codfw rows B, C, D - so the ideal would be any two discrete rows from A, E, and F.
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private
OS Distro: Bookworm
Boot Method: UEFI. (Based on a preference expressed by Infrastructure-Foundations for all Supermicro servers to use UEFI)
Sub-team Technical Contact: @BTullis

Per host setup checklist

dse-k8s-worker2004
  • Receive in system on procurement task T404775 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
dse-k8s-worker2005
  • Receive in system on procurement task T404775 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add site.pp and preseed.yaml information for dse-k8s-worker200[4-5]operations/puppetproduction+13 -7
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedJhancock.wmT405406 Q2:rack/setup/install dse-k8s-worker200[45]

Event Timeline

RobH created this task.Sep 23 2025, 8:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 23 2025, 8:31 PM
RobH renamed this task from Q2:rack/setup/install dse-k8s-worker200[4-5] to Q2:rack/setup/install dse-k8s-worker200[45].Sep 23 2025, 8:31 PM
RobH assigned this task to BTullis.
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH updated the task description. (Show Details)
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

@BTullis,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

Maintenance_bot added a project: SRE.Sep 23 2025, 9:29 PM
Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE.Sep 26 2025, 2:10 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.
gerritbot added a comment.Sep 29 2025, 10:37 AM

Change #1192091 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add site.pp and preseed.yaml information for dse-k8s-worker200[4-5]

https://gerrit.wikimedia.org/r/1192091

gerritbot added a project: Patch-For-Review.Sep 29 2025, 10:37 AM
gerritbot added a comment.Sep 29 2025, 11:10 AM

Change #1192091 merged by Btullis:

[operations/puppet@production] Add site.pp and preseed.yaml information for dse-k8s-worker200[4-5]

https://gerrit.wikimedia.org/r/1192091

BTullis removed BTullis as the assignee of this task.Sep 29 2025, 11:30 AM
BTullis moved this task from Backlog - operations to Done on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.

Should be good to go. Thanks.

Maintenance_bot removed a project: Patch-For-Review.Sep 29 2025, 11:33 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 3 2025, 7:33 AM
Gehel added a project: Essential-Work.Oct 7 2025, 8:51 AM
Jhancock.wm updated the task description. (Show Details)Dec 10 2025, 9:06 PM
Jhancock.wm mentioned this in T412458: root user not on newest batches of supermicro servers..Dec 11 2025, 9:09 PM
ops-monitoring-bot added a comment.Dec 11 2025, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host dse-k8s-worker2004.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 11 2025, 10:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host dse-k8s-worker2005.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 11 2025, 10:40 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host dse-k8s-worker2004.codfw.wmnet with OS bookworm completed:

  • dse-k8s-worker2004 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512112219_jhancock_3621874_dse-k8s-worker2004.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm updated the task description. (Show Details)EditedDec 11 2025, 10:40 PM
Jhancock.wm subscribed.

need to check the dac in the morning on 2005. no media detected on reimage. whomp

ops-monitoring-bot added a comment.Dec 11 2025, 11:25 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host dse-k8s-worker2005.codfw.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker2005 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-worker2005.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Dec 12 2025, 5:16 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host dse-k8s-worker2005.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Dec 12 2025, 7:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host dse-k8s-worker2005.codfw.wmnet with OS bookworm completed:

  • dse-k8s-worker2005 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202512121730_jhancock_4105277_dse-k8s-worker2005.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jhancock.wm closed this task as Resolved.Dec 12 2025, 7:36 PM
Jhancock.wm claimed this task.
Jhancock.wm updated the task description. (Show Details)

@BTullis these are completed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits