Page MenuHomePhabricator
Create Task
Maniphest T399778

Q1:rack/setup/install dse-k8s-worker2003
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of X

Hostname / Racking / Installation Details

Hostnames: dse-k8s-worker2003
Racking Proposal: Any row except C or D (other dse-k8s-workers are in that row)
Networking Setup: 1x 10G on private VLAN
OS Distro: Bookworm (default unless otherwise specified)
Boot Method: UEFI
Sub-team Technical Contact: @bking (inflatador on IRC) or @BTullis (btullis on IRC)

Per host setup checklist

dse-k8s-worker2003:
  • Receive in system on procurement task T399104 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Other Assignee
bking
Related Changes in Gerrit:
SubjectRepoBranchLines +/-
dse-k8s-worker2003: return to production roleoperations/puppetproduction+3 -3
dse-k8s-worker2003: move back to insetupoperations/puppetproduction+2 -2
Add dse-k8s-worker2003 to site.ppoperations/puppetproduction+2 -2
preseed: fix partman config for dse-k8s-worker2003operations/puppetproduction+2 -2
Add dse-k8s-worker2003 to preseed.yamloperations/puppetproduction+4 -0
site: assign the insetup::data_platform_ferm role to dse-k8s-worker1014operations/puppetproduction+7 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedbkingT399778 Q1:rack/setup/install dse-k8s-worker2003
 ResolvedbkingT406656 Reimage failed after prompt...is prompt needed?

Event Timeline

RobH created this task.Jul 16 2025, 10:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 16 2025, 10:21 PM
RobH assigned this task to bking.Jul 16 2025, 10:21 PM
RobH mentioned this in Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

@bking or @BTullis,

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH added a parent task: Unknown Object (Task).Jul 16 2025, 10:23 PM
Maintenance_bot added a project: SRE.Jul 16 2025, 10:29 PM
gerritbot added a comment.Jul 17 2025, 9:19 AM

Change #1170279 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] site: assign the insetup::data_platform_ferm role to dse-k8s-worker1014

https://gerrit.wikimedia.org/r/1170279

gerritbot added a project: Patch-For-Review.Jul 17 2025, 9:19 AM
gerritbot added a comment.Jul 18 2025, 8:24 AM

Change #1170279 merged by Brouberol:

[operations/puppet@production] site: assign the insetup::data_platform_ferm role to dse-k8s-worker1014

https://gerrit.wikimedia.org/r/1170279

Maintenance_bot removed a project: Patch-For-Review.Jul 18 2025, 8:31 AM
Jhancock.wm subscribed.Jul 28 2025, 3:36 PM

note to myself. These must be racked in DH7 cage.

BTullis moved this task from Incoming to Scratch on the Data-Platform-SRE board.Jul 28 2025, 5:24 PM
BTullis edited projects, added Data-Platform-SRE (2025.07.26 - 2025.08.15); removed Data-Platform-SRE.Jul 28 2025, 5:27 PM
BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
BTullis removed bking as the assignee of this task.Aug 7 2025, 3:17 PM
BTullis moved this task from Backlog - operations to Done on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.Aug 20 2025, 10:32 AM
Jhancock.wm updated the task description. (Show Details)Sep 3 2025, 1:17 PM
gerritbot added a comment.Sep 12 2025, 12:21 PM

Change #1187788 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] preseed: fix partman config for dse-k8s-worker2003

https://gerrit.wikimedia.org/r/1187788

gerritbot added a project: Patch-For-Review.Sep 12 2025, 12:21 PM

Change #1187789 had a related patch set uploaded (by Jclark-ctr; author: jclark):

[operations/puppet@production] Add dse-k8s-worker2003 to preseed.yaml

https://gerrit.wikimedia.org/r/1187789

gerritbot added a comment.Sep 12 2025, 12:26 PM

Change #1187789 abandoned by Jclark-ctr:

[operations/puppet@production] Add dse-k8s-worker2003 to preseed.yaml

Reason:

Duplicate to 1187788

https://gerrit.wikimedia.org/r/1187789

gerritbot added a comment.Sep 12 2025, 1:05 PM

Change #1187788 merged by Elukey:

[operations/puppet@production] preseed: fix partman config for dse-k8s-worker2003

https://gerrit.wikimedia.org/r/1187788

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2025, 1:31 PM
Jhancock.wm updated the task description. (Show Details)Sep 17 2025, 2:05 PM
ops-monitoring-bot added a comment.Sep 17 2025, 8:49 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm

Jhancock.wm added a comment.Sep 17 2025, 9:19 PM

[28/50, retrying in 84.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb.<locals>.poll_puppetdb' raised: Nagios_host resource with title dse-k8s-worker2003 not found yet

going to fail. issue with the puppet? need to investigate.

ops-monitoring-bot added a comment.Sep 17 2025, 10:00 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-worker2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 25 2025, 2:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm

Jhancock.wm added a comment.Sep 25 2025, 3:18 PM

@bking can you check the site.pp and preseed.yaml files for accuracy? the reimage cookbook is acting like there's a possible misconfig there. thank you!

bking claimed this task.EditedSep 25 2025, 3:28 PM
bking edited projects, added Data-Platform-SRE (2025.09.05 - 2025.09.26); removed Data-Platform-SRE (2025.07.26 - 2025.08.15).

@Jhancock.wm I think we had a similar ticket for the same hardware in EQIAD (T399779) . I'll take a look there and see if we missed anything.

gerritbot added a comment.Sep 25 2025, 3:50 PM

Change #1191441 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] Add dse-k8s-worker2003 to site.pp

https://gerrit.wikimedia.org/r/1191441

gerritbot added a project: Patch-For-Review.Sep 25 2025, 3:50 PM
gerritbot added a comment.Sep 25 2025, 3:56 PM

Change #1191441 merged by Bking:

[operations/puppet@production] Add dse-k8s-worker2003 to site.pp

https://gerrit.wikimedia.org/r/1191441

bking added a comment.Sep 25 2025, 4:00 PM

@Jhancock.wm it looks like the host was missing from site.pp. I've added it, and you should be good to go now. Feel free to ping me on IRC (inflatador) if you run into any other issues.

bking reassigned this task from bking to Jhancock.wm.Sep 25 2025, 4:01 PM
bking updated Other Assignee, added: bking.
ops-monitoring-bot added a comment.Sep 25 2025, 4:06 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-worker2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Maintenance_bot removed a project: Patch-For-Review.Sep 25 2025, 4:32 PM
ops-monitoring-bot added a comment.Sep 25 2025, 6:08 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 25 2025, 7:11 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-worker2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM
ops-monitoring-bot added a comment.Sep 29 2025, 4:05 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm

Jhancock.wm added a subscriber: Papaul.Sep 29 2025, 5:07 PM

@Papaul Hey we've gotten the pressed and site.pp files cofigured correctly as far as i can tell but still getting this on the reimage script. Could you take a look for me when you have time? thanks.

[46/50, retrying in 138.00s] Attempt to run 'cookbooks.sre.hosts.reimage.ReimageRunner._populate_puppetdb.<locals>.poll_puppetdb' raised: Nagios_host resource with title dse-k8s-worker2003 not found yet

Jhancock.wm reassigned this task from Jhancock.wm to Papaul.Sep 29 2025, 5:08 PM
ops-monitoring-bot added a comment.Sep 29 2025, 5:17 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-worker2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Sep 29 2025, 11:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm

ops-monitoring-bot added a comment.Sep 30 2025, 12:58 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm executed with errors:

  • dse-k8s-worker2003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-worker2003.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Papaul added a comment.Sep 30 2025, 1:25 AM

@Jhancock.wm see below why the server is failing. You have 2 options change the role int site.pp to insetup role to finish the install or have the server owner fix the puppet error below.

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Operator '[]' is not applicable to an Undef Value. (file: /srv/puppet_code/environments/production/modules/profile/manifests/kubernetes/node.pp, line: 140, column: 15) on node dse-k8s-worker2003.codfw.wmnet
Jhancock.wm reassigned this task from Papaul to bking.Sep 30 2025, 1:45 AM

can you help me out with what papaul pointed out when you get in? thanks!

Gehel added a project: Essential-Work.Oct 7 2025, 8:52 AM
bking added a comment.Oct 7 2025, 3:33 PM

Grabbing the ticket per IRC conversation with @Jhancock.wm

gerritbot added a comment.Oct 7 2025, 3:35 PM

Change #1194231 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] dse-k8s-worker2003: move back to insetup

https://gerrit.wikimedia.org/r/1194231

gerritbot added a project: Patch-For-Review.Oct 7 2025, 3:35 PM

Change #1194231 merged by Bking:

[operations/puppet@production] dse-k8s-worker2003: move back to insetup

https://gerrit.wikimedia.org/r/1194231

ops-monitoring-bot added a comment.Oct 7 2025, 3:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Oct 7 2025, 4:31 PM
ops-monitoring-bot added a comment.Oct 7 2025, 8:50 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm completed:

  • dse-k8s-worker2003 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510071557_bking_529588_dse-k8s-worker2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Stashbot added a comment.Oct 7 2025, 10:11 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T22:11:48Z] <bking@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "running per cookbook error suggestion - bking@cumin2002 - T399778"

Stashbot added a comment.Oct 7 2025, 10:12 PM

Mentioned in SAL (#wikimedia-operations) [2025-10-07T22:12:08Z] <bking@cumin2002> END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "running per cookbook error suggestion - bking@cumin2002 - T399778"

gerritbot added a comment.Oct 7 2025, 10:15 PM

Change #1194305 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] dse-k8s-worker2003: return to production role

https://gerrit.wikimedia.org/r/1194305

gerritbot added a project: Patch-For-Review.Oct 7 2025, 10:15 PM
bking added a comment.Oct 7 2025, 10:16 PM

I verified that this host is ready for production, so no need to worry about the above cookbook failure.

gerritbot added a comment.Oct 9 2025, 8:00 PM

Change #1194305 merged by Bking:

[operations/puppet@production] dse-k8s-worker2003: return to production role

https://gerrit.wikimedia.org/r/1194305

ops-monitoring-bot added a comment.Oct 9 2025, 8:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by bking@cumin2002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2025, 8:31 PM
ops-monitoring-bot added a comment.Oct 9 2025, 8:42 PM

Cookbook cookbooks.sre.hosts.reimage started by bking@cumin2002 for host dse-k8s-worker2003.codfw.wmnet with OS bookworm completed:

  • dse-k8s-worker2003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Add puppet_version metadata (7) to Debian installer
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202510092023_bking_2138859_dse-k8s-worker2003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
bking closed this task as Resolved.Oct 10 2025, 1:16 PM
bking moved this task from Backlog - project to Done on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.

This host has been reimaged as a dse-k8s worker, so I'm closing this out. Work to make this host ready for production continues in T406985...

elukey closed subtask T406656: Reimage failed after prompt...is prompt needed? as Declined.Oct 13 2025, 1:52 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.09.26 - 2025.10.17) board.Oct 17 2025, 8:30 AM
bking reopened subtask T406656: Reimage failed after prompt...is prompt needed? as Open.Oct 17 2025, 3:09 PM
bking closed subtask T406656: Reimage failed after prompt...is prompt needed? as Resolved.Dec 2 2025, 6:59 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits