Page MenuHomePhabricator
Create Task
Maniphest T423314

Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8])
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of wdqs103[6-8]

Update: We have decided to rename these servers before bringing them into service.
The new names will be dse-k8s-wdqs100[1-3]

Hostname / Racking / Installation Details

Hostnames: wdqs103[6-8] dse-k8s-wdqs100[1-3]
Racking Proposal: Each on their own row, preference towards rows A and B if they have 10G. Any row is fine though.
Networking Setup: # of Connections:1/2 - 1 Speed:1G/10G. - 10G VLAN:Private/Public/Other(Specify) : Public
OS Distro: Bookworm
Boot Method: UEFI
Sub-team Technical Contact: Brian King (IRC: inflatador) or Ryan Kemper (IRC: ryankemper)

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

dse-k8s-wdqs-1001 (wdqs1036)
  • Receive in system on procurement task T422726 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
dse-k8s-wdqs1001 (wdqs1037)
  • Receive in system on procurement task T422726 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
dse-k8s-wdqs1003 (wdqs1038)
  • Receive in system on procurement task T422726 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
wdqs: Add config for net-new wdqs hostsBkingoperations/puppetproduction+8 -1
Bring dse-k8s-wdqs100[1-3] into serviceBtullisoperations/puppetproduction+8 -1
Update the storage configuration for the new dse-k8s-wdqs nodesBtullisoperations/puppetproduction+21 -12
Fix the hostnames for dse-k8s-wdqs100[1-3]Btullisoperations/puppetproduction+1 -1
Add the new dse-k8s-wdqs nodes to site.pp and preseed.yamlBtullisoperations/puppetproduction+17 -5
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Apr 14 2026, 4:23 PM
RobH assigned this task to bking.
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH mentioned this in Unknown Object (Task).Apr 14 2026, 4:24 PM
RobH added a parent task: Unknown Object (Task).
RobH unsubscribed.
Gehel edited projects, added: Data-Platform-SRE (2026-04-24 - 2026-05-15); removed: Data-Platform-SRE (2026-03-27 - 2026-04-17).Apr 24 2026, 9:17 AM
Jclark-ctr added subscribers: RKemper, Jclark-ctr.May 19 2026, 12:10 PM

i see this is assigned to @bking but @RKemper is listed as contact. The servers have arrived Please update puppet so we will not be blocked

Jclark-ctr updated the task description. (Show Details)May 19 2026, 9:25 PM
Jclark-ctr updated the task description. (Show Details)May 19 2026, 9:42 PM

wdqs1037 is failing to provision will check cabling next time on site

bking moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-04-24 - 2026-05-15) board.May 19 2026, 9:56 PM

Moving to in progress per IRC conversation with @Jclark-ctr . These are our first servers with the hardware profile Config F single CPU, but with smaller NVME drives (2x 2TB). Traffic's cp (CDN) servers use almost the same profile, so we should be able to borrow their partman config. I'm checking now and should have a patch up within the next day or so.

gerritbot added a comment.May 19 2026, 10:10 PM

Change #1289428 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] wdqs: Add config for net-new wdqs hosts

https://gerrit.wikimedia.org/r/1289428

gerritbot added a project: Patch-For-Review.May 19 2026, 10:10 PM
gmodena subscribed.May 21 2026, 12:54 PM
BTullis subscribed.May 22 2026, 3:55 PM

Hello. Apologies for the trouble, but please could we rename these hosts before proceeding?

  • wdqs1036 -> dse-k8s-wdqs1001
  • wdqs1037 -> dse-k8s-wdqs1002
  • wdqs1038 -> dse-k8s-wdqs1003

We're planning to start running the wdqs backend db under kubernetes now, so they will have a different role from all of the existing wdqs servers and a new hostname prefix is appropriate.

I will update site.pp and preseed.yaml with the new names now.
I have added the new prefix to https://wikitech.wikimedia.org/wiki/SRE/Infrastructure_naming_conventions#Hostname_prefixes

BTullis renamed this task from Q4:rack/setup/install wdqs103[6-8] to Q4:rack/setup/install dse-k8s-wdqs100[1-3] (formerly wdqs103[6-8]).May 22 2026, 3:58 PM
BTullis updated the task description. (Show Details)
gerritbot added a comment.May 22 2026, 4:25 PM

Change #1292045 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add the new dse-k8s-wdqs nodes to site.pp and preseed.yaml

https://gerrit.wikimedia.org/r/1292045

Jclark-ctr added a comment.May 27 2026, 12:23 PM

I have updated server names, switchports and provisioned servers. pending puppet being updated @BTullis

Jclark-ctr added a comment.May 28 2026, 3:32 PM

@bking @RKemper can puppet be updated for new host names?

gerritbot added a comment.Jun 2 2026, 11:19 AM

Change #1292045 merged by Btullis:

[operations/puppet@production] Add the new dse-k8s-wdqs nodes to site.pp and preseed.yaml

https://gerrit.wikimedia.org/r/1292045

Jclark-ctr added a comment.Jun 2 2026, 12:07 PM

Dedicated dse-k8s workers for production WDQS in codfw - See #T425653

node /^dse-k8s-wdqs200[1-4]\.codfw\./ {

role(insetup::data_platform_ferm)

}

Dedicated dse-k8s workers for production WDQS in eqiad - See #T425653

node /^dse-k8s-wdqs-100[1-3]\.eqiad\./ {

role(insetup::data_platform_ferm)

}

@BTullis you have an extra - in site.pp file. you have wdqs-100[1-3] instead of wdqs100[1-3]

gerritbot added a comment.Jun 2 2026, 12:21 PM

Change #1296564 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Fix the hostnames for dse-k8s-wdqs100[1-3]

https://gerrit.wikimedia.org/r/1296564

gerritbot added a comment.Jun 2 2026, 12:22 PM

Change #1296564 merged by Btullis:

[operations/puppet@production] Fix the hostnames for dse-k8s-wdqs100[1-3]

https://gerrit.wikimedia.org/r/1296564

ops-monitoring-bot added a comment.Jun 2 2026, 1:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host dse-k8s-wdqs1001.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 2 2026, 1:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Jun 2 2026, 1:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie

Jclark-ctr added a comment.Jun 2 2026, 1:26 PM

Screenshot 2026-06-02 at 9.25.28 AM.png (1,096×608 px, 201 KB)
These are Failing to image for preseed file

ops-monitoring-bot added a comment.Jun 2 2026, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host dse-k8s-wdqs1003.eqiad.wmnet with OS trixie executed with errors:

  • dse-k8s-wdqs1003 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-wdqs1003.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Jun 2 2026, 1:35 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host dse-k8s-wdqs1002.eqiad.wmnet with OS trixie executed with errors:

  • dse-k8s-wdqs1002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-wdqs1002.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
BTullis mentioned this in T425653: Add the wdqs::alternative nodes to the dse-k8s clusters - with a taint to avoid normal jobs being scheduled.Jun 4 2026, 12:32 PM
Gehel edited projects, added: Data-Platform-SRE (2026-06-05 - 2026-06-26); removed: Data-Platform-SRE (2026-04-24 - 2026-05-15).Jun 5 2026, 1:27 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Jclark-ctr mentioned this in T427852: hw troubleshooting: CPU1 thermal fault for wdqs1015.eqiad.wmnet.Jun 8 2026, 11:39 AM
Jclark-ctr mentioned this in T428582: decommission wdqs1015.eqiad.wmnet.Jun 9 2026, 11:44 AM
BTullis added a parent task: Restricted Task.Jun 10 2026, 9:30 AM
BTullis claimed this task.Jun 10 2026, 9:33 AM
BTullis added a subscriber: bking.

I'll take this task, along with T423312, since there is some work on partman required, before the reimage cookbook will work.

BTullis mentioned this in T423312: Q4:rack/setup/install dse-k8s-wdqs200[1-4] (formerly wdqs20[28-31]).Jun 10 2026, 9:34 AM
Jclark-ctr moved this task from Racking Tasks to Blocked on the ops-eqiad board.Tue, Jun 16, 11:59 AM
gerritbot added a comment.Tue, Jun 16, 7:24 PM

Change #1302945 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Update the storage configuration for the new dse-k8s-wdqs nodes

https://gerrit.wikimedia.org/r/1302945

gerritbot added a comment.Tue, Jun 16, 7:38 PM

Change #1302945 merged by Btullis:

[operations/puppet@production] Update the storage configuration for the new dse-k8s-wdqs nodes

https://gerrit.wikimedia.org/r/1302945

ops-monitoring-bot added a comment.Wed, Jun 17, 9:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, Jun 17, 9:39 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm executed with errors:

  • dse-k8s-wdqs1002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console dse-k8s-wdqs1002.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.Wed, Jun 17, 9:48 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, Jun 17, 10:28 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host dse-k8s-wdqs1002.eqiad.wmnet with OS bookworm completed:

  • dse-k8s-wdqs1002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606171004_btullis_420613_dse-k8s-wdqs1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
ops-monitoring-bot added a comment.Wed, Jun 17, 10:31 AM

Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1003 for host dse-k8s-wdqs1003.eqiad.wmnet with OS bookworm

ops-monitoring-bot added a comment.Wed, Jun 17, 11:12 AM

Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1003 for host dse-k8s-wdqs1003.eqiad.wmnet with OS bookworm completed:

  • dse-k8s-wdqs1003 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh bookworm OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202606171050_btullis_430737_dse-k8s-wdqs1003.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
BTullis closed this task as Resolved.Wed, Jun 17, 11:25 AM
BTullis updated the task description. (Show Details)
BTullis moved this task from In Progress to Done on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.
Jclark-ctr claimed this task.Wed, Jun 17, 12:02 PM
gerritbot added a comment.Wed, Jun 17, 12:06 PM

Change #1303406 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Bring dse-k8s-wdqs100[1-3] into service

https://gerrit.wikimedia.org/r/1303406

gerritbot added a comment.Wed, Jun 17, 1:56 PM

Change #1303406 merged by Btullis:

[operations/puppet@production] Bring dse-k8s-wdqs100[1-3] into service

https://gerrit.wikimedia.org/r/1303406

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2026-06-05 - 2026-06-26) board.Fri, Jun 19, 7:28 AM
gerritbot added a comment.Mon, Jun 29, 10:23 PM

Change #1289428 abandoned by Btullis:

[operations/puppet@production] wdqs: Add config for net-new wdqs hosts

Reason:

We have renamed the servers and are using them as dse-k8s workers.

https://gerrit.wikimedia.org/r/1289428

Maintenance_bot removed a project: Patch-For-Review.Mon, Jun 29, 10:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits