Page MenuHomePhabricator
Create Task
Maniphest T418922

Q3:rack/setup/install rdb201[34]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of rdb201[34]

Hostname / Racking / Installation Details

Hostnames: rdb201[34]
Racking Proposal: In two different rows excluding rows E and F (rdb2011 and rdb2012 are there)
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private
OS Distro: Trixie
Boot Method: UEFI.
Sub-team Technical Contact: @Clement_Goubert

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

rdb2013
  • Receive in system on procurement task T418291 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
rdb2014
  • Receive in system on procurement task T418291 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Add new rdb201[34] hostsClément Goubertoperations/puppetproduction+5 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Mar 3 2026, 8:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2026, 8:09 PM
RobH assigned this task to Clement_Goubert.Mar 3 2026, 8:09 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH moved this task from Backlog to Racking Tasks on the ops-codfw board.
RobH unsubscribed.

@Clement_Goubert,

Please double check the racking details in the task description as I made some assumptions on hostnames and defaults in addition to the boilerplate:

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH created subtask T418924: rdb201[34] implementation tracking.Mar 3 2026, 8:12 PM
Clement_Goubert moved this task from Inbox to In Progress on the ServiceOps board.Mar 4 2026, 10:49 AM
Clement_Goubert updated the task description. (Show Details)Mar 4 2026, 10:54 AM
gerritbot added a comment.Mar 4 2026, 10:56 AM

Change #1247969 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] site.pp: Add rdb201[34]

https://gerrit.wikimedia.org/r/1247969

gerritbot added a project: Patch-For-Review.Mar 4 2026, 10:56 AM
MoritzMuehlenhoff subscribed.Mar 4 2026, 11:16 AM

@Clement_Goubert The current rdb* hosts are on Bullseye, and you listed Bookworm as the designated OS, if we move to a new OS, let's directly move to Trixie?

gerritbot added a comment.Mar 4 2026, 11:39 AM

Change #1247969 merged by Clément Goubert:

[operations/puppet@production] Add new rdb201[34] hosts

https://gerrit.wikimedia.org/r/1247969

Clement_Goubert added a comment.Mar 4 2026, 11:41 AM
In T418922#11672695, @MoritzMuehlenhoff wrote:

@Clement_Goubert The current rdb* hosts are on Bullseye, and you listed Bookworm as the designated OS, if we move to a new OS, let's directly move to Trixie?

Good point. Updating task description.

Clement_Goubert removed Clement_Goubert as the assignee of this task.Mar 4 2026, 11:41 AM
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert moved this task from In Progress to Radar (Awareness) on the ServiceOps board.

Updated racking details, changed OS to Trixie, puppet patches merged. All yours.

Raine changed the status of subtask T418924: rdb201[34] implementation tracking from Open to Stalled.Mar 4 2026, 12:29 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 4 2026, 12:30 PM
Jhancock.wm claimed this task.Apr 21 2026, 4:59 PM
Jhancock.wm updated the task description. (Show Details)
ops-monitoring-bot added a comment.Apr 23 2026, 11:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host rdb2013.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Apr 23 2026, 11:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host rdb2014.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.Apr 24 2026, 12:05 AM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host rdb2013.codfw.wmnet with OS trixie executed with errors:

  • rdb2013 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console rdb2013.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr added subscribers: jijiki, Jclark-ctr.Apr 29 2026, 2:12 AM

@Jhancock.wm eqiad servers failed install also. @jijiki when you make change can you fix eqiad and codfw?

Jclark-ctr added a comment.Apr 30 2026, 2:27 PM

Talked to @Jhancock.wm same issues with imaging eqiad servers T418916

ops-monitoring-bot added a comment.May 2 2026, 6:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host rdb2013.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.May 2 2026, 6:07 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host rdb2014.codfw.wmnet with OS trixie

jijiki added a comment.May 6 2026, 2:32 PM
In T418922#11869335, @Jclark-ctr wrote:

@Jhancock.wm eqiad servers failed install also. @jijiki when you make change can you fix eqiad and codfw?

sure, will sort out both tasks

MLechvien-WMF mentioned this in T418916: Q3:rack/setup/install rdb101[56].May 20 2026, 8:58 AM
Jhancock.wm updated the task description. (Show Details)May 26 2026, 6:38 PM
ops-monitoring-bot added a comment.May 26 2026, 6:41 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host rdb2014.codfw.wmnet with OS trixie executed with errors:

  • rdb2014 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console rdb2014.codfw.wmnet" to get a root shell, but depending on the failure this may not work.
ops-monitoring-bot added a comment.May 26 2026, 6:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host rdb2013.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.May 26 2026, 6:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host rdb2014.codfw.wmnet with OS trixie

ops-monitoring-bot added a comment.May 26 2026, 7:40 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host rdb2013.codfw.wmnet with OS trixie completed:

  • rdb2013 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605261903_jhancock_1006998_rdb2013.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
ops-monitoring-bot added a comment.May 26 2026, 7:40 PM

Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host rdb2014.codfw.wmnet with OS trixie completed:

  • rdb2014 (WARN)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605261907_jhancock_1007346_rdb2014.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • Failed to run the sre.puppet.sync-netbox-hiera cookbook, run it manually
Jhancock.wm closed this task as Resolved.May 26 2026, 7:41 PM
Jhancock.wm updated the task description. (Show Details)
jijiki changed the status of subtask T418924: rdb201[34] implementation tracking from Stalled to In Progress.May 27 2026, 12:09 PM
jijiki closed subtask T418924: rdb201[34] implementation tracking as Resolved.Jun 9 2026, 9:19 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits