Page MenuHomePhabricator
Create Task
Maniphest T418916

Q3:rack/setup/install rdb101[56]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of rdb101[56]

Hostname / Racking / Installation Details

Hostnames: rdb101[56]
Racking Proposal: Two different rows excluding B and D
Networking Setup: # of Connections:1 - Speed:10G. - VLAN:Private
OS Distro: Trixie
Boot Method: UEFI.
Sub-team Technical Contact: @Clement_Goubert

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

rdb1015:
  • Receive in system on procurement task T418292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
rdb1016:
  • Receive in system on procurement task T418292 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Fix partman to use standard-efi for new rdb nodesoperations/puppetproduction+2 -2
Add new rdb101[56] hostsoperations/puppetproduction+9 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

RobH created this task.Mar 3 2026, 7:29 PM
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH assigned this task to Clement_Goubert.Mar 3 2026, 7:31 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.
RobH unsubscribed.

@Clement_Goubert,

I have made some assumptions in racking details. The parent order is already in approvals, this isn't blocking it. I've filed this to ensure it isn't forgotten. Please double check my hostname and other assumptions above and clarify racking restrictions (if any) in addition to the boilerplate:

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

RobH created subtask T418918: rdb101[56] implementation tracking.Mar 3 2026, 7:32 PM
Clement_Goubert moved this task from Inbox to In Progress on the ServiceOps new board.Mar 4 2026, 10:49 AM
Clement_Goubert updated the task description. (Show Details)Mar 4 2026, 11:44 AM
gerritbot added a comment.Mar 4 2026, 11:46 AM

Change #1247977 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] Add new rdb101[56] hosts

https://gerrit.wikimedia.org/r/1247977

gerritbot added a project: Patch-For-Review.Mar 4 2026, 11:46 AM
Clement_Goubert added a comment.Mar 4 2026, 11:46 AM

Updated task description with racking details and OS. Waiting for review on the puppet patch.

Raine changed the status of subtask T418918: rdb101[56] implementation tracking from Open to Stalled.Mar 4 2026, 12:30 PM
gerritbot added a comment.Mar 4 2026, 12:43 PM

Change #1247989 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] Add new wikikube-worker23[57-74]

https://gerrit.wikimedia.org/r/1247989

gerritbot added a comment.Mar 9 2026, 2:32 PM

Change #1247977 merged by Clément Goubert:

[operations/puppet@production] Add new rdb101[56] hosts

https://gerrit.wikimedia.org/r/1247977

Clement_Goubert removed Clement_Goubert as the assignee of this task.Mar 9 2026, 2:34 PM
Clement_Goubert moved this task from In Progress to Radar (Awareness) on the ServiceOps new board.
Jclark-ctr claimed this task.Apr 21 2026, 8:02 PM
ops-monitoring-bot added a comment.Apr 21 2026, 9:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host rdb1015.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Apr 21 2026, 9:29 PM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host rdb1016.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Apr 21 2026, 9:47 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host rdb1016.eqiad.wmnet with OS trixie executed with errors:

  • rdb1016 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console rdb1016.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr updated the task description. (Show Details)Apr 21 2026, 9:50 PM

@Clement_Goubert i am having issues with these failing to image this is error on console. They might be missing a partman

│ Error while setting up RAID │
│ An unexpected error occurred while setting up a preseeded RAID │
│ configuration. │
│ Check /var/log/syslog or see virtual console 4 for the details.│

ops-monitoring-bot added a comment.Apr 21 2026, 10:31 PM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host rdb1015.eqiad.wmnet with OS trixie executed with errors:

  • rdb1015 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • The reimage failed, see the cookbook logs for the details. You can also try typing "sudo install-console rdb1015.eqiad.wmnet" to get a root shell, but depending on the failure this may not work.
Jclark-ctr updated the task description. (Show Details)Apr 24 2026, 12:53 PM
MLechvien-WMF added subscribers: jijiki, MLechvien-WMF.Apr 27 2026, 8:36 AM

@Clement_Goubert i am having issues with these failing to image this is error on console. They might be missing a partman

@jijiki could you please help on this while Clement is out?

jijiki added a comment.Apr 27 2026, 9:03 AM
In T418916#11859537, @MLechvien-WMF wrote:

@Clement_Goubert i am having issues with these failing to image this is error on console. They might be missing a partman

@jijiki could you please help on this while Clement is out?

I can take care of it next week and update. @Jclark-ctr thank you!

Jclark-ctr mentioned this in T418922: Q3:rack/setup/install rdb201[34].Apr 30 2026, 2:27 PM
jijiki added a comment.May 6 2026, 2:31 PM
In T418916#11859808, @jijiki wrote:
In T418916#11859537, @MLechvien-WMF wrote:

@Clement_Goubert i am having issues with these failing to image this is error on console. They might be missing a partman

@jijiki could you please help on this while Clement is out?

I can take care of it next week and update. @Jclark-ctr thank you!

@Jclark-ctl I am afraid this is still in the queue, will ping you when I have something, thanks!

Jclark-ctr reassigned this task from Jclark-ctr to Effib.Tue, May 19, 10:07 PM
Jclark-ctr reassigned this task from Effib to jijiki.
Jclark-ctr reassigned this task from jijiki to Clement_Goubert.
Jclark-ctr added a subscriber: Effib.
Jclark-ctr subscribed.
ops-monitoring-bot added a comment.Wed, May 20, 12:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1003 for host rdb1015.eqiad.wmnet with OS trixie

gerritbot added a comment.Wed, May 20, 12:20 AM

Change #1289448 had a related patch set uploaded (by Papaul; author: Papaul):

[operations/puppet@production] Fix partman to use standard-efi for new rdb nodes

https://gerrit.wikimedia.org/r/1289448

gerritbot added a comment.Wed, May 20, 12:23 AM

Change #1289448 merged by Papaul:

[operations/puppet@production] Fix partman to use standard-efi for new rdb nodes

https://gerrit.wikimedia.org/r/1289448

ops-monitoring-bot added a comment.Wed, May 20, 1:30 AM

Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1003 for host rdb1015.eqiad.wmnet with OS trixie completed:

  • rdb1015 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605200054_pt1979_1743822_rdb1015.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Papaul subscribed.Wed, May 20, 1:30 AM

@Jclark-ctr partman fixed 1015 is done you can install 1016. thanks

Papaul updated the task description. (Show Details)Wed, May 20, 1:32 AM
MLechvien-WMF added a comment.Wed, May 20, 8:58 AM

@Jclark-ctr are this and T418922: Q3:rack/setup/install rdb201[34] (which had the same issue IIRC) unblocked now?

Jclark-ctr claimed this task.Wed, May 20, 11:04 AM

@MLechvien-WMF i believe so looks like @Papaul noticed the missing part in puppet and updated both

ops-monitoring-bot added a comment.Wed, May 20, 11:05 AM

Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host rdb1016.eqiad.wmnet with OS trixie

ops-monitoring-bot added a comment.Wed, May 20, 11:40 AM

Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host rdb1016.eqiad.wmnet with OS trixie completed:

  • rdb1016 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced UEFI HTTP Boot for next reboot
    • Host rebooted via Redfish
    • Host up (Debian installer)
    • Host up (new fresh trixie OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202605201123_jclark_1972713_rdb1016.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
Jclark-ctr closed this task as Resolved.Wed, May 20, 11:40 AM
Jclark-ctr updated the task description. (Show Details)
MLechvien-WMF changed the status of subtask T418918: rdb101[56] implementation tracking from Stalled to Open.Fri, May 22, 10:34 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits