Page MenuHomePhabricator
Create Task
Maniphest T419892

Q3:rack/setup/install cloudcephosd105[3456]
Open, MediumPublic
Actions

Description

This task will track the racking, setup, and OS installation of cloudcephosd105[3456]

Hostname / Racking / Installation Details

Hostnames: cloudcephosd105[56]
Racking Proposal: Both servers in D5
Networking Setup: # of Connections:1 - Speed:25G. - VLAN: attach to cloud-hosts1-eqiad
OS Distro: Bookworm
Boot Method: UEFI
Sub-team Technical Contact: @Andrew

Per host setup checklist

cloudcephosd1053
  • Receive in system on procurement task T412570 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
cloudcephosd1054
  • Receive in system on procurement task T412569 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
cloudcephosd1055
  • Receive in system on procurement task T412728 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook
cloudcephosd1056
  • Receive in system on procurement task T412728 & in Coupa
  • Rack system with proposed racking plan (see above) & update Netbox (include all system info plus location, state of planned)
  • Run the Provision a server's network attributes Netbox script - Note that you must run the DNS and Provision cookbook after completing this step
  • Immediately run the sre.dns.netbox cookbook
  • Immediately run the sre.hosts.provision cookbook
  • Run the sre.hardware.upgrade-firmware cookbook
  • Update the operations/puppet repo - this should include updates to preseed.yaml, and site.pp with roles defined by service group: https://wikitech.wikimedia.org/wiki/SRE/Dc-operations
  • Run the sre.hosts.reimage cookbook

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Initial entries for cloudcephosd105[3-6]Andrew Bogottoperations/puppetproduction+7 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 OpenfgiunchediT419892 Q3:rack/setup/install cloudcephosd105[3456]

Event Timeline

RobH created this task.Mar 12 2026, 6:44 PM
RobH renamed this task from Q#:rack/setup/install X to Q3:rack/setup/install cloudcephosd105[56].
RobH updated the task description. (Show Details)
RobH mentioned this in Unknown Object (Task).
RobH added a parent task: Unknown Object (Task).
RobH assigned this task to Andrew.Mar 12 2026, 6:47 PM
RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

Andrew,

Please note there are (3) racking tasks for the (3) orders of cloudcephosd hosts in eqiad that have just been placed. Assumptions have been made about hostname, simply picking the next in sequence. Please review each of these racking tasks and update the racking info for network and rack placement in addition to the boilderplate below:

Please update the site.pp file with the insetup role for your team (detailed on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations) and add the new servers to preseed.yml for partition info.

If possible, please reference this task number in your patch set, so it is clear when complete. Once complete, just un-assign yourself (leaving no assignee) for this task and once the hardware arrives on-site engineerss will claim this task for racking and setup. Please don't re-subscribe me to this task unless there is a direct question for me.

Thank you!

The 3 tasks are as follows:

T416394 cloudcephosd1053

T416395 cloudcephosd1054

T419892 cloudcephosd105[56]

RobH mentioned this in T416394: Q3:rack/setup/install cloudcephosd1053.Mar 12 2026, 6:47 PM
RobH mentioned this in T416395: Q3:rack/setup/install cloudcephosd1054.
RobH unsubscribed.
Andrew updated the task description. (Show Details)Mar 20 2026, 2:07 PM
Andrew updated the task description. (Show Details)Mar 20 2026, 2:10 PM
gerritbot added a comment.Mar 20 2026, 2:20 PM

Change #1256392 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Initial entries for cloudcephosd105[3-6]

https://gerrit.wikimedia.org/r/1256392

gerritbot added a project: Patch-For-Review.Mar 20 2026, 2:20 PM
Andrew added a comment.Mar 20 2026, 2:24 PM

eqiad folks: these hosts are untested hardware with a novel drive configuration. I do not expect partman to work on the first go!

The intended drive setup is:

OS installed on 2-drive mirror raid of the two 480G drives (hw raid if there is a hw raid controller)
The other drives unformatted and unmanaged by the hw raid controller (if there is one)

Feel free to drop this back on me to mess with partman!

gerritbot added a comment.Mar 23 2026, 1:03 PM

Change #1256392 merged by Andrew Bogott:

[operations/puppet@production] Initial entries for cloudcephosd105[3-6]

https://gerrit.wikimedia.org/r/1256392

Andrew removed Andrew as the assignee of this task.Mar 23 2026, 1:04 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 23 2026, 1:31 PM
Jclark-ctr claimed this task.Apr 28 2026, 4:13 PM
Jclark-ctr added a subscriber: • ayounsi.Apr 28 2026, 5:04 PM

@Andrew This server can’t go in D5 that switch only supports 1G/10G.
@ayounsi Can you confirm I’m right? I believe the only WMCS racks that support 25G are E4 and F4, Also they’re getting very full power-wise.

Jclark-ctr renamed this task from Q3:rack/setup/install cloudcephosd105[56] to Q3:rack/setup/install cloudcephosd105[3456].May 1 2026, 2:22 PM
Jclark-ctr updated the task description. (Show Details)
Jclark-ctr moved this task from Racking Tasks to Blocked on the ops-eqiad board.Jun 16 2026, 11:59 AM
Jclark-ctr removed Jclark-ctr as the assignee of this task.Jun 22 2026, 5:48 PM
Jclark-ctr subscribed.
Jclark-ctr assigned this task to Andrew.Jun 24 2026, 3:01 PM

Is there a ticket tracking the rebalancing of CloudVirts for power to make room? Has any decision been made regarding moves, or are we waiting until the refresh switches arrive?

Jclark-ctr added a subscriber: fgiunchedi.Jun 24 2026, 3:05 PM
fgiunchedi added a comment.Jun 24 2026, 3:52 PM

Yes the cloudvirt rebalancing task is T424658: Ensure cloudvirt capacity is more evenly spread out among racks and the last table in the description lists the host moves to get to balance. I'll be taking on provisioning the new cloudvirts (T429563) first thing to have some more room to move VMs around. With respect to existing cloudvirt hosts moves we can certainly prioritize E4/F4 if that'll make things easier (as opposed to decom D5 cloudvirts).

For the new switches to be honest I don't remember what/if we decided

• ayounsi added a comment.Jun 24 2026, 6:47 PM

{T424871} are the switch replacement tracking task to support 25G. But even after they arrive some time will be needed to configure/automate them as they're a new vendor.

fgiunchedi added a comment.Jun 25 2026, 6:26 AM

Good find @ayounsi ! Also with respect to racking these hosts, my preference would be to have one host per rack once we can do 25G across all cloud racks

fgiunchedi claimed this task.Jun 25 2026, 3:52 PM

Taking this on, I'll re-assign as needed once we have a path forward

fgiunchedi added a comment.Jun 29 2026, 10:15 AM

Would we have capacity (power, space) to move two hosts to their final allocation in E4/F4 ?

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits