Page MenuHomePhabricator

cloud: prepare codfw for expansion (racks, switches, ceph)
Closed, DeclinedPublic

Description

As part of the project in T342750: cloud: introduce a kubernetes undercloud to run openstack (via openstack-helm) we will most likely do a build-out of openstack-in-kubernetes in the codfw datacenter.
However, as of this writing, we only have one rack for WMCS in codfw.

We need to expand our WMCS rack, switch and ceph capacity in codfw before we can do any meaningful deployment in there.

At very least, we need 3 total racks, each in 3 different rows, each with one cloudsw device.
Given we already have one (codfw B1) https://netbox.wikimedia.org/dcim/racks/51/ this means we would need 2 more racks and 2 more cloudsw devices.

The base reason is that we need redundancy at switch/rack/row level, meaning that we should not connect all the services to the same switch/rack/row.

Regarding ceph storage, I'm guessing we would start small and expand as required later.

Event Timeline

aborrero renamed this task from cloud: prepare codfw for expansion (racks, switches) to cloud: prepare codfw for expansion (racks, switches, ceph).Sep 19 2023, 8:53 AM
aborrero updated the task description. (Show Details)

@cmooney I raised a similar question about expanding WMCS racks in eqiad and as I understood the answer was the newer rows in eqiad have compatible ToR switches for cloud use cases. Is that similar to what can be expected in codfw? In other words, are there 1 or 2 more racks we could set aside as WMCS racks in codfw without ordering cloudsw devices?

And, if we do move forward with pushing VXLAN responsibilities to the hosts, would that allow any existing switches in codfw to work as WMCS racks?

@cmooney You predicted this question and answered it already here :-) https://phabricator.wikimedia.org/T346724#9206025 <3 Thank you!

@nskaggs hello true that codfw will me moving to the EVPN/VXLAN design but codfw doesn't have that many racks to dedicate 2 racks for WMCS. We have a total of 32 racks in codfw. 2 network racks, 1 fundraising rack, 1 dedicated WMCS rack and 1 rack that is already full so that put us to 27 racks left. If we have to dedicated anoter 2 racks for WMCS we have 25 racks available and right now all the other racks are getting close to be full. So if we have to go with 2 dedicated racks option for WMCS we need to think about expansion.

Thank you.

@Papaul Is it possible there is 1 more rack that could be dedicated in the current setup (so 2 total WMCS racks, 1 existing + 1 new)? It sounds like space is tight without expansion, so perhaps even 1 additional rack isn't possible.

@nskaggs you are correct even 1 additional rack isnt't possible at this time. Sorry about that.