Page MenuHomePhabricator

Ensure cloudvirt capacity is more evenly spread out among racks
Open, Needs TriagePublic

Description

In order to ensure we are properly rack-redundant we need to make sure cloudvirt capacity is spread out evenly across racks. This is my understanding of the situation today, according to data gathered from openstack APIs. (I'll be sending out for review the cookbook I used)

RackHostsVMsvCPU used/totalvCPU%RAM used/totalRAM%
D58154867/576151%2053.0/4028.1 GiB51%
E4143451504/952158%3318.0/7044.1 GiB47%
F4153621703/1016168%3943.0/7547.5 GiB52%

I don't know the full history here of why C doesn't have any cloudvirts; though shuffling capacity to rack C will ensure we can drain a full rack and still have reasonable headroom in the remaining racks, including taking into consideration {T412418} which will refresh cloudvirt104[0-6] (72core / 512GB ram) with 4 hosts (128core / 1TB ram).

There are a few possibilities on the new hardware allocation, though I think the new hosts should be spread out across racks for sure. The hosts to be refreshed are all in D so there will be moves to do from other racks, in the order of 15 moves if we chose to do it and achieve more or less balanced RAM spread.

Below is a table showing the final hardware allocation I (Filippo) am suggesting, with the refreshed cloudvirts (104[0-6]) already out of the picture

RackHostsHost ListvCPU TotalRAM Total
C88new(1077), 1048, 1049, 1051, 1054, 1055, 1056, 10576324608 GiB
D58new(1078), 1047, 1065, 1066, 1067, 1072, 1073, 10746324608 GiB
E49new(1079), 1062, 1063, 1064, 1068, 1069, 1070, 1071, 10756405120 GiB
F49new(1080), 1050, 1052, 1053, 1058, 1059, 1060, 1061, 10767125120 GiB
Tot34261619456 GiB

And a list of moves to get there, including new installs

StepHostsFromTo
11077,1078,1079,1080C8,D5,E4,F4 (new installs)
21048,1049,1051,1054,1055,1056,1057E4C8
31065,1066,1067,1072,1073,1074F4D5
41075F4E4
51040,1041,1042,1043,1044,1045,1046D5decommission

Event Timeline

A minor subtask of this is to add some kind of annotation to the grafana dashboard reminding us that 66% ram usage is actually 100% ram usage

I started to map all this out in a spreadsheet:

https://docs.google.com/spreadsheets/d/15z4PMNhZ1l3oCbUeuEfbDnXRYerHibe7VUDjzFzAoac/edit?gid=220154813#gid=220154813

I started to include ceph OSD shuffling on that doc as well, but it really depends on the timing of T424871.

I don't' have access

Sorry -- is that better now?

@fgiunchedi I was curious to know if there is an estimated date for these moves? Will we be doing it one server at a time.

@fgiunchedi I was curious to know if there is an estimated date for these moves? Will we be doing it one server at a time.

I would like to start next week, I'm going to take back this task and have opened rack-specific subtasks assigned to you like T431682: Rebalance cloudvirts out of E4

What day/time would work best ?