We are going to expand in FQ4 FY2015-2016 with 6x machines in codfw/eqiad. Note this was originally intended as a full refresh of the older 2TB machines, instead we're adding capacity and will refresh older hardware in a year or so.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T130012 expand swift hardware in codfw/eqiad | |||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Unknown Object (Task) | |||||
| Resolved | fgiunchedi | T136630 rack/setup/deploy ms-be202[2-7] | |||
| Resolved | RobH | T138052 codfw: rack/setup/deploy ms-be202[2-7] switch configuration | |||
| Invalid | fgiunchedi | T137439 rack/setup/deploy ms-be202[2-7] | |||
| Unknown Object (Task) | |||||
| Resolved | fgiunchedi | T136631 rack/setup/deploy ms-be102[2-7] | |||
| Resolved | • Cmjohnson | T140374 diagnose failed disks on ms-be1027 | |||
| Resolved | fgiunchedi | T140597 diagnose failed(?) sda on ms-be1022 | |||
| Resolved | None | T141756 audit / test / upgrade hp smartarray P840 firmware |
note that expansion not refresh of swift hw is tracked in {T130713}, though we might be able to batch them together
one of the questions for the next order is 3TB vs 4TB disks, the last order of 3x eqiad and 6x codfw T114500: determine new swift ms-be hostnames (codfw/eqiad) and related was for 4TB.
to gauge the impact of 4TB I've been trying on ms-be1019 to put weight 4000 on Feb 1st
ms-be1019 IO before/after weight 4000
and after that all new 4TB machines in codfw and eqiad have been conservatively switched to weight 3500. On March 23rd I've switched ms-be1020 to weight 3500
ms-be1020 IO before/after weight 3500
also note that 2016-03-10 -> 2016-03-24 there was less load in eqiad due to codfw switchover testing, IOW only thumbs were served from eqiad
for the current refresh we're replacing 12x 2TB machines in eqiad/codfw, each machine has 12x 1.9TB = 22.8TB usable, so 273.6TB and 144 disks per datacenter.
With 3TB machines, that's usable 2.8T * 12 disks = 33.6T per machine, or 9x machines to add up to 302.4T usable and 108 disks per datacenter. In terms of space that would add space for 70d more (we're using 140GB/day * 3x replication) and in terms of number of disks the gap would be filled by an order we're placing for expansion in {T130713}.
tl;dr the refresh should be for 9x 3TB machines (per datacenter)
6x swift systems (all 3TB disks) have been ordered in T130713 and T136336, though we'll be keeping the old swift hw in place for the next 6/9 months as the hardware is just out of warranty. With the old hardware in place, the last 6x order is expected to last ~1y