Page MenuHomePhabricator

expand swift hardware in codfw/eqiad
Closed, ResolvedPublic

Description

We are going to expand in FQ4 FY2015-2016 with 6x machines in codfw/eqiad. Note this was originally intended as a full refresh of the older 2TB machines, instead we're adding capacity and will refresh older hardware in a year or so.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 15 2016, 2:18 PM

note that expansion not refresh of swift hw is tracked in {T130713}, though we might be able to batch them together

one of the questions for the next order is 3TB vs 4TB disks, the last order of 3x eqiad and 6x codfw T114500: determine new swift ms-be hostnames (codfw/eqiad) and related was for 4TB.

to gauge the impact of 4TB I've been trying on ms-be1019 to put weight 4000 on Feb 1st

ms-be1019 IO before/after weight 4000

and after that all new 4TB machines in codfw and eqiad have been conservatively switched to weight 3500. On March 23rd I've switched ms-be1020 to weight 3500

ms-be1020 IO before/after weight 3500

also note that 2016-03-10 -> 2016-03-24 there was less load in eqiad due to codfw switchover testing, IOW only thumbs were served from eqiad

for the current refresh we're replacing 12x 2TB machines in eqiad/codfw, each machine has 12x 1.9TB = 22.8TB usable, so 273.6TB and 144 disks per datacenter.

With 3TB machines, that's usable 2.8T * 12 disks = 33.6T per machine, or 9x machines to add up to 302.4T usable and 108 disks per datacenter. In terms of space that would add space for 70d more (we're using 140GB/day * 3x replication) and in terms of number of disks the gap would be filled by an order we're placing for expansion in {T130713}.

tl;dr the refresh should be for 9x 3TB machines (per datacenter)

fgiunchedi mentioned this in Unknown Object (Task).Apr 18 2016, 3:09 PM
fgiunchedi triaged this task as Normal priority.Apr 27 2016, 9:26 AM
Danny_B renamed this task from [tracking] refresh swift hardware in codfw/eqiad to refresh swift hardware in codfw/eqiad (tracking).May 27 2016, 5:42 PM

6x swift systems (all 3TB disks) have been ordered in T130713 and T136336, though we'll be keeping the old swift hw in place for the next 6/9 months as the hardware is just out of warranty. With the old hardware in place, the last 6x order is expected to last ~1y

fgiunchedi added subtasks: Unknown Object (Task), Unknown Object (Task).May 30 2016, 9:51 AM
Restricted Application added a subscriber: Steinsplitter. · View Herald TranscriptJun 23 2016, 1:27 PM
fgiunchedi renamed this task from refresh swift hardware in codfw/eqiad (tracking) to expand swift hardware in codfw/eqiad (tracking).Jun 23 2016, 2:26 PM
fgiunchedi updated the task description. (Show Details)
Phabricator_maintenance renamed this task from expand swift hardware in codfw/eqiad (tracking) to expand swift hardware in codfw/eqiad.Aug 14 2016, 12:17 AM
RobH closed subtask Unknown Object (Task) as Resolved.Oct 12 2016, 5:47 PM
RobH closed subtask Unknown Object (Task) as Resolved.
fgiunchedi closed this task as Resolved.Dec 1 2016, 6:30 PM

Complete, new hw in place

fgiunchedi mentioned this in Unknown Object (Task).Jan 6 2017, 10:37 PM