Page MenuHomePhabricator

New hardware for production core mysql cluster
Closed, DuplicatePublic

Description

s3 is about to hit the 90% limit, although this time is giving us a 6 month-1 year advance notice:


1db1038.eqiad.wmnet:
2 /dev/sda6 1.7T 1.4T 291G 83% /a
3db1015.eqiad.wmnet:
4 /dev/mapper/tank-data 1.4T 1.3T 157G 89% /srv
5db1027.eqiad.wmnet:
6 /dev/mapper/tank-data 1.5T 1.3T 184G 88% /srv
7db1035.eqiad.wmnet:
8 /dev/mapper/tank-data 1.4T 1.3T 158G 89% /srv
9db1044.eqiad.wmnet:
10 /dev/sda6 1.7T 1.3T 408G 76% /srv

s4 also is complaining about lack of space.

Event Timeline

jcrespo created this task.Jul 24 2015, 4:53 PM
jcrespo raised the priority of this task from to Low.
jcrespo updated the task description. (Show Details)
jcrespo added projects: acl*sre-team, DBA.
jcrespo added subscribers: jcrespo, Springle.
Restricted Application added subscribers: Matanya, Aklapper. · View Herald TranscriptJul 24 2015, 4:53 PM

The latest bump is probably caused by T104278.

jcrespo moved this task from Triage to Backlog on the DBA board.Jul 24 2015, 5:35 PM
jcrespo raised the priority of this task from Low to Medium.Aug 3 2015, 10:52 AM

db1035 died.

jcrespo added a subscriber: RobH.Aug 4 2015, 7:51 AM

@Springle We need to spec a hardware replacement to be used for production hosts and send it to @RobH

jcrespo renamed this task from New s3 production cluster for mysql to New hardware for production core mysql cluster.Aug 7 2015, 9:46 AM
jcrespo updated the task description. (Show Details)
jcrespo set Security to None.

We need to spec a hardware replacement to be used for production hosts and send it to @RobH

Has that happened in the meantime?

Aklapper, 4 months to full failure? that is too much time for operator standards! :-)

I designed with Sean's help an initial spec, have to ask for initial quotes still.

jcrespo raised the priority of this task from Medium to High.Sep 28 2015, 2:21 PM

s3 is in critical state (see T110504), send procurement soon.

jcrespo closed this task as a duplicate of Unknown Object (Task).Nov 16 2015, 11:00 AM