Maniphest T205780

db1067 (enwiki master) disk #7 with errors
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Marostegui
	Sep 29 2018, 4:36 PM

Description

Hello,

db1067 is the enwiki master and has disk number 7 with errors.
Can we replace it before we failover back to eqiad?
Thanks!

Enclosure Device ID: 32
Slot Number: 7
Drive's position: DiskGroup: 0, Span: 1, Arm: 1
Enclosure position: 1
Device Id: 7
WWN: 5000C50071AFE3C8
Sequence Number: 2
Media Error Count: 124
Other Error Count: 0
Predictive Failure Count: 1
Last Predictive Failure Event Seq Number: 191044
PD Type: SAS

Raw Size: 558.911 GB [0x45dd2fb0 Sectors]
Non Coerced Size: 558.411 GB [0x45cd2fb0 Sectors]
Coerced Size: 558.375 GB [0x45cc0000 Sectors]
Sector Size:  0
Firmware state: Online, Spun Up
Device Firmware Level: ES66
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5000c50071afe3c9
SAS Address(1): 0x0
Connected Port Number: 0(path0)
Inquiry Data: SEAGATE ST3600057SS     ES666SL81VXQ
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :41C (105.80 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : Yes

Related Objects

Duplicates Merged Here: T205990: Degraded RAID on db1067

Event Timeline

• Marostegui created this task.Sep 29 2018, 4:36 PM

Restricted Application added a project: SRE. · View Herald TranscriptSep 29 2018, 4:36 PM

• Marostegui triaged this task as High priority.Sep 29 2018, 4:37 PM

• Marostegui moved this task from Triage to In progress on the DBA board.

Mentioned in SAL (#wikimedia-operations) [2018-10-02T16:38:42Z] <cmjohnson1> swapping failed disk db1067 T205780

The disk has been swapped

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Oct 2 2018, 4:41 PM

Thank you! Our alerting detects the rebuild as a down so I had worried at first without context :-) Will close when I can assure the rebuild completed successfully.

jcrespo merged a task: T205990: Degraded RAID on db1067.Oct 2 2018, 5:04 PM

jcrespo added a subscriber: ops-monitoring-bot.

The alert cleared and the RAID is back to optimal!

root@db1067:~# megacli -LDPDInfo -aAll

Adapter #0

Number of Virtual Disks: 1
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 3.271 TB
Sector Size         : 512
Mirror Data         : 3.271 TB
State               : Optimal
Strip Size          : 256 KB
Number Of Drives per span:6
Span Depth          : 2

root@db1067:~# megacli -LDPDInfo -aAll | egrep -i "error|flagged"
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 4
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No
Media Error Count: 0
Other Error Count: 0
Drive has flagged a S.M.A.R.T alert : No

Thank you Chris!!

db1067 (enwiki master) disk #7 with errorsClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

db1067 (enwiki master) disk #7 with errors
Closed, ResolvedPublic
Actions