Page MenuHomePhabricator
Create Task
Maniphest T292486

swift - ms-be1059 - device sdi:3 unavailable
Closed, ResolvedPublic
Actions

Description

on ms-be1059, sdi:3 is gone:

compare to T291988, T291896 in codfw

[ms-be1059:~] $ sudo systemctl status swift-drive-audit
● swift-drive-audit.service - Regular jobs to unmount failed disks
   Loaded: loaded (/lib/systemd/system/swift-drive-audit.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2021-10-04 23:01:12 UTC; 6min ago
  Process: 35328 ExecStart=/usr/bin/swift-drive-audit /etc/swift/swift-drive-audit.conf (code=exited, status=4)
 Main PID: 35328 (code=exited, status=4)

Oct 04 23:01:11 ms-be1059 systemd[1]: Started Regular jobs to unmount failed disks.
Oct 04 23:01:12 ms-be1059 drive-audit[35328]: Errors found but device unavailable: sdi:3

Related Objects

Event Timeline

Dzahn created this task.Oct 4 2021, 11:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 4 2021, 11:13 PM
Dzahn mentioned this in T222362: swift-drive-audit unmounting a drive doesn't produce any alerts or notifications.Oct 4 2021, 11:13 PM
Dzahn added projects: SRE-swift-storage, ops-eqiad.Oct 4 2021, 11:15 PM
Dzahn added a project: SRE.
• Cmjohnson subscribed.Oct 5 2021, 6:34 PM

These are just out of warranty by a few months. I do have spare disks on-site, my guess is a disk went bad on you. Is there any way you can tell me which disk slot corresponds to /dev/sdi?

Dzahn added a comment.Oct 5 2021, 6:54 PM

@Cmjohnson The failed disk should have an amber light on it when you're onsite.

Dzahn added a comment.Oct 5 2021, 6:55 PM

serial PEYHD0DRHC617F

• Cmjohnson moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Oct 5 2021, 8:15 PM
• Cmjohnson added a comment.Oct 6 2021, 4:40 PM

I replaced what I think was /dev/sdi. The server did not show any amber led to let me know which disk was failed.

• Cmjohnson added a comment.Oct 6 2021, 4:41 PM

It will need to be added back to the array. It was slot 8 that was replaced

Dzahn added subscribers: Papaul, fgiunchedi.Oct 6 2021, 6:45 PM

Thanks @Cmjohnson !

@Papaul @fgiunchedi looking at the linked tickets that had similar cases it seemed as if just replacing the disk auto-fixed things, or was there a step where one of you added the disk back to the array? Also note how in the codfw case we had the amber light but not here. Not sure what is next exactly, was just reporting.

wiki_willy assigned this task to • Cmjohnson.Oct 6 2021, 11:50 PM
fgiunchedi closed this task as Resolved.Oct 7 2021, 7:36 AM

Thanks @Dzahn and @Cmjohnson ! I've done the procedure and documented it at https://wikitech.wikimedia.org/wiki/Swift/How_To#Replacing_a_disk_without_touching_the_rings. We're back so I'm resolving the task

Dzahn awarded a token.Oct 7 2021, 6:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL