The system is out of warranty, will open a task for procurement of new disk.

These are my comments after rechecking both the disks and the server roles:

db2004 slot 8 - confirmed, but unused? Maybe pending decommission? I need to confirm it
db2007 slot 0 - confirmed, but unused? Maybe pending decommission? I need to confirm it
db2010 slot 0 and slot 8 - confirmed, m1 slave
db2011 slot 7 and slot 11 - confirmed, m2 slave
db2013 slot 7 - I do not have access, and it is not part of the regular mysql core production, ask someone else (fundrising? decommissoned?)
db2018 slot 3 - confirmed, s3 master, important
db2017 slot 11 - confirmed, s2 master, important
db2021 slot 6 and slot 7 - I do not have access, and it is not part of the regular mysql core production, ask someone else (fundrising? decommissoned?)
db2023 slot 9 - confirmed, s5 master, important
db2024 slot 4 - I do not have access, and it is not part of the regular mysql core production, ask someone else (fundrising? decommissoned?)

Let me check if db2004 and db2007 are unused. If they are, we could decommission them and reuse its disks.

The following servers respond to salt but do not have any known function:

db2001.codfw.wmnet: 
db2002.codfw.wmnet:
db2003.codfw.wmnet:
db2004.codfw.wmnet:
db2005.codfw.wmnet:
db2007.codfw.wmnet:

I do not know if there was any plan to decommission them, but they may be used for parts.

Related T125827

@jcrespo thanks so i will be waiting on @RobH for final confirmation on T125827

db2017 slot 11 failed completely today.

@Papaul @RobH: any news on this?

In particular for db2017 (failed), db2018 (failed) and db2023 (predicted failure), that are masters in codfw, it would be better to have those replaced and the RAID rebuild before the codfw switchover.

Thanks to @Papaul replaced 2 failed disks on db2017 and 1 on db2018.
Keeping the last spare disk as spare for now in case another DB disk breaks in the next days, db2023 is only in predictive failure. I'll check with @jcrespo if it's ok for him.

volans@db2017:~$ sudo megacli -PDRbld -ShowProg -PhysDrv [32:02] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 2 Completed 0% in 2 Minutes.

Exit Code: 0x00
volans@db2017:~$ sudo megacli -PDRbld -ShowProg -PhysDrv [32:11] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 11 Completed 1% in 6 Minutes.

Exit Code: 0x00

volans@db2018:~$ sudo megacli -PDRbld -ShowProg -PhysDrv [32:03] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 3 Completed 0% in 0 Minutes.

Exit Code: 0x00

Drive replacement on db2017 slot 11 and slot 2
Drive replacement on db2018 slot 3

Rebuild halfway trough, I've disabled notification for the RAID check for db2017 and db2018 to avoid to be awaken in the middle of the night by the recovery. I'll re-activate them tomorrow after the rebuild is complete.

volans@db2017:~$ sudo megacli -PDRbld -ShowProg -PhysDrv [32:11] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 11 Completed 53% in 206 Minutes.

Exit Code: 0x00
volans@db2017:~$ sudo megacli -PDRbld -ShowProg -PhysDrv [32:02] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 2 Completed 52% in 202 Minutes.

Exit Code: 0x00

volans@db2018:~$ sudo megacli -PDRbld -ShowProg -PhysDrv [32:03] -aALL

Rebuild Progress on Device at Enclosure 32, Slot 3 Completed 53% in 198 Minutes.

Exit Code: 0x00

db2017 and db2018 RAID is back to optimal, re-enabled notifications on icinga for RAID checks.
Leaving the task open for the remaining hosts.

We are going to have lots of spares with T129452. I would retire es2005-es2010 and use its disks as spares, reuse es2001-4 for ES disaster recovery and archival.