Page MenuHomePhabricator
Create Task
Maniphest T222753

db2114 hardware problem
Closed, ResolvedPublic
Actions

Description

On of the server received in T220431 have some hardware issues. This task will track that the issue. The OS install complete with a problem see T221532

Critical,Tue 07 May 2019 10:04:30,Fan redundancy is lost.,
Normal,Tue 07 May 2019 10:03:33,The fans are redundant.,
Critical,Tue 07 May 2019 10:00:35,Fan redundancy is lost.,
Normal,Tue 07 May 2019 09:59:35,The fans are redundant.,
Critical,Mon 06 May 2019 10:59:53,Fan redundancy is lost.,
Critical,Mon 06 May 2019 10:59:51,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 10:59:51,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 10:35:11,Fan redundancy is lost.,
Normal,Mon 06 May 2019 10:34:04,The fans are redundant.,
Critical,Mon 06 May 2019 05:51:45,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:51:02,The fans are redundant.,
Critical,Mon 06 May 2019 05:49:50,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 05:49:50,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:49:46,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:48:54,The fans are redundant.,
Normal,Mon 06 May 2019 05:48:52,Fan 3B RPM is within range.,
Warning,Mon 06 May 2019 05:48:52,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:46:50,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 05:46:49,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:46:46,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:42:41,The fans are redundant.,
Normal,Mon 06 May 2019 05:42:37,Fan 3B RPM is within range.,
Warning,Mon 06 May 2019 05:42:36,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:42:36,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:41:55,The fans are redundant.,
Critical,Mon 06 May 2019 05:39:35,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 05:39:35,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:39:29,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:38:50,Fan 3B RPM is within range.,
Warning,Mon 06 May 2019 05:38:49,Fan 3B RPM is less than the lower warning threshold.,
Normal,Mon 06 May 2019 05:38:26,The fans are redundant.,
Critical,Mon 06 May 2019 05:32:43,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 05:32:42,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:32:36,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:30:46,The fans are redundant.,
Normal,Mon 06 May 2019 05:30:36,Fan 3B RPM is within range.,
Warning,Mon 06 May 2019 05:30:36,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:30:20,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 05:30:20,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:30:15,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:27:11,Fan 3B RPM is within range.,
Warning,Mon 06 May 2019 05:27:11,Fan 3B RPM is less than the lower warning threshold.,
Normal,Mon 06 May 2019 05:27:05,The fans are redundant.,
Critical,Mon 06 May 2019 05:25:28,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 05:25:27,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:25:26,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:21:19,The fans are redundant.,
Normal,Mon 06 May 2019 05:21:12,Fan 3B RPM is within range.,
Warning,Mon 06 May 2019 05:21:12,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:21:02,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:20:29,The fans are redundant.,
Critical,Mon 06 May 2019 05:19:42,Fan redundancy is lost.,
Normal,Mon 06 May 2019 05:19:09,The fans are redundant.,
Critical,Mon 06 May 2019 05:18:18,Fan 3B RPM is less than the lower critical threshold.,
Warning,Mon 06 May 2019 05:18:18,Fan 3B RPM is less than the lower warning threshold.,
Critical,Mon 06 May 2019 05:18:15,Fan redundancy is lost.,
Normal,Tue 23 Apr 2019 14:49:58,Log cleared.,

it looks like we are missng FAN 3B i am going to open the server and double check

Status Name Type PWM (% of Max) RPM
System Board Fan1A Standard Performance 100% 15960
System Board Fan1B Standard Performance 100% 17040
System Board Fan2A Standard Performance 100% 14760
System Board Fan2B Standard Performance 100% 17520
System Board Fan3A Standard Performance 100% 14760
System Board Fan3B Standard Performance N/A 0
System Board Fan4A Standard Performance 100% 14760
System Board Fan4B Standard Performance 100% 18240
System Board Fan5A Standard Performance 100% 14880
System Board Fan5B Standard Performance 100% 18000
System Board Fan6A Standard Performance 100% 15840
System Board Fan6B Standard Performance 100% 17040

I swapped FAN 3 with FAN 5 still have the same issue so the problem is not the FAN it has to be on the main board.

Related Objects

Event Timeline

Papaul created this task.May 7 2019, 7:04 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2019, 7:04 PM
Papaul triaged this task as Medium priority.May 7 2019, 7:05 PM
Papaul mentioned this in T221532: rack/setup/install db2[103-120].codfw.wmnet (18 hosts).
Papaul added a comment.May 7 2019, 9:38 PM

I spoke with Dell, they are going to replace the main board.

Enterprise Service Request 990336510

• Marostegui subscribed.May 8 2019, 5:12 AM

Thanks @Papaul for troubleshooting this

• Marostegui mentioned this in T222772: Productionize db2[103-120].May 8 2019, 5:40 AM
• Marostegui moved this task from Triage to In progress on the DBA board.May 8 2019, 7:46 AM
Papaul added a comment.May 8 2019, 1:32 PM

Hey Paul

Your dispatch number is 707804261.

Once I get a waybill number I will send that over to you.

Thank you

Reedy added a project: ops-codfw.May 8 2019, 1:50 PM
Restricted Application added a project: SRE. · View Herald TranscriptMay 8 2019, 1:50 PM
jcrespo subscribed.May 8 2019, 3:12 PM

Thanks.

Papaul added a comment.May 8 2019, 6:53 PM

Technician will be on site tomorrow between 9:30 and 10am CT

• Marostegui added a comment.May 8 2019, 7:11 PM

Great news! Thanks!

Papaul closed this task as Resolved.May 9 2019, 4:08 PM

@Marostegui @jcrespo main board replacement complete on db2114. The problem has been resolved. You can take over now.
System Board Fan1A 14% 3480 RPM 840 RPM N/A 480 RPM N/A
System Board Fan1B 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan2A 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan2B 14% 5040 RPM 840 RPM N/A 480 RPM N/A
System Board Fan3A 14% 3480 RPM 840 RPM N/A 480 RPM N/A
System Board Fan3B 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan4A 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan4B 14% 5160 RPM 840 RPM N/A 480 RPM N/A
System Board Fan5A 14% 3720 RPM 840 RPM N/A 480 RPM N/A
System Board Fan5B 14% 5160 RPM 840 RPM N/A 480 RPM N/A
System Board Fan6A 14% 3480 RPM 840 RPM N/A 480 RPM N/A
System Board Fan6B 14% 3840 RPM 840 RPM N/A 480 RPM N/A

• Marostegui added a comment.May 9 2019, 4:09 PM

Thanks Papaul!

• Marostegui added a comment.May 10 2019, 5:40 AM

This host has been re-imaged successfully

Aklapper removed a subscriber: ops-codfw.May 17 2023, 7:34 AM
Maintenance_bot moved this task from In progress to Done on the DBA board.May 17 2023, 8:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL