hw troubleshooting: DIMM_B2 for mc2040.codfw.wmnet
Closed, ResolvedPublicRequest
Actions

Assigned To

Authored By

	Dzahn
	Jan 12 2023, 5:54 PM

Description

- Provide FQDN of system.
- If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
- Put system into a failed state in Netbox.
- Provide urgency of request, along with justification (redundancy, dependencies, etc)
- Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
- Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

FQDN: mc2040.codfw.wmnet
Issue: Bad DIMM_B2
Urgency: Medium

Related Objects

Mentioned In: T345802: hw troubleshooting: DIMM failure for mc2040.codfw.wmnet
Mentioned Here: P43145 mc2040 - getsel

Event Timeline

Dzahn created this task.Jan 12 2023, 5:54 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 12 2023, 5:54 PM

andrea.denisse subscribed.Jan 12 2023, 5:55 PM

P43145 mc2040 - getsel

1	racadm>>racadm getsel
2	Record: 1
3	Date/Time: 11/19/2021 21:38:45
4	Source: system
5	Severity: Ok
6	Description: Log cleared.
7	-------------------------------------------------------------------------------
8	Record: 2
9	Date/Time: 08/02/2022 16:52:09
10	Source: system
11	Severity: Critical
12	Description: The power input for power supply 1 is lost.
13	-------------------------------------------------------------------------------
14	Record: 3
15	Date/Time: 08/02/2022 16:52:49
16	Source: system
17	Severity: Critical
18	Description: Power supply redundancy is lost.
19	-------------------------------------------------------------------------------
20	Record: 4
21	Date/Time: 08/02/2022 16:57:43
22	Source: system
23	Severity: Ok
24	Description: The input power for power supply 1 has been restored.
25	-------------------------------------------------------------------------------
26	Record: 5
27	Date/Time: 08/02/2022 16:57:44
28	Source: system
29	Severity: Ok
30	Description: The power supplies are redundant.
31	-------------------------------------------------------------------------------
32	Record: 6
33	Date/Time: 12/14/2022 02:04:52
34	Source: system
35	Severity: Critical
36	Description: The system memory has faced an uncorrectable multi-bit memory errors in the non-execution path of a memory device at the location DIMM_B2.
37	-------------------------------------------------------------------------------
38	Record: 7
39	Date/Time: 12/14/2022 02:04:52
40	Source: system
41	Severity: Ok
42	Description: An OEM diagnostic event occurred.
43	-------------------------------------------------------------------------------
44	Record: 8
45	Date/Time: 12/14/2022 02:04:52
46	Source: system
47	Severity: Ok
48	Description: An OEM diagnostic event occurred.
49	-------------------------------------------------------------------------------
50	Record: 9
51	Date/Time: 12/14/2022 02:04:52
52	Source: system
53	Severity: Ok
54	Description: An OEM diagnostic event occurred.
55	-------------------------------------------------------------------------------
56	Record: 10
57	Date/Time: 12/14/2022 02:32:11
58	Source: system
59	Severity: Non-Critical
60	Description: The memory health monitor feature has detected a degradation in the DIMM installed in DIMM_B2. Reboot system to initiate self-heal process.
61	-------------------------------------------------------------------------------
62	Record: 11
63	Date/Time: 01/09/2023 18:34:18
64	Source: system
65	Severity: Ok
66	Description: A problem was detected during Power-On Self-Test (POST).
67	-------------------------------------------------------------------------------
68	Record: 12
69	Date/Time: 01/09/2023 18:34:18
70	Source: system
71	Severity: Ok
72	Description: The self-heal operation successfully completed at DIMM DIMM_B2.
73	-------------------------------------------------------------------------------
74	Record: 13
75	Date/Time: 01/09/2023 18:34:18
76	Source: system
77	Severity: Ok
78	Description: The self-heal operation successfully completed at DIMM DIMM_B2.
79	-------------------------------------------------------------------------------
80	Record: 14
81	Date/Time: 01/12/2023 13:48:14
82	Source: system
83	Severity: Non-Critical
84	Description: The memory health monitor feature has detected a degradation in the DIMM installed in DIMM_B2. Reboot system to initiate self-heal process.
85	-------------------------------------------------------------------------------
86	Record: 15
87	Date/Time: 01/12/2023 13:48:25
88	Source: system
89	Severity: Critical
90	Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
91	-------------------------------------------------------------------------------
92	Record: 16
93	Date/Time: 01/12/2023 13:50:29
94	Source: system
95	Severity: Ok
96	Description: A problem was detected related to the previous server boot.
97	-------------------------------------------------------------------------------
98	Record: 17
99	Date/Time: 01/12/2023 13:50:29
100	Source: system
101	Severity: Ok
102	Description: The self-heal operation successfully completed at DIMM DIMM_B2.
103	-------------------------------------------------------------------------------
104	Record: 18
105	Date/Time: 01/12/2023 13:50:29
106	Source: system
107	Severity: Ok
108	Description: The self-heal operation successfully completed at DIMM DIMM_B2.
109	-------------------------------------------------------------------------------
110	Record: 19
111	Date/Time: 01/12/2023 13:50:29
112	Source: system
113	Severity: Ok
114	Description: The self-heal operation successfully completed at DIMM DIMM_B2.
115	-------------------------------------------------------------------------------
116	Record: 20
117	Date/Time: 01/12/2023 13:50:29
118	Source: system
119	Severity: Critical
120	Description: Multi-bit memory errors detected on a memory device at location(s) DIMM_B2.
121	-------------------------------------------------------------------------------
122	racadm>>

cgoubert@mc2040:~$ sudo ipmi-sel | grep Jan-12-2023
14  | Jan-12-2023 | 13:48:14 | Mem ECC Warning  | Memory                      | Monitor
15  | Jan-12-2023 | 13:48:25 | ECC Uncorr Err   | Memory                      | Uncorrectable memory error
16  | Jan-12-2023 | 13:50:29 | Additional Info  | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data2 code = 02h ; OEM Event Data3 code = 00h
17  | Jan-12-2023 | 13:50:29 | POST Pkg Repair  | Memory                      | Redundancy Degraded from Fully Redundant
18  | Jan-12-2023 | 13:50:29 | POST Pkg Repair  | Memory                      | Redundancy Degraded from Fully Redundant
19  | Jan-12-2023 | 13:50:29 | POST Pkg Repair  | Memory                      | Redundancy Degraded from Fully Redundant
20  | Jan-12-2023 | 13:50:29 | ECC Uncorr Err   | Memory                      | Uncorrectable memory error
``

Dzahn added a subscriber: jijiki.Jan 12 2023, 5:58 PM

Dzahn updated the task description. (Show Details)Jan 12 2023, 6:01 PM

Dzahn updated the task description. (Show Details)

Clement_Goubert updated the task description. (Show Details)Jan 12 2023, 6:02 PM

jijiki added a project: ops-eqiad.Jan 12 2023, 6:03 PM

@jijiki mc2040 that is codfw not eqiad

Dzahn edited projects, added ops-codfw; removed ops-eqiad.Jan 12 2023, 6:08 PM

From what I understand, you can work on it any time, and we don't need to depool it. We may want to downtime it before y'all work on it.

@Clement_Goubert can you downtime the server? Please let me know when I can work on the server.

Icinga downtime and Alertmanager silence (ID=4016a17a-817d-4d48-be1d-b36713ff2632) set by cgoubert@cumin1001 for 1 day, 0:00:00 on 1 host(s) and their services with reason: hardware troubleshooting

mc2040.codfw.wmnet

@Jhancock.wm Done.

Maintenance_bot added a project: SRE.Jan 12 2023, 6:29 PM

@Clement_Goubert Thank you.

We powered down and swapped the A2 and B2 DIMM to see if the error carries over. as of right now we are not seeing any errors and the log has been cleared. We can close the task for now. If the error happens again, this task can be reopened.

@Jhancock.wm Thanks for the super quick turnaround. That was fast, wow.

someone needs to follow-up, for example do we set the status back to active in netbox, does it have to be taken back into production, so we should not forget that even though the ticket is closed

set back to active in netbox

Aklapper added a project: serviceops.May 16 2023, 1:16 PM

Aklapper removed a subscriber: serviceops.

Jhancock.wm mentioned this in T345802: hw troubleshooting: DIMM failure for mc2040.codfw.wmnet.Sep 7 2023, 1:29 PM

hw troubleshooting: DIMM_B2 for mc2040.codfw.wmnetClosed, ResolvedPublicRequestActions

Description

Related Objects

Event Timeline

hw troubleshooting: DIMM_B2 for mc2040.codfw.wmnet
Closed, ResolvedPublicRequest
Actions