Page MenuHomePhabricator
Create Task
Maniphest T339340

hw troubleshooting: CPU machine check failure for parse1002.eqiad.wmnet
Closed, ResolvedPublicRequest
Actions

Description

  • - Provide FQDN of system.
  • - If other than a hard drive issue, please depool the machine (and confirm that it’s been depooled) for us to work on it. If not, please provide time frame for us to take the machine down.
  • - Put system into a failed state in Netbox.
  • - Provide urgency of request, along with justification (redundancy, dependencies, etc)
  • - Describe issue and/or attach hardware failure log. (Refer to https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook if you need help)
  • - Assign correct project tag and appropriate owner (based on above). Also, please ensure the service owners of the host(s) are added as subscribers to provide any additional input.

Warranty: Machine was purchased on 2022-01-21, it's should still be in warranty
Urgency: Low

Host fully removed from conftool (set to inactive).

History: This is the 3rd repeat incident. See T324949, T326119. Also yesterday per https://sal.toolforge.org/log/-LPbvogBGiVuUzOdSQAs @claime powercycle the host. I also powercycled it today

ipmi-sel

ID  | Date        | Time     | Name             | Type                        | Event
1   | Jan-09-2023 | 14:38:27 | SEL              | Event Logging Disabled      | Log Area Reset/Cleared
2   | Jun-15-2023 | 12:05:35 | CPU Machine Chk  | Processor                   | transition to Non-recoverable ; OEM Event Data2 code = 02h ; OEM Event Data3 code = 20h
3   | Jun-15-2023 | 12:05:35 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 09h ; OEM Event Data3 code = 00h
4   | Jun-15-2023 | 12:05:35 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
5   | Jun-15-2023 | 12:05:35 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh
6   | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
7   | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Ah ; OEM Event Data3 code = 00h
8   | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
9   | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh ; OEM Event Data2 code = 35h ; OEM Event Data3 code = 20h
10  | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
11  | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh ; OEM Event Data3 code = 00h
12  | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
13  | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 08h ; OEM Event Data2 code = 80h ; OEM Event Data3 code = A0h
14  | Jun-15-2023 | 12:05:36 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 01h ; OEM Event Data2 code = A4h ; OEM Event Data3 code = 00h
15  | Jun-15-2023 | 11:08:44 | Additional Info  | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data2 code = 02h ; OEM Event Data3 code = 00h
16  | Jun-15-2023 | 11:08:44 | CPU Machine Chk  | Processor                   | transition to Non-recoverable ; OEM Event Data2 code = 01h ; OEM Event Data3 code = 00h
17  | Jun-15-2023 | 11:08:44 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
18  | Jun-15-2023 | 11:08:44 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
19  | Jun-15-2023 | 11:08:44 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
20  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
21  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Eh
22  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
23  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data2 code = 9Bh
24  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh
25  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh
26  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
27  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data2 code = 9Bh
28  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh
29  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 01h ; OEM Event Data3 code = 00h
30  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
31  | Jun-15-2023 | 11:08:45 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
32  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
33  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data3 code = 00h
34  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
35  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh ; OEM Event Data3 code = 63h
36  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
37  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 03h ; OEM Event Data3 code = 00h
38  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
39  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh ; OEM Event Data3 code = 63h
40  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
41  | Jun-15-2023 | 11:08:46 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 09h ; OEM Event Data3 code = 00h
42  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
43  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Ch
44  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
45  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Ah ; OEM Event Data3 code = 00h
46  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
47  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh ; OEM Event Data2 code = 35h ; OEM Event Data3 code = 20h
48  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
49  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh ; OEM Event Data3 code = 00h
50  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
51  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
52  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 04h
53  | Jun-15-2023 | 11:08:47 | CPU Machine Chk  | Processor                   | transition to Non-recoverable ; OEM Event Data2 code = 02h ; OEM Event Data3 code = 20h
54  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 09h ; OEM Event Data3 code = 00h
55  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
56  | Jun-15-2023 | 11:08:47 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh
57  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
58  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Ah ; OEM Event Data3 code = 00h
59  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
60  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh ; OEM Event Data2 code = 35h ; OEM Event Data3 code = 20h
61  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
62  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh ; OEM Event Data3 code = 00h
63  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
64  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 08h ; OEM Event Data2 code = 80h ; OEM Event Data3 code = A0h
65  | Jun-15-2023 | 11:08:48 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 01h ; OEM Event Data2 code = A4h ; OEM Event Data3 code = 00h
66  | Jun-15-2023 | 11:08:48 | Chipset Info     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 40h ; OEM Event Data3 code = 00h
67  | Jun-15-2023 | 11:08:48 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
68  | Jun-15-2023 | 11:08:48 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
69  | Jun-15-2023 | 11:08:48 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
70  | Jun-15-2023 | 11:08:48 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
71  | Jun-15-2023 | 11:08:49 | Chipset Info     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 40h ; OEM Event Data3 code = 00h
72  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
73  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
74  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
75  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
76  | Jun-15-2023 | 11:08:49 | Chipset Info     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = F2h ; OEM Event Data3 code = 17h
77  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
78  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
79  | Jun-15-2023 | 11:08:49 | Chipset Info     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 40h ; OEM Event Data3 code = 80h
80  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
81  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
82  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
83  | Jun-15-2023 | 11:08:49 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
84  | Jun-15-2023 | 11:08:49 | Chipset Info     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 40h ; OEM Event Data3 code = 80h
85  | Jun-15-2023 | 11:08:50 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
86  | Jun-15-2023 | 11:08:50 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
87  | Jun-15-2023 | 11:08:50 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
88  | Jun-15-2023 | 11:08:50 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
89  | Jun-15-2023 | 11:08:50 | Chipset Info     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = F2h ; OEM Event Data3 code = 85h
90  | Jun-15-2023 | 11:08:50 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
91  | Jun-15-2023 | 11:08:50 | Err Reg Pointer  | OEM Reserved                | OEM Event Offset = 00h
92  | Jun-15-2023 | 11:08:50 | CPU Machine Chk  | Processor                   | transition to Non-recoverable ; OEM Event Data2 code = 01h ; OEM Event Data3 code = 02h
93  | Jun-15-2023 | 11:08:50 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
94  | Jun-15-2023 | 11:08:50 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
95  | Jun-15-2023 | 11:08:50 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
96  | Jun-15-2023 | 11:08:50 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
97  | Jun-15-2023 | 11:08:50 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
98  | Jun-15-2023 | 11:08:50 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
99  | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
100 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
101 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Eh
102 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
103 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh ; OEM Event Data3 code = 63h
104 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
105 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh
106 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
107 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh ; OEM Event Data3 code = 63h
108 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
109 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
110 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
111 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
112 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
113 | Jun-15-2023 | 11:08:51 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
114 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
115 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
116 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
117 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Eh
118 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
119 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 09h ; OEM Event Data3 code = 63h
120 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
121 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh
122 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
123 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 09h ; OEM Event Data3 code = 63h
124 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
125 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
126 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
127 | Jun-15-2023 | 11:08:52 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
128 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
129 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
130 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
131 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
132 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
133 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
134 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
135 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
136 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
137 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Dh
138 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
139 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 00h
140 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
141 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Eh
142 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
143 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 08h ; OEM Event Data3 code = 63h
144 | Jun-15-2023 | 11:08:53 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
145 | Jun-15-2023 | 11:08:54 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Fh
146 | Jun-15-2023 | 11:08:54 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
147 | Jun-15-2023 | 11:08:54 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 08h ; OEM Event Data3 code = 63h
148 | Jun-15-2023 | 11:08:54 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 05h ; OEM Event Data2 code = 00h ; OEM Event Data3 code = 00h
149 | Jun-15-2023 | 15:53:12 | CPU Machine Chk  | Processor                   | transition to Non-recoverable ; OEM Event Data2 code = 02h ; OEM Event Data3 code = 20h
150 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 09h ; OEM Event Data3 code = 00h
151 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
152 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh
153 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
154 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Ah ; OEM Event Data3 code = 00h
155 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
156 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 2Bh ; OEM Event Data3 code = 10h
157 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
158 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh ; OEM Event Data3 code = 00h
159 | Jun-15-2023 | 15:53:13 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
160 | Jun-15-2023 | 15:53:14 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 08h ; OEM Event Data2 code = 80h ; OEM Event Data3 code = 20h
161 | Jun-15-2023 | 15:53:14 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data2 code = A4h ; OEM Event Data3 code = 00h
162 | Jun-15-2023 | 15:55:01 | Additional Info  | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data2 code = 02h ; OEM Event Data3 code = 00h
163 | Jun-15-2023 | 15:55:01 | CPU Machine Chk  | Processor                   | transition to Non-recoverable ; OEM Event Data2 code = 02h ; OEM Event Data3 code = 20h
164 | Jun-15-2023 | 15:55:01 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 09h ; OEM Event Data3 code = 00h
165 | Jun-15-2023 | 15:55:01 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
166 | Jun-15-2023 | 15:55:01 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh
167 | Jun-15-2023 | 15:55:02 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
168 | Jun-15-2023 | 15:55:02 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Ah ; OEM Event Data3 code = 00h
169 | Jun-15-2023 | 15:55:02 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
170 | Jun-15-2023 | 15:55:02 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h ; OEM Event Data2 code = 2Bh ; OEM Event Data3 code = 10h
171 | Jun-15-2023 | 15:55:02 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
172 | Jun-15-2023 | 15:55:02 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 0Bh ; OEM Event Data3 code = 00h
173 | Jun-15-2023 | 15:55:02 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 00h
174 | Jun-15-2023 | 15:55:03 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 08h ; OEM Event Data2 code = 80h ; OEM Event Data3 code = 20h
175 | Jun-15-2023 | 15:55:03 | MSR Info Log     | OEM Reserved                | OEM Event Offset = 02h ; OEM Event Data2 code = A4h ; OEM Event Data3 code = 00h

Related Objects

Event Timeline

akosiaris created this task.Jun 16 2023, 9:00 AM
akosiaris renamed this task from hw troubleshooting: CPU machine check failure for parse1002 to hw troubleshooting: CPU machine check failure for parse1002.eqiad.wmnet.
akosiaris updated the task description. (Show Details)Jun 16 2023, 9:05 AM

Host appears functional right now, but I don't trust it to put it back into rotation.

Maintenance_bot added a project: SRE.Jun 16 2023, 9:29 AM
Jclark-ctr subscribed.Jun 16 2023, 12:29 PM

Dell Service Request 170238017

ops-monitoring-bot added a comment.Jun 16 2023, 1:55 PM

Icinga downtime and Alertmanager silence (ID=449aef54-88d6-4ae4-81c0-075c731ff7c3) set by cgoubert@cumin1001 for 7 days, 0:00:00 on 1 host(s) and their services with reason: hw troubleshooting

parse1002.eqiad.wmnet
Jclark-ctr added a comment.Jun 21 2023, 7:00 PM

updated firmware.
Started cpu stress test per dell request has ran for 24 hours with no errors on tsr report

ops-monitoring-bot added a comment.Jun 23 2023, 2:12 PM

Icinga downtime and Alertmanager silence (ID=10c1e3f4-0e17-4298-b048-57e9021a0c6f) set by vgutierrez@cumin1001 for 7 days, 0:00:00 on 1 host(s) and their services with reason: HW issues

parse1002.eqiad.wmnet
wiki_willy assigned this task to Jclark-ctr.Jun 23 2023, 10:00 PM
Jclark-ctr added a comment.Jun 26 2023, 10:58 PM

Performed stresstest on cpu for additional 24 hours with no errors restarting 3rd time

Jclark-ctr added a comment.Jun 29 2023, 10:08 AM

@akosiaris Server Ran in cpu stress test for in total 5 days with no errors prior to running firmware was updated. At this time dell will not replace any parts the only recommendation they had was possibly reimagine server.

akosiaris claimed this task.Jun 29 2023, 10:11 AM
In T339340#8975564, @Jclark-ctr wrote:

@akosiaris Server Ran in cpu stress test for in total 5 days with no errors prior to running firmware was updated. At this time dell will not replace any parts the only recommendation they had was possibly reimagine server.

OK, maybe the firwmare update did the trick and we are going to be ok this time around. Fingers crossed at least. I am not sure what reimaging would achieve tbh, on the other hand, it won't cost much more time, so I 'll do that and then put the server back in rotation.

Thanks for you work on this!

ops-monitoring-bot added a comment.Jun 29 2023, 10:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1001 for host parse1002.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jun 29 2023, 12:55 PM

Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1001 for host parse1002.eqiad.wmnet with OS buster completed:

  • parse1002 (WARN)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202306291035_akosiaris_1117585_parse1002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status failed -> active
    • The sre.puppet.sync-netbox-hiera cookbook was run successfully
akosiaris closed this task as Resolved.Jun 29 2023, 12:58 PM

I 've just re-imaged the server and set it back in conftool as active. Made sure to scap pull too. I am gonna resolve this, let's cross fingers we don't have a 4th occurence.

Joe reopened this task as Open.Jul 20 2023, 4:37 AM
Joe subscribed.

The server went down twice in one day yesterday, see T342298. So you can sadly uncross your fingers, @akosiaris :(

Joe added a comment.Jul 20 2023, 4:38 AM

The host is again set to inactive, and still not powercycled so that any further debugging can be performed, if needed.

Jclark-ctr added a comment.Jul 20 2023, 2:56 PM

Reopening ticket with dell

Jclark-ctr added a comment.Jul 21 2023, 11:53 AM

Service Request 172470150 was successfully submitted.

Clement_Goubert subscribed.Jul 25 2023, 12:43 PM

Note for ServiceOps later: once fixed, the host will need to be updated to pick up T340087: Deploy wikidiff2 1.14.1

Clement_Goubert mentioned this in T340087: Deploy wikidiff2 1.14.1.Jul 25 2023, 12:43 PM
Stashbot added a comment.Jul 25 2023, 1:20 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-25T13:20:03Z] <godog> powercycle parse1002 - T339340

fgiunchedi subscribed.Jul 25 2023, 1:22 PM

I rebooted the host because I needed a puppet run on it, I'll leave it alone now!

Stashbot added a comment.Jul 25 2023, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-25T13:42:09Z] <cgoubert@cumin1001> START - Cookbook sre.hosts.downtime for 15 days, 0:00:00 on parse1002.eqiad.wmnet with reason: T339340 - hw troubleshooting

Stashbot added a comment.Jul 25 2023, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2023-07-25T13:42:22Z] <cgoubert@cumin1001> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15 days, 0:00:00 on parse1002.eqiad.wmnet with reason: T339340 - hw troubleshooting

Jclark-ctr added a comment.Jul 27 2023, 3:31 PM

Dell they originally requested we take to min configuration to troubleshoot. i advised them that the time between errors is to far apart and troubleshooting could take months I pushed back at every request this time for troubleshooting and prolonged server downtime. They have agreed to replace mainboard and both CPU's. Tomorrow parts should arrive.

Jclark-ctr added a comment.Jul 28 2023, 11:58 PM

Both CPU's replaced and mainboard replaced today @akosiaris

Jclark-ctr moved this task from Backlog to Hardware Failure / Troubleshoot on the ops-eqiad board.Jul 29 2023, 12:13 AM
Clement_Goubert claimed this task.Jul 31 2023, 9:35 AM

I'm going to put it back in the pool to see if that fixed it, thanks @Jclark-ctr

Clement_Goubert edited projects, added: serviceops-deprecated; removed: serviceops-radar.Jul 31 2023, 9:36 AM
Clement_Goubert moved this task from Incoming 🐫 to 🛠 Upgrades and Hardware on the serviceops-deprecated board.
Stashbot added a comment.Jul 31 2023, 10:36 AM

Mentioned in SAL (#wikimedia-operations) [2023-07-31T10:36:04Z] <claime> Repooling parse1002 following CPU replacement - T339340

Clement_Goubert closed this task as Resolved.Jul 31 2023, 10:37 AM

Resolving for now, we will reopen if issues reappear.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits