Page MenuHomePhabricator
Create Task
Maniphest T233184

db2127 memory issues
Closed, ResolvedPublic
Actions

Description

@Papaul reported the following on irc:

15:08:57 <papaul> marostegui: hello can you check db2127 it is making a lot of noise and i have a yellow blinking lid

And these are the logs HW which show memory issues

P9119 (An Untitled Masterwork)
1Record: 1
2Date/Time: 07/12/2019 03:59:07
3Source: system
4Severity: Ok
5Description: Log cleared.
6-------------------------------------------------------------------------------
7Record: 2
8Date/Time: 08/14/2019 21:10:59
9Source: system
10Severity: Non-Critical
11Description: Correctable Machine Check Exception detected on CPU 1.
12-------------------------------------------------------------------------------
13Record: 3
14Date/Time: 08/14/2019 21:10:59
15Source: system
16Severity: Ok
17Description: An OEM diagnostic event occurred.
18-------------------------------------------------------------------------------
19Record: 4
20Date/Time: 08/14/2019 21:10:59
21Source: system
22Severity: Ok
23Description: An OEM diagnostic event occurred.
24-------------------------------------------------------------------------------
25Record: 5
26Date/Time: 08/14/2019 21:10:59
27Source: system
28Severity: Ok
29Description: An OEM diagnostic event occurred.
30-------------------------------------------------------------------------------
31Record: 6
32Date/Time: 08/14/2019 21:10:59
33Source: system
34Severity: Ok
35Description: An OEM diagnostic event occurred.
36-------------------------------------------------------------------------------
37Record: 7
38Date/Time: 08/14/2019 21:10:59
39Source: system
40Severity: Ok
41Description: An OEM diagnostic event occurred.
42-------------------------------------------------------------------------------
43Record: 8
44Date/Time: 08/14/2019 21:10:59
45Source: system
46Severity: Ok
47Description: An OEM diagnostic event occurred.
48-------------------------------------------------------------------------------
49Record: 9
50Date/Time: 08/14/2019 21:10:59
51Source: system
52Severity: Ok
53Description: An OEM diagnostic event occurred.
54-------------------------------------------------------------------------------
55Record: 10
56Date/Time: 08/14/2019 21:10:59
57Source: system
58Severity: Ok
59Description: An OEM diagnostic event occurred.
60-------------------------------------------------------------------------------
61Record: 11
62Date/Time: 08/14/2019 21:10:59
63Source: system
64Severity: Ok
65Description: An OEM diagnostic event occurred.
66-------------------------------------------------------------------------------
67Record: 12
68Date/Time: 08/14/2019 21:11:00
69Source: system
70Severity: Ok
71Description: An OEM diagnostic event occurred.
72-------------------------------------------------------------------------------
73Record: 13
74Date/Time: 08/14/2019 21:11:00
75Source: system
76Severity: Ok
77Description: An OEM diagnostic event occurred.
78-------------------------------------------------------------------------------
79Record: 14
80Date/Time: 08/14/2019 21:11:00
81Source: system
82Severity: Ok
83Description: An OEM diagnostic event occurred.
84-------------------------------------------------------------------------------
85Record: 15
86Date/Time: 08/14/2019 21:11:00
87Source: system
88Severity: Critical
89Description: Correctable memory error logging disabled for a memory device at location DIMM_A6.
90-------------------------------------------------------------------------------

After putting down the host @Papaul reports that the noise and blinking led is gone as well as the DIMM issue cleared out.

Related Objects

Event Timeline

Marostegui created this task.Sep 18 2019, 5:02 AM
Restricted Application added a project: SRE. · View Herald TranscriptSep 18 2019, 5:02 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Sep 18 2019, 5:03 AM
Marostegui moved this task from Triage to In progress on the DBA board.

We are leaving this task opened for a few days to see if the errors get back.

Stashbot added a comment.Sep 18 2019, 5:03 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T05:03:41Z] <marostegui> Start MySQL on db2127 T233184

Stashbot added a comment.Sep 18 2019, 5:47 AM

Mentioned in SAL (#wikimedia-operations) [2019-09-18T05:47:56Z] <marostegui@cumin1001> dbctl commit (dc=all): 'Repool host after onsite checks T233184', diff saved to https://phabricator.wikimedia.org/P9123 and previous config saved to /var/cache/conftool/dbconfig/20190918-054755-marostegui.json

Marostegui added a comment.Sep 20 2019, 9:02 AM

No more errors, if it continues clean on Monday I will close this task

Marostegui closed this task as Resolved.Sep 23 2019, 7:42 AM
Marostegui assigned this task to Papaul.

HW logs look clean, closing this!
Thanks @Papaul for catching this!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits