Page MenuHomePhabricator
Create Task
Maniphest T239569

cloudstore1008 crash - Memory error
Closed, ResolvedPublic
Actions

Description

Just now I got an alert that cloudstore1008 was down. I cycled power via mgmt.

After checking the web console, this was caused by a memory error in "DIMM_B2" apparently.

Related Objects
Search...

Event Timeline

Andrew created this task.Dec 2 2019, 7:19 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 2 2019, 7:19 AM
Andrew added a comment.Dec 2 2019, 8:43 AM

I see nothing at all in the syslog that would explain this crash -- just an empty spot

Dec  2 07:13:09 cloudstore1008 ulogd[1910]: [fw-in-drop] IN=eno1 OUT= MAC=d0:94:66:26:d5:6a:5c:5e:ab:3d:87:c4:08:00 SRC=172.16.2.185 DST=208.80.155.125 LEN=44 TOS=00 PREC=0x00 TTL=48 ID=62385 PROTO=TCP SPT=49648 DPT=873 SEQ=4054715434 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0
Dec  2 07:13:09 cloudstore1008 ulogd[1910]: [fw-in-drop] IN=eno1 OUT= MAC=d0:94:66:26:d5:6a:5c:5e:ab:3d:87:c4:08:00 SRC=172.16.2.185 DST=208.80.155.125 LEN=44 TOS=00 PREC=0x00 TTL=48 ID=54280 PROTO=TCP SPT=49649 DPT=873 SEQ=4054780971 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0
Dec  2 07:13:09 cloudstore1008 ulogd[1910]: [fw-in-drop] IN=eno1 OUT= MAC=d0:94:66:26:d5:6a:5c:5e:ab:3d:87:c4:08:00 SRC=172.16.2.185 DST=208.80.155.125 LEN=44 TOS=00 PREC=0x00 TTL=57 ID=14724 PROTO=TCP SPT=49650 DPT=873 SEQ=4054584360 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0
Dec  2 07:13:15 cloudstore1008 ulogd[1910]: [fw-in-drop] IN=eno1 OUT= MAC=d0:94:66:26:d5:6a:5c:5e:ab:3d:87:c4:08:00 SRC=172.16.2.185 DST=208.80.155.125 LEN=44 TOS=00 PREC=0x00 TTL=56 ID=21729 PROTO=TCP SPT=49648 DPT=1207 SEQ=4054715434 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0
Dec  2 07:13:15 cloudstore1008 ulogd[1910]: [fw-in-drop] IN=eno1 OUT= MAC=d0:94:66:26:d5:6a:5c:5e:ab:3d:87:c4:08:00 SRC=172.16.2.185 DST=208.80.155.125 LEN=44 TOS=00 PREC=0x00 TTL=56 ID=60664 PROTO=TCP SPT=49649 DPT=1207 SEQ=4054780971 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0
Dec  2 07:13:15 cloudstore1008 ulogd[1910]: [fw-in-drop] IN=eno1 OUT= MAC=d0:94:66:26:d5:6a:5c:5e:ab:3d:87:c4:08:00 SRC=172.16.2.185 DST=208.80.155.125 LEN=44 TOS=00 PREC=0x00 TTL=35 ID=26501 PROTO=TCP SPT=49650 DPT=1207 SEQ=4054584360 ACK=0 WINDOW=1024 SYN URGP=0 MARK=0
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Dec  2 07:21:03 cloudstore1008 systemd-modules-load[1656]: Inserted module 'nf_conntrack'
Dec  2 07:21:03 cloudstore1008 systemd-modules-load[1656]: Inserted module 'ipmi_devintf'
Dec  2 07:21:03 cloudstore1008 systemd[1]: Started Load Kernel Modules.
Bstorm added a subscriber: Bstorm.Dec 2 2019, 3:50 PM

When it was rebooted, it filled the root disk due to lack of bind mounts. I'm going to add the silly bind mounts to fstab today.
That doesn't explain the above crash, though.

Aklapper added a project: Cloud-Services.Dec 2 2019, 3:55 PM
Bstorm added a comment.Dec 2 2019, 7:22 PM

I see the debugfs is mounted. Not sure why that is.

Bstorm added a comment.Dec 2 2019, 10:09 PM

Well, that was easy. Once I logged into the web console the error is right there 🙂

Screen Shot 2019-12-02 at 3.08.26 PM.png (190×1 px, 65 KB)

Bstorm renamed this task from cloudstore1008 crash to cloudstore1008 crash - Memory error.Dec 2 2019, 10:11 PM
Bstorm triaged this task as Medium priority.
Bstorm updated the task description. (Show Details)
Bstorm added a subscriber: wiki_willy.

This host is currently the passive partner in the cluster and can be power cycled for troubleshooting if it is downtimed and syncs stopped. Assigning to @wiki_willy to have DCOps take a look.

Bstorm added projects: cloud-services-team (Kanban), ops-eqiad.Dec 2 2019, 10:12 PM
Restricted Application added a project: SRE. · View Herald TranscriptDec 2 2019, 10:12 PM
Bstorm assigned this task to wiki_willy.Dec 2 2019, 10:13 PM

Please coordinate with me for restarts and downtime.

wiki_willy reassigned this task from wiki_willy to Jclark-ctr.Dec 2 2019, 11:34 PM
wiki_willy added subscribers: Jclark-ctr, • JHedden.

Re-assigning to @Jclark-ctr , who can coordinate with @JHedden during their weekly sync up on Tuesdays. Thanks, Willy

Jclark-ctr added a comment.Dec 3 2019, 1:40 AM

Confirmed: Service Request 1004932600 was successfully submitted.

Bstorm added a subtask: T239721: Set static bind mounts on cloudstore1008/9.Dec 3 2019, 3:52 PM
Bstorm closed subtask T239721: Set static bind mounts on cloudstore1008/9 as Resolved.Dec 3 2019, 4:04 PM
Jclark-ctr added a comment.Dec 4 2019, 10:09 PM

Dimm has arrived

Stashbot added a comment.Dec 4 2019, 10:39 PM

Mentioned in SAL (#wikimedia-operations) [2019-12-04T22:39:32Z] <bstorm_> powered off cloudstore1008, disabled sync from cloudstore1009, and downtimed both cloudstore1008 and cloudstore1009 for memory module replacement T239569

Jclark-ctr added a comment.Dec 4 2019, 10:44 PM

Finished replacement of DIMM_B2

Bstorm added a comment.Dec 4 2019, 11:21 PM

The server shows the right amount of RAM, so that's a good start. Checking logs on the web console

Bstorm closed this task as Resolved.Dec 4 2019, 11:23 PM
Bstorm added a parent task: T193655: rack/setup/install cloudstore1008 & cloudstore1009.

No errors for now. Hopefully, we don't see more pop up. Thanks!

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL