Page MenuHomePhabricator

mw2269 rebooted/crashed unexpectedly on Jul 17th ~15:30UTC
Closed, ResolvedPublic

Description

mw2269 rebooted today, perhaps a crash?

syslog stop at 15:21 only to resume at 15:40

Jul 17 15:21:10 mw2269 rsyslog_exporter[15337]: 2019/07/17 15:21:10 error handling stats line: unknown pstat type: 0, line was: 2019-07-17T15:21:10.177968+00:00 mw2269 rsyslogd-pstats: { "name": "imudp(w0)", "origin": "imudp", "called.recvmmsg": 2, "called.recvmsg": 0, "msgs.received": 1 }
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Jul 17 15:40:30 mw2269 systemd-modules-load[650]: Inserted module 'nf_conntrack'
Jul 17 15:40:30 mw2269 systemd-modules-load[650]: Inserted module 'ipmi_devintf'

Grafana is not particularly telling either https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=mw2269&var-datasource=codfw%20prometheus%2Fops&var-cluster=appserver&from=1563375288506&to=1563378959652 but there is a 15m gap in some graphs

Event Timeline

akosiaris triaged this task as Medium priority.Jul 17 2019, 3:59 PM
akosiaris added a project: SRE.
Marostegui subscribed.

I am calling this resolved as it's been almost 10 months there is no way to debug this anymore :(
Maybe it was a one time thing, if it happens again we can reopen!