Page MenuHomePhabricator

Investigate db1061 crash
Closed, ResolvedPublic

Description

It crashed/become unresponsive at 2016-09-19 01:13:40 UTC. Probably unrelated to T145533, as it is different vendor, model and batch.

Serial console was unresponsive, and no signs of kernel panic at the moment or before the crash.

No action is required from dc ops.

Event Timeline

Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald Transcript
jcrespo triaged this task as Medium priority.
jcrespo moved this task from Triage to In progress on the DBA board.

Accoording to lifecycle logs "System is turning off." is the cause of the issue. No logs. No signs of a crash. Logs continue as usual:

Sep 19 01:09:39 db1061 sshd[47512]: Set /proc/self/oom_score_adj to 0
Sep 19 01:09:39 db1061 sshd[47512]: Connection from 208.80.154.14 port 32646 on 10.64.48.14 port 22
Sep 19 01:09:39 db1061 sshd[47512]: Connection closed by 208.80.154.14 [preauth]
Sep 19 10:25:19 db1061 CRON[1261]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 19 10:25:19 db1061 sshd[1264]: Set /proc/self/oom_score_adj from 0 to -1000
Sep 19 10:25:19 db1061 sshd[1264]: Server listening on 0.0.0.0 port 22.
Sep 19 10:25:19 db1061 sshd[1264]: Server listening on :: port 22.
Sep 19 01:02:01 db1061 CRON[45477]: (root) CMD (/usr/local/sbin/puppet-run > /dev/null 2>&1)
Sep 19 01:02:06 db1061 puppet-agent[45551]: Sleeping for 3 seconds (splay is enabled)
Sep 19 01:02:09 db1061 puppet-agent[45551]: Retrieving pluginfacts
Sep 19 01:02:09 db1061 puppet-agent[45551]: Retrieving plugin
Sep 19 01:02:10 db1061 puppet-agent[45551]: Loading facts
Sep 19 01:02:15 db1061 puppet-agent[45551]: Caching catalog for db1061.eqiad.wmnet
Sep 19 01:02:16 db1061 puppet-agent[45551]: Applying configuration version '1474246647'
Sep 19 01:02:31 db1061 puppet-agent[45551]: Finished catalog run in 15.08 seconds
Sep 19 01:05:01 db1061 CRON[47136]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 19 10:25:19 db1061 rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="1335" x-info="http://www.rsyslog.com"] start
Sep 19 10:25:19 db1061 systemd[1]: Started Apply Kernel Variables.

I think there is not much left to do here, except wait if it happens again.