Alert in Icinga about NFS being down (alert history).
$ uptime 12:04:20 up 2:35, 1 user, load average: 0.30, 0.45, 0.57 $ last gtirloni pts/0 bast1002.wikimed Sat Mar 2 11:50 still logged in reboot system boot 4.9.0-0.bpo.8-am Sat Mar 2 09:29 - 12:04 (02:35) wtmp begins Sat Mar 2 09:29:01 2019
Lots of garbage before the reboot but no specific reason in /var/log/syslog:
Mar 2 09:21:01 labstore1006 CRON[20317]: (dumpsgen) CMD (bash -c '/usr/bin/rsync -rt --chmod=go-w stat1007.eqiad.wmnet::srv/dumps/pagecounts-ez/ /srv/dumps/xmldatadumps/public/other/pagecounts-ez/') Mar 2 09:21:01 labstore1006 CRON[20318]: (prometheus) CMD (/usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom) Mar 2 09:22:01 labstore1006 CRON[20444]: (prometheus) CMD (/usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom) Mar 2 09:23:01 labstore1006 CRON[20585]: (prometheus) CMD (/usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom) ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Mar 2 09:29:09 labstore1006 systemd-modules-load[356]: Inserted module 'nf_conntrack' Mar 2 09:29:09 labstore1006 systemd-modules-load[356]: Inserted module 'fuse' Mar 2 09:29:09 labstore1006 systemd-modules-load[356]: Inserted module 'ipmi_devintf' Mar 2 09:29:09 labstore1006 systemd[1]: Starting Create Static Device Nodes in /dev...
/var/log/auth.log:
Mar 2 09:23:16 labstore1006 sudo: nagios : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/lib/nagios/plugins/check_ferm Mar 2 09:23:16 labstore1006 sudo: pam_unix(sudo:session): session opened for user root by (uid=0) Mar 2 09:23:16 labstore1006 sudo: pam_unix(sudo:session): session closed for user root ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Mar 2 09:29:09 labstore1006 CRON[665]: pam_unix(cron:session): session opened for user root by (uid=0) Mar 2 09:29:09 labstore1006 systemd-logind[669]: New seat seat0. Mar 2 09:29:09 labstore1006 systemd-logind[669]: Watching system buttons on /dev/input/event0 (Power Button)
/var/log/daemon:
Mar 2 09:19:10 labstore1006 systemd[1]: Starting Time & Date Service... Mar 2 09:19:10 labstore1006 dbus[674]: [system] Successfully activated service 'org.freedesktop.timedate1' Mar 2 09:19:10 labstore1006 systemd-timedated[20092]: /etc/localtime should be a symbolic link to a time zone data file in /usr/share/zoneinfo/. Mar 2 09:19:10 labstore1006 systemd[1]: Started Time & Date Service. Mar 2 09:29:09 labstore1006 systemd-modules-load[356]: Inserted module 'nf_conntrack' Mar 2 09:29:09 labstore1006 systemd-modules-load[356]: Inserted module 'fuse' Mar 2 09:29:09 labstore1006 systemd-modules-load[356]: Inserted module 'ipmi_devintf'
/var/log/kern.log:
Feb 28 06:25:03 labstore1006 kernel: [7478114.408796] Process accounting resumed Mar 1 06:25:04 labstore1006 kernel: [7564515.091137] Process accounting resumed Mar 2 06:25:03 labstore1006 kernel: [7650914.361789] Process accounting resumed Mar 2 09:29:09 labstore1006 kernel: [ 0.000000] microcode: microcode updated early to revision 0xb00002e, date = 2018-04-19 Mar 2 09:29:09 labstore1006 kernel: [ 0.000000] Linux version 4.9.0-0.bpo.8-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10+deb8u1) ) #1 SMP Debian 4.9.110-3+deb9u5~deb8u1 (2018-10-03)
/var/log/messages:
Mar 2 09:14:41 labstore1006 puppet-agent-cronjob: INFO:debmonitor:Found 4 upgradable binary packages (including new dependencies) Mar 2 09:14:41 labstore1006 puppet-agent-cronjob: INFO:debmonitor:Successfully sent the upgradable update to the DebMonitor server Mar 2 09:29:09 labstore1006 kernel: [ 0.000000] microcode: microcode updated early to revision 0xb00002e, date = 2018-04-19 Mar 2 09:29:09 labstore1006 kernel: [ 0.000000] Linux version 4.9.0-0.bpo.8-amd64 (debian-kernel@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10+deb8u1) ) #1 SMP Debian 4.9.110-3+deb9u5~deb8 u1 (2018-10-03)
Last firmware messages:
/system1/log1/record60
Targets
Properties
number=60
severity=Caution
date=[NOT SET]
time=
description=Option ROM POST Error: bay: 11 (SAS)
/system1/log1/record61
Targets
Properties
number=61
severity=Caution
date=[NOT SET]
time=
description=Option ROM POST Error: bay: 12 (SAS)
/system1/log1/record62
Targets
Properties
number=62
severity=Caution
date=03/02/2019
time=09:27
description=Option ROM POST Error: 1719-Slot 1 Drive Array - A controller failure event occurred prior to this power-up. (Previous lock up code = 0x12) Action: Install the latest controller firmware. If the problem persists, replace the controller.
/system1/log1/record63
Targets
Properties
number=63
severity=Caution
date=[NOT SET]
time=
description=Option ROM POST Error: 1719-Slot 3 Drive Array - A controller failure event occurred prior to this power-up. (Previous lock up code = 0x12) Action: Install the latest controller firmware. If the problem persists, replace the controller.