Page MenuHomePhabricator
Paste P25274

Icinga alerts for ml-server1005 and elastic1089-1005
ActivePublic
Actions

Authored by cmooney on Apr 19 2022, 8:06 AM.
Tags
None
Referenced Files
F35057802: Icinga alerts for ml-server1005 and elastic1089-1005
Apr 19 2022, 8:18 AM
F35057790: Icinga alerts for ml-server1005 and elastic1089-1005
Apr 19 2022, 8:06 AM
Subscribers
None
Alerts first fired on the morning of Friday April 8th, following their re-image being kicked off late on the 7th (see T299609 and T294949):
Apr 8 04:25:12 alert1001 icinga: HOST ALERT: ml-serve1005;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 08:17:14 alert1001 icinga: HOST ALERT: elastic1089;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 08:22:32 alert1001 icinga: HOST ALERT: elastic1091;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 08:22:48 alert1001 icinga: HOST ALERT: elastic1092;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 08:23:14 alert1001 icinga: HOST ALERT: elastic1090;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 08:27:14 alert1001 icinga: HOST ALERT: elastic1094;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 08:40:23 alert1001 icinga: HOST ALERT: elastic1095;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 08:47:37 alert1001 icinga: HOST ALERT: elastic1096;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 09:01:18 alert1001 icinga: HOST ALERT: elastic1101;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 09:01:30 alert1001 icinga: HOST ALERT: elastic1100;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
Apr 8 09:02:26 alert1001 icinga: HOST ALERT: elastic1102;DOWN;HARD;2;PING CRITICAL - Packet loss = 100%
They remained in this state until the mac-ip table was cleared on associated top-of-rack switches on April 19th:
Apr 19 07:32:31 alert1001 icinga: HOST ALERT: elastic1089;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.52 ms
Apr 19 07:32:59 alert1001 icinga: HOST ALERT: elastic1092;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.32 ms
Apr 19 07:32:59 alert1001 icinga: HOST ALERT: elastic1091;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.28 ms
Apr 19 07:33:05 alert1001 icinga: HOST ALERT: elastic1090;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.27 ms
Apr 19 07:33:09 alert1001 icinga: HOST ALERT: elastic1094;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.23 ms
Apr 19 07:33:09 alert1001 icinga: HOST ALERT: ml-serve1005;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.29 ms
Apr 19 07:33:37 alert1001 icinga: HOST ALERT: elastic1096;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.47 ms
Apr 19 07:34:05 alert1001 icinga: HOST ALERT: elastic1095;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.42 ms
Apr 19 07:34:27 alert1001 icinga: HOST ALERT: elastic1101;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 3.31 ms
Apr 19 07:34:57 alert1001 icinga: HOST ALERT: elastic1102;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.91 ms
Apr 19 07:34:57 alert1001 icinga: HOST ALERT: elastic1100;UP;HARD;1;PING OK - Packet loss = 0%, RTA = 0.27 ms
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL