Page MenuHomePhabricator
Paste P7010

Duplicate Icinga IRC logs
ActivePublic

Authored by bearND on Apr 18 2018, 7:36 PM.
Tags
None
Referenced Files
F17096898: Duplicate Icinga IRC logs
Apr 18 2018, 7:36 PM
Subscribers
PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
On April 18, 2018 All times in MDT
1:16 PM
1:20 PM
1:23 PM
1:28 PM
~~~~
<icinga-wm> IRC echo bot PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:16 PM PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:18 PM PROBLEM - mobileapps endpoints health on scb2003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:18 PM PROBLEM - mobileapps endpoints health on scb2005 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:18 PM PROBLEM - mobileapps endpoints health on scb2004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:18 PM → closedmouth joined (mouthy@wikipedia/closedmouth)
1:19 PM <mutante> Accion Mutante notice how that's a WARN and not a CRIT but icinga-wm reports it , interesting
1:20 PM <mobrovac> Marko Obrovac this is really weird ^
1:20 PM bearND: mdholloway: ^ ?
1:20 PM <icinga-wm> IRC echo bot PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:20 PM PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:21 PM <mobrovac> Marko Obrovac mutante: and that's good, because it's not a critical :)
1:21 PM <bearND> Bernd but why does Icinga say critical in this case?
1:22 PM <mutante> Accion Mutante mobrovac: it's good, i was trying to solve that people have these permissions
1:22 PM i just thought that icinga-wm was filtering out WARNS
1:22 PM that's also not bad, i just expected it to behave differently ..slightly
1:23 PM <icinga-wm> IRC echo bot PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:23 PM PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:23 PM <mobrovac> Marko Obrovac oh right, but mutante the host is critical, but the check is warn
1:23 PM haha
1:23 PM icinga fail
1:23 PM <mutante> Accion Mutante bearND: it's defined somewhere in the check_plugin script
1:23 PM or puppet
1:23 PM <bearND> Bernd why did it just announce the same two hosts with same message? scb1003 + scb1004
1:24 PM <mobrovac> Marko Obrovac there might be a mismatch between the service-checker script and nagios/icinga
1:24 PM <mutante> Accion Mutante because they are separate issues from Icinga's point of view
1:24 PM it's not configured to know these should have a relationship
1:24 PM <mobrovac> Marko Obrovac no, it's the same fail
1:24 PM <mutante> Accion Mutante service on host A != service on host B
1:25 PM if the entire host is down that would be CRIT more than just a service on it.. that seems to make sense?
1:25 PM <bearND> Bernd mutante: it said it for the same hosts at least twice, see :20 and :23
1:26 PM <icinga-wm> IRC echo bot PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/page/summary/{title}{/revision}{/tid} (Get summary for Manitowoc, Wisconsin) timed out before a response was received: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:26 PM PROBLEM - restbase endpoints health on restbase1010 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
1:26 PM <bearND> Bernd ok, this one^ is different.
1:26 PM <mutante> Accion Mutante bearND: can you paste 2 identical lines?
1:26 PM not sure if we are talking about the same
1:27 PM <bearND> Bernd mutante: "PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0"
1:27 PM <icinga-wm> IRC echo bot PROBLEM - restbase endpoints health on restbase1012 is CRITICAL: /en.wikipedia.org/v1/feed/onthisday/{type}/{mm}/{dd} (Retrieve selected the events for Jan 01) timed out before a response was received
1:28 PM PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:28 PM PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0
1:28 PM RECOVERY - restbase endpoints health on restbase1010 is OK: All endpoints are healthy
1:28 PM RECOVERY - restbase endpoints health on restbase1012 is OK: All endpoints are healthy
1:28 PM <bearND> Bernd mutante: actually happened four times in last 12 minutes
1:28 PM <•wikibugs> Wikibugs v2.1, https://tools.wmflabs.org/wikibugs/ (CR) Paladox: [C: 1] Gerrit: Disable auto-reindexing of changes [puppet] - https://gerrit.wikimedia.org/r/427471 (owner: Chad)
1:28 PM <•logmsgbot> !log ppchelko@tin Finished deploy [restbase/deploy@8d8f1df]: Test concurrent worker startups (duration: 15m 23s)
1:28 PM <•stashbot> https://tools.wmflabs.org/stashbot/ Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
1:29 PM <mutante> Accion Mutante could you make a pastebin with those lines including the timestamps?
1:30 PM yes, it is normal that it would repeat ongoing issues at the check interval