PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 On April 18, 2018 All times in MDT 1:16 PM 1:20 PM 1:23 PM 1:28 PM ~~~~ IRC echo bot PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:16 PM PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:18 PM PROBLEM - mobileapps endpoints health on scb2003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:18 PM PROBLEM - mobileapps endpoints health on scb2005 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:18 PM PROBLEM - mobileapps endpoints health on scb2004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:18 PM → closedmouth joined (mouthy@wikipedia/closedmouth) 1:19 PM Accion Mutante notice how that's a WARN and not a CRIT but icinga-wm reports it , interesting 1:20 PM Marko Obrovac this is really weird ^ 1:20 PM bearND: mdholloway: ^ ? 1:20 PM IRC echo bot PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:20 PM PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:21 PM Marko Obrovac mutante: and that's good, because it's not a critical :) 1:21 PM Bernd but why does Icinga say critical in this case? 1:22 PM Accion Mutante mobrovac: it's good, i was trying to solve that people have these permissions 1:22 PM i just thought that icinga-wm was filtering out WARNS 1:22 PM that's also not bad, i just expected it to behave differently ..slightly 1:23 PM IRC echo bot PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:23 PM PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:23 PM Marko Obrovac oh right, but mutante the host is critical, but the check is warn 1:23 PM haha 1:23 PM icinga fail 1:23 PM Accion Mutante bearND: it's defined somewhere in the check_plugin script 1:23 PM or puppet 1:23 PM Bernd why did it just announce the same two hosts with same message? scb1003 + scb1004 1:24 PM Marko Obrovac there might be a mismatch between the service-checker script and nagios/icinga 1:24 PM Accion Mutante because they are separate issues from Icinga's point of view 1:24 PM it's not configured to know these should have a relationship 1:24 PM Marko Obrovac no, it's the same fail 1:24 PM Accion Mutante service on host A != service on host B 1:25 PM if the entire host is down that would be CRIT more than just a service on it.. that seems to make sense? 1:25 PM Bernd mutante: it said it for the same hosts at least twice, see :20 and :23 1:26 PM IRC echo bot PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/page/summary/{title}{/revision}{/tid} (Get summary for Manitowoc, Wisconsin) timed out before a response was received: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:26 PM PROBLEM - restbase endpoints health on restbase1010 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received 1:26 PM Bernd ok, this one^ is different. 1:26 PM Accion Mutante bearND: can you paste 2 identical lines? 1:26 PM not sure if we are talking about the same 1:27 PM Bernd mutante: "PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0" 1:27 PM IRC echo bot PROBLEM - restbase endpoints health on restbase1012 is CRITICAL: /en.wikipedia.org/v1/feed/onthisday/{type}/{mm}/{dd} (Retrieve selected the events for Jan 01) timed out before a response was received 1:28 PM PROBLEM - mobileapps endpoints health on scb1003 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:28 PM PROBLEM - mobileapps endpoints health on scb1004 is CRITICAL: /{domain}/v1/feed/announcements (Retrieve announcements) is WARNING: Test Retrieve announcements responds with unexpected value at path /announce = Expected 1 array elements, gotten 0 1:28 PM RECOVERY - restbase endpoints health on restbase1010 is OK: All endpoints are healthy 1:28 PM RECOVERY - restbase endpoints health on restbase1012 is OK: All endpoints are healthy 1:28 PM Bernd mutante: actually happened four times in last 12 minutes 1:28 PM <•wikibugs> Wikibugs v2.1, https://tools.wmflabs.org/wikibugs/ (CR) Paladox: [C: 1] Gerrit: Disable auto-reindexing of changes [puppet] - https://gerrit.wikimedia.org/r/427471 (owner: Chad) 1:28 PM <•logmsgbot> !log ppchelko@tin Finished deploy [restbase/deploy@8d8f1df]: Test concurrent worker startups (duration: 15m 23s) 1:28 PM <•stashbot> https://tools.wmflabs.org/stashbot/ Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log 1:29 PM Accion Mutante could you make a pastebin with those lines including the timestamps? 1:30 PM yes, it is normal that it would repeat ongoing issues at the check interval