Page MenuHomePhabricator

Host page did not auto-resolve in VO
Open, Needs TriagePublic

Description

A recent example of this behaviour below. The problem seems to be that incidents from host alerts (as opposed to service alerts) don't self-resolve when the recovery email comes in from icinga.

Incident 1227 involved cr3-eqsin being unpingable from alert1001: link to splunk oncall

The triggering email from the link above:

Critical: Host cr3-eqsin - PING - Packet loss = 100%
From: nagios@alert1001.wikimedia.org
Notification Type: PROBLEM Host: cr3-eqsin State: DOWN Address: 103.102.166.131 Info: PING CRITICAL - Packet loss = 100% Date/Time: Fri Jun 18 08:30:25 UTC 2021 Acknowledged by :

Alert Payload
Alert Data
Alert Fields
Show Null Fields (3)
Splunk On-Call Fields
agent	m
alert_received_time_utc	2021-06-18T08:30:36Z
alert_received_week_time_utc	2021-W24-5T08:30:36Z
alert_type	CRITICAL
api_key	redacted
entity_display_name	Host cr3-eqsin - PING  - Packet loss = 100%
entity_id	Host cr3-eqsin - PING  - Packet loss = 100%
entity_is_host	false
entity_state	CRITICAL
message_type	CRITICAL
monitor_name	nagios@alert1001.wikimedia.org
monitoring_tool	Email
NOTIFICATIONTYPE	CRITICAL
routing_key	icinga
sender	nagios@alert1001.wikimedia.org
SERVICESTATE	CRITICAL
state_message	Notification Type: PROBLEM
Host: cr3-eqsin
State: DOWN
Address: 103.102.166.131
Info: PING CRITICAL - Packet loss = 100%

Date/Time: Fri Jun 18 08:30:25 UTC 2021

Acknowledged by :
state_start_time	1624005036995
subject	PROBLEM Host cr3-eqsin - PING CRITICAL - Packet loss = 100%
timestamp	1624005036995
VO_ALERT_RCV_TIME	1624005036995
VO_ALERT_TYPE	SERVICE
VO_MONITOR_TYPE	8
VO_ORGANIZATION_ID	wikimedia
VO_UUID	079ca093-a3ce-42bc-8a5a-6d090aaf88a0

The alert was then acknowledged by Riccardo and finally resolved by me.

Icinga sending the alert and recovery emails:

Jun 18 08:30:25 alert1001 icinga: HOST NOTIFICATION: victorops;cr3-eqsin;DOWN;vo-host-notify-by-email;PING CRITICAL - Packet loss = 100%
Jun 18 08:31:03 alert1001 icinga: HOST NOTIFICATION: victorops;cr3-eqsin;UP;vo-host-notify-by-email;PING OK - Packet loss = 0%, RTA = 238.16 ms

And the recovery email looks sth like below, and I couldn't find it in the incident's timeline:

Delivered-To: fgiunchedi@wikimedia.org
Received: by 2002:a92:a812:0:0:0:0:0 with SMTP id o18csp1063253ilh;
        Fri, 18 Jun 2021 01:31:04 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJyic1JRYuI+9yMpiaWzn7qltxK7leWWMdA9SZWdxCJ4wXhClSNlY/bx4Uhku7FZZniXiVJT
X-Received: by 2002:ac8:4b42:: with SMTP id e2mr9376616qts.210.1624005064605;
        Fri, 18 Jun 2021 01:31:04 -0700 (PDT)
Return-Path: <root@wikimedia.org>
Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
        by mx.google.com with ESMTPS id j6si4555812qko.196.2021.06.18.01.31.04
        (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
        Fri, 18 Jun 2021 01:31:04 -0700 (PDT)
Received: from alert1001.wikimedia.org ([2620:0:861:3:208:80:154:88]:38128) by mx1001.wikimedia.org with esmtp (Exim 4.89) (envelope-from <root@wikimedia.org>) id 1lu9u3-0002vz-VC for alerts@wikimedia.org; Fri, 18 Jun 2021 08:31:03 +0000
Received: from nagios by alert1001.wikimedia.org with local (Exim 4.92) (envelope-from <root@wikimedia.org>) id 1lu9u3-0001iI-Ug for alerts@wikimedia.org; Fri, 18 Jun 2021 08:31:03 +0000
To: alerts@wikimedia.org
Subject: RECOVERY Host cr3-eqsin - PING OK - Packet loss = 0%, RTA = 238.16 ms
Reply-To: alerts@wikimedia.org
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Message-Id: <E1lu9u3-0001iI-Ug@alert1001.wikimedia.org>
From: nagios@alert1001.wikimedia.org
Date: Fri, 18 Jun 2021 08:31:03 +0000

Notification Type: RECOVERY
Host: cr3-eqsin
State: UP
Address: 103.102.166.131
Info: PING OK - Packet loss = 0%, RTA = 238.16 ms

Date/Time: Fri Jun 18 08:31:03 UTC 2021

Acknowledged by :

Older example

We got an host alert (labweb1002) that paged in VO but the recovery email didn't auto-resolve the related incident (possibly similar to T263423: librenms page didn't auto-resolve in VO)

Alert details

Alert Data

is_vo_ack	1
TIMET	1601307976737
Show Null Fields (0)
VictorOps Fields

ack_author	xionox
ack_msg	
agent	m
alert_received_time_utc	2020-09-28T15:41:00Z
alert_received_week_time_utc	2020-W40-1T15:41:00Z
alert_type	ACKNOWLEDGEMENT
api_key	redacted
entity_display_name	Host labweb1002 - PING  - Packet loss = 100%
entity_id	Host labweb1002 - PING  - Packet loss = 100%
entity_is_host	false
entity_state	CRITICAL
host_name	
INCIDENT_ID	511
message_type	ACKNOWLEDGEMENT
monitor_name	nagios@alert1001.wikimedia.org
monitoring_tool	Email
NOTIFICATIONTYPE	ACKNOWLEDGEMENT
routing_key	icinga
sender	nagios@alert1001.wikimedia.org
SERVICESTATE	CRITICAL
state_message	Notification Type: PROBLEM
Host: labweb1002
State: DOWN
Address: 208.80.155.109
Info: PING CRITICAL - Packet loss = 100%

Date/Time: Mon Sept 28 15:41:00 UTC 2020

Acknowledged by :
state_start_time	1601307976737
subject	PROBLEM Host labweb1002 - PING CRITICAL - Packet loss = 100%
timestamp	1601307976737
VO_ALERT_RCV_TIME	1601307976737
VO_ALERT_TYPE	SERVICE
VO_MONITOR_TYPE	8
VO_ORGANIZATION_ID	wikimedia
VO_UUID	9aa38dfc-8a82-4b8c-8530-bceba8d68ef3

Recovery details

Alert Data

Payload is empty

VictorOps Fields

ack_author	
ack_msg	
agent	m
alert_received_time_utc	2020-09-28T15:47:23Z
alert_received_week_time_utc	2020-W40-1T15:47:23Z
alert_type	RECOVERY
api_key	redacted
entity_display_name	Host labweb1002 - PING  - Packet loss = 0%, RTA = 0.22 ms
entity_id	Host labweb1002 - PING  - Packet loss = 0%, RTA = 0.22 ms
entity_is_host	false
entity_state	OK
host_name	
message_type	RECOVERY
monitor_name	nagios@alert1001.wikimedia.org
monitoring_tool	Email
NOTIFICATIONTYPE	RECOVERY
routing_key	icinga
sender	nagios@alert1001.wikimedia.org
SERVICESTATE	OK
state_message	Notification Type: RECOVERY
Host: labweb1002
State: UP
Address: 208.80.155.109
Info: PING OK - Packet loss = 0%, RTA = 0.22 ms

Date/Time: Mon Sept 28 15:47:22 UTC 2020

Acknowledged by :
state_start_time	1601308043663
subject	RECOVERY Host labweb1002 - PING OK - Packet loss = 0%, RTA = 0.22 ms
timestamp	1601308043663
VO_ALERT_RCV_TIME	1601308043663
VO_ALERT_TYPE	SERVICE
VO_MONITOR_TYPE	8
VO_ORGANIZATION_ID	wikimedia
VO_UUID	7e71bad7-d171-4f38-9edb-5adb5157c7a7

Event Timeline