A recent example of this behaviour below. The problem seems to be that incidents from **host** alerts (as opposed to **service** alerts) don't self-resolve when the recovery email comes in from icinga.
Incident 1227 involved cr3-eqsin being unpingable from alert1001: [[ https://portal.victorops.com/ui/wikimedia/incident/1227/timeline | link to splunk oncall ]]
The triggering email from the link above:
```
Critical: Host cr3-eqsin - PING - Packet loss = 100%
From: nagios@alert1001.wikimedia.org
Notification Type: PROBLEM Host: cr3-eqsin State: DOWN Address: 103.102.166.131 Info: PING CRITICAL - Packet loss = 100% Date/Time: Fri Jun 18 08:30:25 UTC 2021 Acknowledged by :
Alert Payload
Alert Data
Alert Fields
Show Null Fields (3)
Splunk On-Call Fields
agent m
alert_received_time_utc 2021-06-18T08:30:36Z
alert_received_week_time_utc 2021-W24-5T08:30:36Z
alert_type CRITICAL
api_key redacted
entity_display_name Host cr3-eqsin - PING - Packet loss = 100%
entity_id Host cr3-eqsin - PING - Packet loss = 100%
entity_is_host false
entity_state CRITICAL
message_type CRITICAL
monitor_name nagios@alert1001.wikimedia.org
monitoring_tool Email
NOTIFICATIONTYPE CRITICAL
routing_key icinga
sender nagios@alert1001.wikimedia.org
SERVICESTATE CRITICAL
state_message Notification Type: PROBLEM
Host: cr3-eqsin
State: DOWN
Address: 103.102.166.131
Info: PING CRITICAL - Packet loss = 100%
Date/Time: Fri Jun 18 08:30:25 UTC 2021
Acknowledged by :
state_start_time 1624005036995
subject PROBLEM Host cr3-eqsin - PING CRITICAL - Packet loss = 100%
timestamp 1624005036995
VO_ALERT_RCV_TIME 1624005036995
VO_ALERT_TYPE SERVICE
VO_MONITOR_TYPE 8
VO_ORGANIZATION_ID wikimedia
VO_UUID 079ca093-a3ce-42bc-8a5a-6d090aaf88a0
```
The alert was then acknowledged by Riccardo and finally resolved by me.
Icinga sending the alert and recovery emails:
```
Jun 18 08:30:25 alert1001 icinga: HOST NOTIFICATION: victorops;cr3-eqsin;DOWN;vo-host-notify-by-email;PING CRITICAL - Packet loss = 100%
Jun 18 08:31:03 alert1001 icinga: HOST NOTIFICATION: victorops;cr3-eqsin;UP;vo-host-notify-by-email;PING OK - Packet loss = 0%, RTA = 238.16 ms
```
And the recovery email looks sth like below, and I couldn't find it in the incident's timeline:
```
Delivered-To: fgiunchedi@wikimedia.org
Received: by 2002:a92:a812:0:0:0:0:0 with SMTP id o18csp1063253ilh;
Fri, 18 Jun 2021 01:31:04 -0700 (PDT)
X-Google-Smtp-Source: ABdhPJyic1JRYuI+9yMpiaWzn7qltxK7leWWMdA9SZWdxCJ4wXhClSNlY/bx4Uhku7FZZniXiVJT
X-Received: by 2002:ac8:4b42:: with SMTP id e2mr9376616qts.210.1624005064605;
Fri, 18 Jun 2021 01:31:04 -0700 (PDT)
Return-Path: <root@wikimedia.org>
Received: from mx1001.wikimedia.org (mx1001.wikimedia.org. [208.80.154.76])
by mx.google.com with ESMTPS id j6si4555812qko.196.2021.06.18.01.31.04
(version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256);
Fri, 18 Jun 2021 01:31:04 -0700 (PDT)
Received: from alert1001.wikimedia.org ([2620:0:861:3:208:80:154:88]:38128) by mx1001.wikimedia.org with esmtp (Exim 4.89) (envelope-from <root@wikimedia.org>) id 1lu9u3-0002vz-VC for alerts@wikimedia.org; Fri, 18 Jun 2021 08:31:03 +0000
Received: from nagios by alert1001.wikimedia.org with local (Exim 4.92) (envelope-from <root@wikimedia.org>) id 1lu9u3-0001iI-Ug for alerts@wikimedia.org; Fri, 18 Jun 2021 08:31:03 +0000
To: alerts@wikimedia.org
Subject: RECOVERY Host cr3-eqsin - PING OK - Packet loss = 0%, RTA = 238.16 ms
Reply-To: alerts@wikimedia.org
MIME-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Message-Id: <E1lu9u3-0001iI-Ug@alert1001.wikimedia.org>
From: nagios@alert1001.wikimedia.org
Date: Fri, 18 Jun 2021 08:31:03 +0000
Notification Type: RECOVERY
Host: cr3-eqsin
State: UP
Address: 103.102.166.131
Info: PING OK - Packet loss = 0%, RTA = 238.16 ms
Date/Time: Fri Jun 18 08:31:03 UTC 2021
Acknowledged by :
```
=== Older example ===
We got an host alert (labweb1002) that paged in VO but the recovery email didn't auto-resolve the related incident (possibly similar to {T263423})
Alert details
```
Alert Data
is_vo_ack 1
TIMET 1601307976737
Show Null Fields (0)
VictorOps Fields
ack_author xionox
ack_msg
agent m
alert_received_time_utc 2020-09-28T15:41:00Z
alert_received_week_time_utc 2020-W40-1T15:41:00Z
alert_type ACKNOWLEDGEMENT
api_key redacted
entity_display_name Host labweb1002 - PING - Packet loss = 100%
entity_id Host labweb1002 - PING - Packet loss = 100%
entity_is_host false
entity_state CRITICAL
host_name
INCIDENT_ID 511
message_type ACKNOWLEDGEMENT
monitor_name nagios@alert1001.wikimedia.org
monitoring_tool Email
NOTIFICATIONTYPE ACKNOWLEDGEMENT
routing_key icinga
sender nagios@alert1001.wikimedia.org
SERVICESTATE CRITICAL
state_message Notification Type: PROBLEM
Host: labweb1002
State: DOWN
Address: 208.80.155.109
Info: PING CRITICAL - Packet loss = 100%
Date/Time: Mon Sept 28 15:41:00 UTC 2020
Acknowledged by :
state_start_time 1601307976737
subject PROBLEM Host labweb1002 - PING CRITICAL - Packet loss = 100%
timestamp 1601307976737
VO_ALERT_RCV_TIME 1601307976737
VO_ALERT_TYPE SERVICE
VO_MONITOR_TYPE 8
VO_ORGANIZATION_ID wikimedia
VO_UUID 9aa38dfc-8a82-4b8c-8530-bceba8d68ef3
```
Recovery details
```
Alert Data
Payload is empty
VictorOps Fields
ack_author
ack_msg
agent m
alert_received_time_utc 2020-09-28T15:47:23Z
alert_received_week_time_utc 2020-W40-1T15:47:23Z
alert_type RECOVERY
api_key redacted
entity_display_name Host labweb1002 - PING - Packet loss = 0%, RTA = 0.22 ms
entity_id Host labweb1002 - PING - Packet loss = 0%, RTA = 0.22 ms
entity_is_host false
entity_state OK
host_name
message_type RECOVERY
monitor_name nagios@alert1001.wikimedia.org
monitoring_tool Email
NOTIFICATIONTYPE RECOVERY
routing_key icinga
sender nagios@alert1001.wikimedia.org
SERVICESTATE OK
state_message Notification Type: RECOVERY
Host: labweb1002
State: UP
Address: 208.80.155.109
Info: PING OK - Packet loss = 0%, RTA = 0.22 ms
Date/Time: Mon Sept 28 15:47:22 UTC 2020
Acknowledged by :
state_start_time 1601308043663
subject RECOVERY Host labweb1002 - PING OK - Packet loss = 0%, RTA = 0.22 ms
timestamp 1601308043663
VO_ALERT_RCV_TIME 1601308043663
VO_ALERT_TYPE SERVICE
VO_MONITOR_TYPE 8
VO_ORGANIZATION_ID wikimedia
VO_UUID 7e71bad7-d171-4f38-9edb-5adb5157c7a7
```