Maniphest T263423

librenms page didn't auto-resolve in VO
Closed, InvalidPublic
Actions

Assigned To

None

Authored By

	fgiunchedi
	Sep 21 2020, 9:45 AM

Tags

Referenced Files

None

Subscribers

Description

A librenms alert caused a page to go out via VO, and the incident go ack'd. However the ack expired 24h later and the incident paged again. The alert in icinga looks like it recovered, so the related incident should have been resolved automatically in VO

You have 1 incident.

Incident: 491
State:    Critical
Service:  icinga1001/LibreNMS has a  alert #page
Message:  Notification Type: RECOVERY

Service: LibreNMS has a critical alert #page
Host: icinga1001
Address: 208.80.154.84
State: OK

Date/Time: Fri Sept 18 12:46:13 UTC 2020

Notes URLs: https://bit.ly/wmf-librenms

Acknowledged by :

Additional Info:

OK: zero critical LibreNMS alerts
Link:     https://portal.victorops.com/client/wikimedia#/incident/491/incidentTimeline

Related Objects

Mentioned In: T266570: Two close pages for idle workers api + appserver didn't auto-resolve on recovery
T264016: Host page did not auto-resolve in VO
Mentioned Here: T264016: Host page did not auto-resolve in VO

Event Timeline

fgiunchedi created this task.Sep 21 2020, 9:45 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 21 2020, 9:45 AM

• crusnov triaged this task as Medium priority.Sep 22 2020, 11:18 PM

fgiunchedi mentioned this in T264016: Host page did not auto-resolve in VO.Sep 28 2020, 4:01 PM

• lmata moved this task from Inbox to Backlog on the observability board.Oct 26 2020, 3:42 PM

herron subscribed.Feb 8 2021, 4:03 PM

RhinosF1 subscribed.Feb 17 2021, 5:10 PM

• lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM

• lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:08 AM

• lmata edited projects, added Observability-Alerting; removed SRE Observability.Aug 9 2021, 3:32 AM

• lmata moved this task from Inbox to Backlog on the Observability-Alerting board.Aug 10 2021, 3:18 PM

andrea.denisse mentioned this in T266570: Two close pages for idle workers api + appserver didn't auto-resolve on recovery.Jan 10 2025, 12:45 AM

I don't think we've seen a recorrence of this problem, and we fixed the host-related recoveries in T264016: Host page did not auto-resolve in VO