It's worth noting that essentially the same happened also happened today during the late EMEA shift of @Raine and myself: https://portal.victorops.com/ui/wikimedia/incident/8123/details paged again 24 hours after the originally and then about an hour later the same happened for https://portal.victorops.com/ui/wikimedia/incident/8135/details and https://portal.victorops.com/ui/wikimedia/incident/8134/details (but a least it was all during normal work hours)

Resolved problems should always resolve the issue in victor ops.

Agreed. This is the ideal way things should work.

With our low volume of paging alerts, maybe we should disable the "ACK" button, and only allow to click "resolve".

I personally think there is some merit in that, in my mind it is resolving the page. I think Victorops is only needed for the escalation bit, we don't need it to track the actual issue status.

To provide some context/history: the "re-page on acked but not resolved incidents" is a VO setting which we set to 24h and can be disabled (cfr T259465: VictorOps behavior on long-ack'd incidents). I am +1 on changing the behavior to not re-page

On the specifics on why exactly this happens, speaking as a former o11y member, I'm not going to look deeper into icinga-issued pages because IMHO we shouldn't be doing that anymore in the first place.

https://portal.victorops.com/ui/wikimedia/incident/8123/details is interesting though, PyBalBGPUnstable indeed fired 3x, and I can't find the recovery in alert logs https://logstash.wikimedia.org/goto/ab4f05cce372d6e2e55aeae0b5aaaa65

In T431030#12084106, @fgiunchedi wrote:

https://portal.victorops.com/ui/wikimedia/incident/8123/details is interesting though, PyBalBGPUnstable indeed fired 3x, and I can't find the recovery in alert logs https://logstash.wikimedia.org/goto/ab4f05cce372d6e2e55aeae0b5aaaa65

If it's relevant the initial page should not have happened, it was during a planned works and not service affecting. I had forgotten to downtime the LVS hosts.

After it paged I downtimed the hosts (as further recovery/down transitions were expected). So by the time the RECOVERY came in the hosts were downtimed, not sure if that would affect the alerting/recovery.

In T431030#12084102, @fgiunchedi wrote:

To provide some context/history: the "re-page on acked but not resolved incidents" is a VO setting which we set to 24h and can be disabled (cfr T259465: VictorOps behavior on long-ack'd incidents). I am +1 on changing the behavior to not re-page

Agreed, we should do this.

The alerts have been acknowledged (and then silenced) in Alertmanager: 38e40dbc-0d5b-4267-ad1f-338d2e651a23 (https://github.com/prometheus/alertmanager/issues/226).

Adding changing the setting as an item for the Monday meeting just to get wider awareness, will change after that.

I disagree with the idea that we can solve a page rather than acknowledge it. It means that at that point, we cant unacknowledge a page, to let it escalate; nor we can receive a clear notification on our pager when the issue is resolved.

Sometimes issues self-resolve in a few minutes, and our oncall response time is 30 minutes. Say the other person oncall responds - right now I know from my pager that the issue is not resolved, and I know I am needed back at a computer - or, it it is resolved, it means I am not needed and I can go on with my life.

I support the idea of not re-paging

+1 to ack and resolve being two different things, for exactly the reasons @Joe stated.

In T431030#12085591, @hnowlan wrote:

Adding changing the setting as an item for the Monday meeting just to get wider awareness, will change after that.

This setting has been disabled - acked incidents will no longer re-page after 24 hours.