Page MenuHomePhabricator
Create Task
Maniphest T364006

Splunk On-Call resets overrides after changes to the escalation
Open, MediumPublic
Actions

Description

In T350192#9762987, @Volans wrote:

@lmata FYI today VO thought that Jaime and me were both oncall instead of the people we had swapped with. I fixed it for this week but also noticed that for all the existing overrides, the EMEA escalation routing have been unassigned. Possibly related to the May 1st batphone change for EMEA.
I didn't change the other overrides (that hence need fixing) and also wanted to brought up the issue because maybe the current procedure to put the batphone for a day might be the cause of this.

It seems the overrides were "reset" after adding back the EMEA rotation to the escalation, tracking tasks to follow up with Splunk and identify the issue/fix.

Related Objects

Event Timeline

lmata created this task.May 2 2024, 1:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 2 2024, 1:58 PM
lmata triaged this task as Medium priority.May 2 2024, 1:58 PM
lmata mentioned this in T350192: On-call batphone escalation configuration holidays FY2023-24.May 2 2024, 2:02 PM
fgiunchedi subscribed.May 10 2024, 9:19 AM

I've run into this today, what seems to happen is the following:

  • The escalation "SRE Business Hours (Escalation)" gets modified. Not sure if removing steps or leaving the steps unmodified and changing the rotations trigger the behavior
  • If the escalation was currently in an override, then the person taking the override gets reset and noone is overriding (the override itself stays in place)
lmata moved this task from Inbox to In progress on the SRE Observability (FY2023/2024-Q4) board.May 13 2024, 6:35 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits