Page MenuHomePhabricator
Create Task
Maniphest T313228

Deploy Dispatch for SRE incident workflow automation
Closed, DeclinedPublic
Actions

Description

Incident response workflow automation has been a source of discomfort for SREs. Our current manual process consists of creating documents and updating several chats and data sources in addition to troubleshooting. This effort aims to reduce friction for SRE responders and incident coordinators during incidents and free up much-needed time to focus on addressing the problems.

We have selected Netflix's Dispatch as our tool of choice for this project. https://github.com/Netflix/dispatch

This is the parent tracking task

In rough sequence:

Related Objects
Search...

Event Timeline

herron triaged this task as Medium priority.Jul 18 2022, 2:21 PM
herron created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 18 2022, 2:21 PM
herron added subtasks: T309033: Set up POC Dispatch environment and evaluate its viability, T313229: Production Dispatch Infrastructure, T313230: Dispatch IRC Integration, T313231: Dispatch Google workspace integration, T313232: Dispatch Documentation, T313233: Dispatch Splunk On-call / VictorOps Integration, T313234: Dispatch Phabricator Integration.Jul 18 2022, 2:23 PM
herron mentioned this in T309033: Set up POC Dispatch environment and evaluate its viability.
herron updated the task description. (Show Details)Jul 18 2022, 2:30 PM
colewhite subscribed.Jul 21 2022, 2:30 PM
andrea.denisse subscribed.Jul 21 2022, 2:33 PM
lmata subscribed.Jul 21 2022, 4:16 PM

The team met today and spent some time planning we made an initial decision to use WMCS will be our initial deployment target; this will allow us to move fast and keep some reasonable separation from production.

We can, after a quarter or two of using the tool, discuss another hosting target if we desire a better OOB solution.

lmata moved this task from Inbox to In progress on the SRE Observability (FY2022/2023-Q1) board.Jul 21 2022, 4:16 PM
herron mentioned this in T313229: Production Dispatch Infrastructure.Jul 21 2022, 6:12 PM
herron updated the task description. (Show Details)
lmata mentioned this in T313958: Evaluate viable candidates for incident paging.Jul 27 2022, 8:14 PM
lmata renamed this task from Deploy Dispatch for SRE incident tracking to Deploy Dispatch for SRE incident workflow automation.Aug 9 2022, 2:30 PM
lmata changed the task status from Open to In Progress.
lmata assigned this task to herron.
lmata raised the priority of this task from Medium to Needs Triage.
lmata updated the task description. (Show Details)
lmata triaged this task as High priority.Aug 9 2022, 2:35 PM

This task our team's top priority for the quarter.

herron closed subtask T309033: Set up POC Dispatch environment and evaluate its viability as Resolved.Aug 9 2022, 2:41 PM
BTullis subscribed.Aug 22 2022, 4:18 PM
Volans subscribed.Sep 7 2022, 7:54 AM
colewhite added a project: Incident Tooling.Sep 13 2022, 4:28 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q2); removed SRE Observability (FY2022/2023-Q1).Oct 28 2022, 6:28 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2022/2023-Q2) board.Nov 8 2022, 4:14 PM
JMeybohm subscribed.Dec 20 2022, 4:40 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q3); removed SRE Observability (FY2022/2023-Q2).Jan 3 2023, 2:33 PM
herron removed herron as the assignee of this task.Jan 5 2023, 5:59 PM
herron added a project: User-herron.
herron closed subtask T313231: Dispatch Google workspace integration as Resolved.
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2022/2023-Q3) board.Jan 12 2023, 12:17 AM
lmata added a parent task: T308467: implementing an incident response workflow automation tool for SRE.Jan 12 2023, 8:19 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q4); removed SRE Observability (FY2022/2023-Q3).May 2 2023, 1:38 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2022/2023-Q4) board.May 2 2023, 1:50 PM
lmata moved this task from Inbox to Prioritized on the Incident Tooling board.May 16 2023, 3:32 AM
lmata edited projects, added SRE Observability (FY2023/2024-Q1); removed SRE Observability (FY2022/2023-Q4).Jul 18 2023, 4:56 PM
lmata removed a project: SRE Observability (FY2023/2024-Q1).Jul 18 2023, 8:51 PM
BCornwall closed subtask T313229: Production Dispatch Infrastructure as Declined.Aug 24 2023, 4:27 PM
MoritzMuehlenhoff reopened subtask T313229: Production Dispatch Infrastructure as Open.Aug 24 2023, 4:33 PM
herron closed subtask T313229: Production Dispatch Infrastructure as Resolved.Aug 24 2023, 4:40 PM
herron changed the status of subtask T313229: Production Dispatch Infrastructure from Resolved to Declined.
herron closed this task as Declined.Aug 24 2023, 4:49 PM
herron added a subscriber: BCornwall.
In T313229#9117902, @BCornwall wrote:

Closing as dispatch has been ruled out as an option: See T308467 for follow-up discussion of where we're going.

BCornwall closed subtask T313230: Dispatch IRC Integration as Declined.Aug 24 2023, 4:55 PM
BCornwall closed subtask T313232: Dispatch Documentation as Declined.
BCornwall closed subtask T313233: Dispatch Splunk On-call / VictorOps Integration as Declined.
BCornwall closed subtask T313234: Dispatch Phabricator Integration as Declined.
BCornwall closed subtask T317192: implement on call shift hand-off / hand-overs in dispatch as Declined.
herron updated the task description. (Show Details)Sep 25 2023, 5:18 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL