Improve Automation for Alert Reviews

Overview

This task tracks the ideas and improvements for the Alert Review process automation.

Wikitech page: https://wikitech.wikimedia.org/wiki/Alert_review
GitLab repository: https://gitlab.wikimedia.org/repos/sre/alertreview

1. Prerequisites

• Create the "alertreview" GCP project
• Create GitLab repository
• Create shared Drive folder for resources

2. Google Workspace features

• Analyze SplunkOnCall data Implement top 5 paging alerts feature
• Analyze email alerts data Add email alert support to the alert review process

3. Logstash integration

• Integrate with Logstash to read data
• Gather top Alertmanager alerts
• Gather top Icinga alerts
• Gather loneliest Icinga alerts

4. Phabricator integration

• Integrate with Phabricator to read and write data
• Create a Phabricator task for the loneliest Icinga Alerts

5. SplunkOnCall integration

• Integrate with SplunkOnCall to read data

Document generation

• Enable exporting the data to a Google Document