After some research and a chat with the Observability team, there are some answers, insights and suggestions for the future.

General insights

• Wikimedia already utilizes Alertmanager for sending alerts to email, IRC and pagers, managed by the Observability team.
  • It may be possible to add Slack to the outputs, but that would require configuration change within Alertmanager. A ticket was posted to request for this addition T401730: Add a pathway for Alertmanager to send alerts in Slack
• Alerts will be sent to a specific slack channel so we will need to see how the messages sent into the channel can be used by multiple teams. Alerts can be sent with their modules, labels and keywords, and there are ways in Slack teams can choose to highlight specific keywords that are relevant to them. This will also be the most simplest straightforward MVP for the system as we learn more about how to adjust it.
• Once the Slack method is added, it should be possible for teams to set up their own alerts using the regular means (through the grafana UI) to be sent to the alerts slack channel.
• Alerts depend on the monitoring labels that are sent by Prometheus; currently, the REST API system creates consistent labels that utilize the module names and parameters, and those can be shown in grafana graphs. They can also be used for alerts with threshholds and severities.
• Alerts can be created from any and all labels that are in Prometheus, regardless of where they come from -- which means that alerts can be cut across monitoring that comes from mediawiki or the API gateway or any other service, as long as the labels are consistent.

Monitoring

• We should try to preserve the current labels as much as possible, even for the sake of historical and continuous data.
• Whatever labels are chosen (if changed, or added to the system) we should make sure we strive for consistency across the system (potentially looking at holistic consistency beyond "just" the REST API, if possible)
• There are recommendations and "best practices" for monitoring across mediawiki code https://www.mediawiki.org/wiki/Manual:Stats#Notes

Recommended next steps

The project is definitely doable within the time period and resourcing for the hypothesis for the quarter.
General next steps are:

• Add "Slack" as an output for Alertmanager T401730: Add a pathway for Alertmanager to send alerts in Slack
• Review the current monitoring schema (labels) and decide what might be missing to the type of questions MW Interfaces engineering will want to know about or be alerted for
• Come up with 1-2 "generic" monitoring label/topics that are cross-cutting, and add them to the code.
• Set up an alert with threshholds into the new Slack channel
• Set up 2-3 more general alerts into Slack
• Provide documentation on how to properly add prometheus to API modules + how to use the available labels + how to set up an alert.

Search for "My Keywords" on this Slack doc page for details on having Slack alert on specific keywords.