[25Q2] Add alert monitoring for Wikifunctions services
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Sharvaniharan
	Oct 7 2024, 9:39 PM

Description

Background
We want to introduce alert monitoring for our services so that we not only know when something goes wrong or is about to go wrong, but also to have a good pulse on the performance of our services.

Approach

Use log severity levels to determine how or whether to alert
- Tools: Prometheus Alertmanager
Determine the channels for alerting
We are able to get alerts in these channels

Acceptance Criteria/Success Metrics

We can detect when an outage is about to happen because of an alert
If there is an outage, we can resolve it more quickly than we did for the last P0

Stretch Goal

Alerting based on keywords

Related Objects
Search...

Status	Assigned	Task
Resolved	ecarg	T376664 [25Q2] Add alert monitoring for Wikifunctions services
Resolved	ecarg	T371426 Create log-based alerting
Resolved	ecarg	T376227 Instrument new metrics creation on the Grafana dashboard
Resolved	ecarg	T375922 enable tracing data into services

Event Timeline

Sharvaniharan created this task.Oct 7 2024, 9:39 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2024, 9:39 PM

Sharvaniharan triaged this task as Low priority.Oct 7 2024, 9:40 PM

Jdforrester-WMF renamed this task from [25Q2] WF Services Alert Monitoring to [25Q2] Add alert monitoring for Wikifunctions services.Oct 8 2024, 1:53 PM

Jdforrester-WMF added a subtask: T375634: Allow to route generic alerts (like ProbeDown) to team receivers.

ecarg added a subtask: T371426: Create log-based alerting.Oct 8 2024, 5:41 PM

ecarg added a subtask: T376227: Instrument new metrics creation on the Grafana dashboard .Oct 8 2024, 5:51 PM

Sharvaniharan assigned this task to ecarg.Oct 10 2024, 11:17 PM

ecarg added a subtask: T369770: Convert the temporary logs we added to the back-end services into new metrics if we're keeping them, and delete the ones we're not.Oct 17 2024, 6:56 PM

ecarg closed subtask T376227: Instrument new metrics creation on the Grafana dashboard as Resolved.Nov 1 2024, 5:23 PM

ecarg closed subtask T371426: Create log-based alerting as Resolved.

ecarg added a subtask: T375922: enable tracing data into services.

ecarg changed the status of subtask T375922: enable tracing data into services from Open to In Progress.

ecarg closed subtask T375922: enable tracing data into services as Resolved.Nov 14 2024, 8:55 PM

Jdforrester-WMF moved this task from Epics for this Quarter to Needs Sign-off on the Abstract Wikipedia team (25Q2 (Oct–Dec)) board.Nov 21 2024, 7:48 PM

Jdforrester-WMF removed a subtask: T369770: Convert the temporary logs we added to the back-end services into new metrics if we're keeping them, and delete the ones we're not.Nov 21 2024, 7:52 PM