Page MenuHomePhabricator

text-https:443 has failed probes (retrospective task)
Closed, ResolvedPublic

Description

This task is here for prosperity to capture the incident from 2022-05-25 where a flood of post requests caused an outage to the text service

20:08 <+jinxer-wm> (ProbeDown) firing: (8) Service text-https:443 has failed probes (http_text-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#ProbeDown -
https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=http - https://alerts.wikimedia.org/?q=alertname%3DProbeDo

The traffic ultimately stopped on its own, however mitigations has also been implemented to limit future impact

Related Objects

StatusSubtypeAssignedTask
Resolvedjbond
OpenNone

Event Timeline

jbond triaged this task as Medium priority.May 25 2022, 10:07 AM
jbond added a subtask: Restricted Task.

Incident report is here https://wikitech.wikimedia.org/wiki/Incidents/2022-05-25_de.wikipedia.org please review and update with any further information thanks

Krinkle assigned this task to jbond.