| Nirmos | |
| Sep 18 2021, 12:44 AM |
| F34646794: image.png | |
| Sep 18 2021, 9:30 AM |
| F34646789: image.png | |
| Sep 18 2021, 9:26 AM |
| F34646785: image.png | |
| Sep 18 2021, 9:26 AM |
| F34646551: image.png | |
| Sep 18 2021, 12:51 AM |
| F34646548: image.png | |
| Sep 18 2021, 12:51 AM |
Wikimedia sites seem to be down right now. After a very long time, the error message is upstream connect error or disconnect/reset before headers. reset reason: overflow
Incident document: https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-09-18_appserver_latency
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | BUG REPORT | RLazarus | T291311 2021-09-18 Wikimedia sites down | ||
| Restricted Task |
From T291312, some other errors I have experienced:
Some stats (thanks TNT on the unofficial Wikimedia Discord for the links):
https://grafana.wikimedia.org/d/000000479/frontend-traffic?orgId=1&from=1631915084782&to=1631925884782&var-site=All&var-cache_type=text&var-cache_type=upload&var-status_type=1&var-status_type=2&var-status_type=3&var-status_type=4&viewPanel=7
en.wp (at least) was agonisingly slow shortly before it all went down, and even after en.wp came back up I wasn't able to allow Oauth access to login here.
Second this, I was doing a few quick edits on hr.wp, and loading after each one got progressively slower, the last page loaded for a good minute or two before the error showed. Gadgets and .js imports stopped working while the site was still up but slow.
Can confirm : fr.wp went slow around 02:10 AM CEST, came back up around 03:00 AM CEST. No abnormal filtered detections from fr.wp's AbuseFilter.
This should be fully resolved as of about 1:10 UTC, sorry for the trouble and thanks for all the reports.
Because the root cause was a DoS vector, we can't publish an incident report yet -- but we'll do so (and make the private task T284419 public) as soon as the vulnerability is addressed.
Resolving this but please let us know if you continue to experience any issues from this incident.