Page MenuHomePhabricator

High rate of errors and increased latency on uncached MediaWiki requests due to infrastructure outage
Open, HighPublic

Description

On 2023-05-25, for approximately 15-20 minutes (~14:08 - ~14:27 UTC), logged in users connecting to to Wikimedia wikis through our Ashburn datacenter, and editors in general (approximately 7000 requests/s) may have received 503 errors or experienced higher latency than usual. Edit activity stopped almost completely.

Screenshot_20230525_175641.png (1×2 px, 163 KB)

The issue was mitigated and an investigation is ongoing to prevent the issue from reappearing. More info and details to come soon in the form of an Incident Document: https://wikitech.wikimedia.org/wiki/Incident_status

The ongoing status was kept updated at https://www.wikimediastatus.net/incidents/lx0vwl2p01xd (now resolved)

Event Timeline

jcrespo triaged this task as High priority.Thu, May 25, 3:56 PM
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)
jcrespo updated the task description. (Show Details)