Steps to replicate the issue (include links if applicable):
What happens?:
upstream connect error or disconnect/reset before headers. reset reason: overflow
What should have happened instead?:
Returned suggestions
Software version (skip for WMF-hosted wikis like Wikipedia):
Other information (browser name/version, screenshots, etc.):
Multiple users (all?) affected. Editing on Wikidata impossible
The API i now responding, There was a site issue just before which probably caused this.
This was brought up on Telegram. When editing statements no suggestions are shown anymore. The top-right searchbox on enwp and Wikidata both don't show any suggestions either.
Works for me again too. I noticed a spike on https://www.wikimediastatus.net/ for the "Wiki error responses" to 940 errors/second and a drop for "Successful edits".
Anyone able to provide a RFO (reason for outage)? Or just close if you don't care that the service was down.
@Mutante: adding as IC, can you please let people know when the incident report from last night is ready?
@Multichill: I’ve added the Incident Co-ordinator from last night, lots of people cared and it was responded to quickly. SRE will do an incident report which will be available on wikitech if it can be made public.
We can confirm we served a lot of 5xx's in a time span from about 21:04 to 21:14 UTC yesterday.
Assuming you hit that ~ 10 minute window.
The reason was an overloaded database, db1143, in the s4 cluster, which is commons.
The fix was that DBAs depooled that database temporarily.
The actual incident is over, it was mitigated within minutes.
Regarding the report it's still WIP.
@Multichill @RhinosF1 @Lydia_Pintscher
Here you go: https://wikitech.wikimedia.org/wiki/Incidents/2023-01-09_API_errors_-_db1143#Graphs
It's related to "onLinksUpdateComplete" looking up if image usage has changed on commons.