Steps to replicate the issue (include links if applicable):
- Users are reporting errors during upload (T328872#9956803 and below)
- "An unknown error occurred in storage backend "local-swift-eqiad"
- "Could not store upload in the stash (UploadStashFileException)"
- Could not read the status of file "mwstore://local-multiwrite/local-public/archive/9/96/20240705164303!At_Swindon_Steam_Railway_Museum_2024_176.jpg".
The MediaWiki error backend-fail-stat occured: Could not read the status of file "mwstore://local-multiwrite/local-public/archive/3/3a/20240705173523!At_Swindon_Steam_Railway_Museum_2024_206.jpg".
What happens?:
- Grafana and logstash showing high levels of 503s for swift
- https://logstash.wikimedia.org/app/dashboards#/view/AXFV7JE83bOlOASGccsT?_g=h@bbef750&_a=h@2ee6f1d
- https://grafana-rw.wikimedia.org/d/OPgmB1Eiz/swift?forceLogin=&from=1719294248878&orgId=1&to=1720203381602&var-site=All&viewPanel=37
What should have happened instead?:
- No increase in errors ;)
Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):
Other information (browser name/version, screenshots, etc.):