Removing MediaWiki-File-management as I doubt that this is a bug in MediaWiki core code. I get a 404 error here (Central Europe).

The problem is that this object has been uploaded to codfw OK, but not eqiad; all being equal, this will get picked up and copied to eqiad by next week's rclone_sync run.

Looking through the logs (grep -F 'Statue_of_Saint_Anne_on_P%25C3%25ADsek_Stone_Bridge.jpg' /var/log/swift/proxy-access.log), there is exactly one PUT (ignoring thumbs):

ms-fe2013.codfw.wmnet: Oct 15 03:38:58 ms-fe2013 proxy-server: 10.194.184.117 10.192.0.87 15/Oct/2024/03/38/58 PUT /v1/AUTH_mw/wikipedia-commons-local-public.d6/d/d6/Statue_of_Saint_Anne_on_P%25C3%25ADsek_Stone_Bridge.jpg HTTP/1.0 201 - wikimedia/multi-http-client%20v1.1 AUTH_tk17bc298db... 3187995 - c56ce4bb7a3561b96f40bada5999db65 tx4e75f1d6a665438680878-00670de3d2 - 0.0555 - - 1728963538.870228291 1728963538.925695181 0

@Aklapper I don't mean to play tag-tennis, but I've put file-management back - swift has only 1 copy (where it should have two) because only one PUT attempt was made, so this is a bug in the MW stack (which might be uploadwizard if that's what the uploader used); feel free to allocate onwards as you see fit, but from my swift-admin POV this is swift behaving as expected.

uploadwizard itself is not doing the stores, that goes through the intermediate filerepo layer. But, if MW code was consistently uploading only one copy, we would be seeing loads of this. Maybe a race condition, or a problem specific of this server.

Do the logs show that the same apache server sends two PUTs when uploading other images? Or is it consistently sending only one unlike their peers?

It's not in general a consistent problem, no - mostly MW manages to successfully upload two copies (one to eqiad, one to codfw); but it's unreliable, and there is a slow trickle of tickets like this where its only managed to write to one cluster (typically the primary DC-local one). I've not tried to track down this particular server in logs (it's much less rewarding now they're all effectively k8s pods), but if it was always failing, we'd be seeing a lot more problem reports.

Maybe the filerepo layer should actively confirm if the write happened to both servers, and kick of a retry job or something if it cannot confirm ?

Is FileBackendMultiWrite what ensures that it is written to both SwiftFileBackends ?