Steps to replicate the issue (include links if applicable):
Undelete: https://commons.wikimedia.org/wiki/Special:Undelete/File:Place_de_Chamb%C3%A9sy.JPG
What happens?:
Some or all of the undeletion failed: The file "mwstore://local-multiwrite/local-public/8/8c/Place_de_Chambésy.JPG" is in an inconsistent state within the internal storage backends
What should have happened instead?:
File should be restored.
FileBackendMultiWrite::doOperationsInternal: failed sync check: ["mwstore://local-multiwrite/local-deleted/f/z/v/fzvpnmgrl21khoymipkve3ld45fmbxe.jpg","mwstore://local-multiwrite/local-public/8/8c/Place_de_Chamb\u00e9sy.JPG","mwstore://local-multiwrite/loc
Another file with the same issue
https://commons.wikimedia.org/wiki/Special:Undelete/File:BBC2_striped_ident_1.jpg
Some or all of the undeletion failed:
The file "mwstore://local-multiwrite/local-public/a/a8/BBC2_striped_ident_1.jpg" is in an inconsistent state within the internal storage backends
The file "mwstore://local-multiwrite/local-public/archive/a/a8/20211214164438!BBC2_striped_ident_1.jpg" is in an inconsistent state within the internal storage backends
The file "mwstore://local-multiwrite/local-public/archive/a/a8/20161115150846!BBC2_striped_ident_1.jpg" is in an inconsistent state within the internal storage backends
The file "mwstore://local-multiwrite/local-public/archive/a/a8/20151215185311!BBC2_striped_ident_1.jpg" is in an inconsistent state within the internal storage backends
Did we switch DCs recently ? I remember last year when we did that the whole thing got itself in a mess as well because the DCs basically were not sharing the same truth.
We did switch DCs recently, the impact of which is more load on thumbor (and during the switchover we discovered there was a shortage of thumbor pods, but we fixed that). Our dashboards for both Thumbor and Swift are not showing an increase in errors, though.