"Error undeleting file: The file "mwstore://local-multiwrite/local-public/b/b8/Kit_body_FCF2016a.png" is in an inconsistent state within the internal storage backends"

Occurred while executing a history split. I was able to split out the overwritten file (now at https://commons.wikimedia.org/wiki/File:Kit_body_FCF2016a_(no_logos).png ) but cannot restore the original.

Error undeleting file: The file "mwstore://local-multiwrite/local-public/8/84/張善政參訪2016臺灣國際蘭展_02.jpg" is in an inconsistent state within the internal storage backends

Another example.

As an update, the first file I mentioned being unable to restore worked just now. The second still gives the error.

In T141704#2558555, @Revent wrote:

As an update, the first file I mentioned being unable to restore worked just now. The second still gives the error.

The only error it gave was the "inconsistent" state one, and nothing before that?

Yes, nothing until I actually hit 'restore'. This is apparently quite uncommon, but a bit catastrophic when it does occur.

As a further note, the deleted file, and page revisions, are visible from the history (when, of course, looking at deleted revisions).

Same problem here:

Error undeleting file: The file "mwstore://local-multiwrite/local-public/9/9f/Chocolate(bgFFF).jpg" is in an inconsistent state within the internal storage backends

https://commons.wikimedia.org/wiki/File:%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg Error deleting file: The file "mwstore://local-multiwrite/local-deleted/a/i/e/aief7c28s8bvsd9y33eha3hyfs50w4a.jpg" is in an inconsistent state within the internal storage backends

In T141704#3144756, @Yann wrote:

https://commons.wikimedia.org/wiki/File:%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg Error deleting file: The file "mwstore://local-multiwrite/local-deleted/a/i/e/aief7c28s8bvsd9y33eha3hyfs50w4a.jpg" is in an inconsistent state within the internal storage backends

Unsure if related, but an attempt to view the deleted version of this file also gives an error:
https://commons.wikimedia.org/w/index.php?title=Special:Undelete&target=File%3A%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg&file=aief7c28s8bvsd9y33eha3hyfs50w4a.jpg&token=8b5083b250dbeb28e279537f17c96fc358dd59c2%2B\

Not Found

The resource could not be found.

@Ankry Just now, for me at least, that link works. The errors seem to be intermittent.

@aaron there seems to be a surge on WikiPage::doDeleteArticleReal errors possibly related to this ticket and T132921 on both enwiki and commons https://logstash.wikimedia.org/goto/f17e899253c33bac573fbc0527f927a4 due to "Lock wait timeout exceeded":

SELECT  rev_id,rev_page,rev_text_id,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,COALESCE( comment_rev_comment.comment_text, rev_comment ) AS `rev_comment_text`,comment_rev_comment.comment_data AS `rev_comment_data`,comment_rev_comment.comment_id AS `rev_comment_cid`,rev_user,rev_user_text,NULL AS `rev_actor`,rev_content_format,rev_content_model  FROM `revision` LEFT JOIN `revision_comment_temp` `temp_rev_comment` ON ((temp_rev_comment.revcomment_rev = rev_id)) LEFT JOIN `comment` `comment_rev_comment` ON ((comment_rev_comment.comment_id = temp_rev_comment.revcomment_comment_id))   WHERE rev_page = '38966726'   FOR UPDATE

Adding @Anomie as it could be a regression due to comments reformatting.

I suppose the query might be failing to lock the comment table rows somehow or other (which aren't going to be updated, so there's no actual need for them to be locked). But in that case I'd think the errors would have begun earlier, the comment table has been in use on Commons since February 20 and everywhere since February 27.

The latest errors are : "Deadlock found when trying to get lock; try restarting transaction". Maybe there is some kind of loop dependency (e.g. trying to lock and at the same time delete the row?) https://logstash.wikimedia.org/goto/4465b31314253ab4ea5ebc501fe6b2c2

Got this on enwiki when attempting to delete https://en.wikipedia.org/wiki/File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg

Error deleting file: The file "mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg" is in an inconsistent state within the internal storage backends

Logstash: https://logstash.wikimedia.org/goto/ef9aa6a4c5e5703f1dd5fb39f883382a

@MusikAnimal Was that message shown within the wiki interface? I would expect that to correlate with an "ERROR"-level message in the logs, but it seems it only correlates with INFO-level messages.

The request ID for your action was W5t4uQrAEDYAAA8@XVQAAAHC, which emitted three messages:

host: mw2255, wiki: enwiki, url: /w/index.php?title=File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg&action=delete

channel: UpdateRepo, level: INFO
> Couldn't find an item for File:Dr. Richard Pierzchajlo, Head Shot, 2017.jpg

channel: FileOperation, level: INFO
> FileBackendMultiWrite failed to resync: ["mwstore://local-multiwrite/local-public/e/ed/Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg","mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg"]


channel: FileOperation, level: INFO,
> FileBackendMultiWrite failed sync check: ["mwstore://local-multiwrite/local-public/e/ed/Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg","mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg"]

In T141704#4594564, @Krinkle wrote:

@MusikAnimal Was that message shown within the wiki interface? I would expect that to correlate with an "ERROR"-level message in the logs, but it seems it only correlates with INFO-level messages.

Yep. I just tried again and got the same error: https://en.wikipedia.org/w/index.php?title=File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg&action=delete

When I try to delete File:Dr. Richard Pierzchajlo, Head Shot, 2017.jpg I get

Error deleting file: The file "mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg" is in an inconsistent state within the internal storage backends

Could someone please fix this so I can delete the file with the reason, [[WP:CSD#F8|F8]]: Media file available on Commons