Page MenuHomePhabricator

Unable to delete certain files due to "inconsistent state within the internal storage backends"
Closed, DeclinedPublicPRODUCTION ERROR

Assigned To
None
Authored By
DMacks
Jul 31 2016, 4:32 AM
Tokens
"Burninate" token, awarded by JJMC89."The World Burns" token, awarded by Liuxinyu970226."The World Burns" token, awarded by Thibaut120094."Hungry Hippo" token, awarded by Steinsplitter."100" token, awarded by Riley_Huntley."Burninate" token, awarded by Josve05a."The World Burns" token, awarded by Revent.

Description

Over the past few hours, I've gotten lots of errors on commons when trying to delete or undelete pages. Annoyingly, it's sporadic. [[:commons:File:Hoogendijkwiki2 cropped.jpg]] is where I'm trying it. A few hours ago (sorry, lost more specific timestamp), I could delete it, but then undeletion of selected revisions failed while undeletion of the whole history succeeded. Errors were like:

Errors were encountered while undeleting the file:

  • The file "mwstore://local-multiwrite/local-deleted/g/q/6/gq6ktqygf7g9jfsl8kty1dkf8bj44bd.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/3/3a/Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends

15 minutes ago, I was able to delete that page, but then got a similar error for attempted selected-revision undeletion. And even undeletion of the whole thing failed:

Errors were encountered while undeleting the file:

  • The file "mwstore://local-multiwrite/local-deleted/c/d/f/cdfi1yksj5px1dbco6uldmbatg320mf.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/3/3a/Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/archive/3/3a/20130707102719!Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends

And just now, trying to undelete the whole thing failed:

Errors were encountered while undeleting the file:

  • The file "mwstore://local-multiwrite/local-public/3/3a/Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/archive/3/3a/20130707102719!Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends

See also: T139873: Can't delete en:File:Testing_protection_level_reset_on_delete_and_restore.png, https://lists.wikimedia.org/pipermail/wikitech-l/2016-July/086165.html

Event Timeline

"Error undeleting file: The file "mwstore://local-multiwrite/local-public/b/b8/Kit_body_FCF2016a.png" is in an inconsistent state within the internal storage backends"

Occurred while executing a history split. I was able to split out the overwritten file (now at https://commons.wikimedia.org/wiki/File:Kit_body_FCF2016a_(no_logos).png ) but cannot restore the original.

Error undeleting file: The file "mwstore://local-multiwrite/local-public/8/84/張善政參訪2016臺灣國際蘭展_02.jpg" is in an inconsistent state within the internal storage backends

Another example.

aaron removed aaron as the assignee of this task.Aug 7 2016, 6:53 AM
aaron added a project: Multimedia.
aaron added a subscriber: aaron.

As an update, the first file I mentioned being unable to restore worked just now. The second still gives the error.

As an update, the first file I mentioned being unable to restore worked just now. The second still gives the error.

The only error it gave was the "inconsistent" state one, and nothing before that?

Yes, nothing until I actually hit 'restore'. This is apparently quite uncommon, but a bit catastrophic when it does occur.

As a further note, the deleted file, and page revisions, are visible from the history (when, of course, looking at deleted revisions).

Same problem here:

Error undeleting file: The file "mwstore://local-multiwrite/local-public/9/9f/Chocolate(bgFFF).jpg" is in an inconsistent state within the internal storage backends

https://commons.wikimedia.org/wiki/File:%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg Error deleting file: The file "mwstore://local-multiwrite/local-deleted/a/i/e/aief7c28s8bvsd9y33eha3hyfs50w4a.jpg" is in an inconsistent state within the internal storage backends

https://commons.wikimedia.org/wiki/File:%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg Error deleting file: The file "mwstore://local-multiwrite/local-deleted/a/i/e/aief7c28s8bvsd9y33eha3hyfs50w4a.jpg" is in an inconsistent state within the internal storage backends

Unsure if related, but an attempt to view the deleted version of this file also gives an error:
https://commons.wikimedia.org/w/index.php?title=Special:Undelete&target=File%3A%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg&file=aief7c28s8bvsd9y33eha3hyfs50w4a.jpg&token=8b5083b250dbeb28e279537f17c96fc358dd59c2%2B\

Not Found

The resource could not be found.

@Ankry Just now, for me at least, that link works. The errors seem to be intermittent.

jcrespo added subscribers: Anomie, jcrespo.

@aaron there seems to be a surge on WikiPage::doDeleteArticleReal errors possibly related to this ticket and T132921 on both enwiki and commons https://logstash.wikimedia.org/goto/f17e899253c33bac573fbc0527f927a4 due to "Lock wait timeout exceeded":

SELECT  rev_id,rev_page,rev_text_id,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,COALESCE( comment_rev_comment.comment_text, rev_comment ) AS `rev_comment_text`,comment_rev_comment.comment_data AS `rev_comment_data`,comment_rev_comment.comment_id AS `rev_comment_cid`,rev_user,rev_user_text,NULL AS `rev_actor`,rev_content_format,rev_content_model  FROM `revision` LEFT JOIN `revision_comment_temp` `temp_rev_comment` ON ((temp_rev_comment.revcomment_rev = rev_id)) LEFT JOIN `comment` `comment_rev_comment` ON ((comment_rev_comment.comment_id = temp_rev_comment.revcomment_comment_id))   WHERE rev_page = '38966726'   FOR UPDATE

Adding @Anomie as it could be a regression due to comments reformatting.

I suppose the query might be failing to lock the comment table rows somehow or other (which aren't going to be updated, so there's no actual need for them to be locked). But in that case I'd think the errors would have begun earlier, the comment table has been in use on Commons since February 20 and everywhere since February 27.

The latest errors are : "Deadlock found when trying to get lock; try restarting transaction". Maybe there is some kind of loop dependency (e.g. trying to lock and at the same time delete the row?) https://logstash.wikimedia.org/goto/4465b31314253ab4ea5ebc501fe6b2c2

Got this on enwiki when attempting to delete https://en.wikipedia.org/wiki/File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg

Error deleting file: The file "mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg" is in an inconsistent state within the internal storage backends

Logstash: https://logstash.wikimedia.org/goto/ef9aa6a4c5e5703f1dd5fb39f883382a

Krinkle added a subscriber: Krinkle.

@MusikAnimal Was that message shown within the wiki interface? I would expect that to correlate with an "ERROR"-level message in the logs, but it seems it only correlates with INFO-level messages.

The request ID for your action was W5t4uQrAEDYAAA8@XVQAAAHC, which emitted three messages:

reqId:W5t4uQrAEDYAAA8@XVQAAAHC
host: mw2255, wiki: enwiki, url: /w/index.php?title=File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg&action=delete

channel: UpdateRepo, level: INFO
> Couldn't find an item for File:Dr. Richard Pierzchajlo, Head Shot, 2017.jpg

channel: FileOperation, level: INFO
> FileBackendMultiWrite failed to resync: ["mwstore://local-multiwrite/local-public/e/ed/Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg","mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg"]


channel: FileOperation, level: INFO,
> FileBackendMultiWrite failed sync check: ["mwstore://local-multiwrite/local-public/e/ed/Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg","mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg"]

@MusikAnimal Was that message shown within the wiki interface? I would expect that to correlate with an "ERROR"-level message in the logs, but it seems it only correlates with INFO-level messages.

Yep. I just tried again and got the same error: https://en.wikipedia.org/w/index.php?title=File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg&action=delete

Krinkle renamed this task from Storage backend errors on commons when deleting/restoring pages to Unable to delete certain files due to "inconsistent state within the internal storage backends".Sep 18 2018, 4:53 PM

When I try to delete File:Dr. Richard Pierzchajlo, Head Shot, 2017.jpg I get

Error deleting file: The file "mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg" is in an inconsistent state within the internal storage backends

Could someone please fix this so I can delete the file with the reason, [[WP:CSD#F8|F8]]: Media file available on Commons

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
daniel added a subscriber: daniel.

No activity since 2019, nothing relevant in the logs for a year. Closing for now. Please reopen if it happens again.