Page MenuHomePhabricator

Unable to delete certain files due to "inconsistent state within the internal storage backends"
Open, HighPublic

Subscribers
Tokens
"Burninate" token, awarded by JJMC89."The World Burns" token, awarded by Liuxinyu970226."The World Burns" token, awarded by Thibaut120094."Hungry Hippo" token, awarded by Steinsplitter."100" token, awarded by Riley_Huntley."Burninate" token, awarded by Josve05a."The World Burns" token, awarded by Revent.
Assigned To
None
Authored By
DMacks, Jul 31 2016

Description

Over the past few hours, I've gotten lots of errors on commons when trying to delete or undelete pages. Annoyingly, it's sporadic. [[:commons:File:Hoogendijkwiki2 cropped.jpg]] is where I'm trying it. A few hours ago (sorry, lost more specific timestamp), I could delete it, but then undeletion of selected revisions failed while undeletion of the whole history succeeded. Errors were like:

Errors were encountered while undeleting the file:

  • The file "mwstore://local-multiwrite/local-deleted/g/q/6/gq6ktqygf7g9jfsl8kty1dkf8bj44bd.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/3/3a/Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends

15 minutes ago, I was able to delete that page, but then got a similar error for attempted selected-revision undeletion. And even undeletion of the whole thing failed:

Errors were encountered while undeleting the file:

  • The file "mwstore://local-multiwrite/local-deleted/c/d/f/cdfi1yksj5px1dbco6uldmbatg320mf.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/3/3a/Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/archive/3/3a/20130707102719!Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends

And just now, trying to undelete the whole thing failed:

Errors were encountered while undeleting the file:

  • The file "mwstore://local-multiwrite/local-public/3/3a/Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends
  • The file "mwstore://local-multiwrite/local-public/archive/3/3a/20130707102719!Hoogendijkwiki2_cropped.jpg" is in an inconsistent state within the internal storage backends

See also: T139873: Can't delete en:File:Testing_protection_level_reset_on_delete_and_restore.png, https://lists.wikimedia.org/pipermail/wikitech-l/2016-July/086165.html

Event Timeline

DMacks created this task.Jul 31 2016, 4:32 AM
Restricted Application added subscribers: Steinsplitter, Aklapper. · View Herald TranscriptJul 31 2016, 4:32 AM
Revent added a subscriber: Revent.Jul 31 2016, 8:21 AM

"Error undeleting file: The file "mwstore://local-multiwrite/local-public/b/b8/Kit_body_FCF2016a.png" is in an inconsistent state within the internal storage backends"

Occurred while executing a history split. I was able to split out the overwritten file (now at https://commons.wikimedia.org/wiki/File:Kit_body_FCF2016a_(no_logos).png ) but cannot restore the original.

Nemo_bis updated the task description. (Show Details)Jul 31 2016, 8:23 AM
Nemo_bis added a subscriber: binbot.

Error undeleting file: The file "mwstore://local-multiwrite/local-public/8/84/張善政參訪2016臺灣國際蘭展_02.jpg" is in an inconsistent state within the internal storage backends

Another example.

Restricted Application added a subscriber: Poyekhali. · View Herald TranscriptAug 7 2016, 12:55 AM
aaron removed aaron as the assignee of this task.Aug 7 2016, 6:53 AM
aaron added a project: Multimedia.
aaron added a subscriber: aaron.
Restricted Application added a subscriber: Matanya. · View Herald TranscriptAug 7 2016, 6:53 AM
Ankry added a subscriber: Ankry.Aug 11 2016, 2:19 PM

As an update, the first file I mentioned being unable to restore worked just now. The second still gives the error.

aaron added a comment.Aug 16 2016, 9:11 PM

As an update, the first file I mentioned being unable to restore worked just now. The second still gives the error.

The only error it gave was the "inconsistent" state one, and nothing before that?

Yes, nothing until I actually hit 'restore'. This is apparently quite uncommon, but a bit catastrophic when it does occur.

As a further note, the deleted file, and page revisions, are visible from the history (when, of course, looking at deleted revisions).

MarkTraceur moved this task from Untriaged to Tracking on the Multimedia board.Nov 30 2016, 7:32 PM

Same problem here:

Error undeleting file: The file "mwstore://local-multiwrite/local-public/9/9f/Chocolate(bgFFF).jpg" is in an inconsistent state within the internal storage backends
Thibaut120094 awarded a token.
Yann added a subscriber: Yann.Mar 30 2017, 7:09 PM

https://commons.wikimedia.org/wiki/File:%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg Error deleting file: The file "mwstore://local-multiwrite/local-deleted/a/i/e/aief7c28s8bvsd9y33eha3hyfs50w4a.jpg" is in an inconsistent state within the internal storage backends

Ankry added a comment.Mar 30 2017, 7:21 PM

https://commons.wikimedia.org/wiki/File:%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg Error deleting file: The file "mwstore://local-multiwrite/local-deleted/a/i/e/aief7c28s8bvsd9y33eha3hyfs50w4a.jpg" is in an inconsistent state within the internal storage backends

Unsure if related, but an attempt to view the deleted version of this file also gives an error:
https://commons.wikimedia.org/w/index.php?title=Special:Undelete&target=File%3A%D0%92%D0%B0%D0%BC%D0%BF%D0%B0%D0%BB.jpg&file=aief7c28s8bvsd9y33eha3hyfs50w4a.jpg&token=8b5083b250dbeb28e279537f17c96fc358dd59c2%2B\

Not Found

The resource could not be found.
Revent added a comment.Apr 1 2017, 1:05 PM

@Ankry Just now, for me at least, that link works. The errors seem to be intermittent.

fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.
jcrespo added subscribers: Anomie, jcrespo.

@aaron there seems to be a surge on WikiPage::doDeleteArticleReal errors possibly related to this ticket and T132921 on both enwiki and commons https://logstash.wikimedia.org/goto/f17e899253c33bac573fbc0527f927a4 due to "Lock wait timeout exceeded":

SELECT  rev_id,rev_page,rev_text_id,rev_timestamp,rev_minor_edit,rev_deleted,rev_len,rev_parent_id,rev_sha1,COALESCE( comment_rev_comment.comment_text, rev_comment ) AS `rev_comment_text`,comment_rev_comment.comment_data AS `rev_comment_data`,comment_rev_comment.comment_id AS `rev_comment_cid`,rev_user,rev_user_text,NULL AS `rev_actor`,rev_content_format,rev_content_model  FROM `revision` LEFT JOIN `revision_comment_temp` `temp_rev_comment` ON ((temp_rev_comment.revcomment_rev = rev_id)) LEFT JOIN `comment` `comment_rev_comment` ON ((comment_rev_comment.comment_id = temp_rev_comment.revcomment_comment_id))   WHERE rev_page = '38966726'   FOR UPDATE

Adding @Anomie as it could be a regression due to comments reformatting.

Anomie added a comment.Apr 2 2018, 3:48 PM

I suppose the query might be failing to lock the comment table rows somehow or other (which aren't going to be updated, so there's no actual need for them to be locked). But in that case I'd think the errors would have begun earlier, the comment table has been in use on Commons since February 20 and everywhere since February 27.

The latest errors are : "Deadlock found when trying to get lock; try restarting transaction". Maybe there is some kind of loop dependency (e.g. trying to lock and at the same time delete the row?) https://logstash.wikimedia.org/goto/4465b31314253ab4ea5ebc501fe6b2c2

Got this on enwiki when attempting to delete https://en.wikipedia.org/wiki/File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg

Error deleting file: The file "mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg" is in an inconsistent state within the internal storage backends

Logstash: https://logstash.wikimedia.org/goto/ef9aa6a4c5e5703f1dd5fb39f883382a

Krinkle added a subscriber: Krinkle.

@MusikAnimal Was that message shown within the wiki interface? I would expect that to correlate with an "ERROR"-level message in the logs, but it seems it only correlates with INFO-level messages.

The request ID for your action was W5t4uQrAEDYAAA8@XVQAAAHC, which emitted three messages:

reqId:W5t4uQrAEDYAAA8@XVQAAAHC
host: mw2255, wiki: enwiki, url: /w/index.php?title=File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg&action=delete

channel: UpdateRepo, level: INFO
> Couldn't find an item for File:Dr. Richard Pierzchajlo, Head Shot, 2017.jpg

channel: FileOperation, level: INFO
> FileBackendMultiWrite failed to resync: ["mwstore://local-multiwrite/local-public/e/ed/Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg","mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg"]


channel: FileOperation, level: INFO,
> FileBackendMultiWrite failed sync check: ["mwstore://local-multiwrite/local-public/e/ed/Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg","mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg"]

@MusikAnimal Was that message shown within the wiki interface? I would expect that to correlate with an "ERROR"-level message in the logs, but it seems it only correlates with INFO-level messages.

Yep. I just tried again and got the same error: https://en.wikipedia.org/w/index.php?title=File:Dr._Richard_Pierzchajlo,_Head_Shot,_2017.jpg&action=delete

Krinkle renamed this task from Storage backend errors on commons when deleting/restoring pages to Unable to delete certain files due to "inconsistent state within the internal storage backends".Sep 18 2018, 4:53 PM
Liuxinyu970226 awarded a token.

When I try to delete File:Dr. Richard Pierzchajlo, Head Shot, 2017.jpg I get

Error deleting file: The file "mwstore://local-multiwrite/local-deleted/q/q/2/qq22rrn3rkspulntfholqkxlpie4wo1.jpg" is in an inconsistent state within the internal storage backends

Could someone please fix this so I can delete the file with the reason, [[WP:CSD#F8|F8]]: Media file available on Commons

fgiunchedi moved this task from Radar to Backlog on the User-fgiunchedi board.Jan 2 2019, 10:38 AM
fgiunchedi moved this task from Backlog to Radar on the User-fgiunchedi board.Apr 23 2019, 12:29 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:11 PM
JJMC89 awarded a token.Sep 7 2019, 7:44 AM
JJMC89 added a subscriber: JJMC89.