Page MenuHomePhabricator
Create Task
Maniphest T270994

Re-deleting a Commons file: "Error deleting file: The file "mwstore://local-multiwrite/local-deleted/..." is in an inconsistent state within the internal storage backends".
Closed, ResolvedPublic
Actions

Description

Recently, I deleted https://commons.wikimedia.org/wiki/File:Jack_Settleman_at_Super_Bowl_LIV.jpg as a copyvio. Shortly thereafter, I undeleted it to more closely examine it because of an appeal. Satisfied that this is indeed a copyvio, attempts to again delete the file are unsuccessful, consistently producing this error message:

"Error deleting file: The file "mwstore://local-multiwrite/local-deleted/e/m/0/em06ntsy84nncjbi6k7a9ca6xqk9kh3.jpg" is in an inconsistent state within the internal storage backends".

Details

SubjectRepoBranchLines +/-
filebackend: fix bogus replication error log entries in FileBackendMultiWritemediawiki/coremaster+9 -6
filebackend: inject the proxy backend logger into FileBackendMultiWrite sub-backendsmediawiki/coremaster+13 -12
filebackend: add more logging to SwiftFileBackend for "conservative" sync modemediawiki/coremaster+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

JGHowes created this task.Jan 1 2021, 4:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 1 2021, 4:38 PM
taavi added projects: SRE-swift-storage, SRE.Jan 1 2021, 4:42 PM
JGHowes updated the task description. (Show Details)Jan 1 2021, 4:51 PM
RhinosF1 subscribed.Jan 1 2021, 4:52 PM
Aklapper renamed this task from Bug in "internal storage backends" in re-deleting a Commons file after undelete to Re-deleting a Commons file: "Error deleting file: The file "mwstore://local-multiwrite/local-deleted/..." is in an inconsistent state within the internal storage backends"..Jan 1 2021, 4:52 PM
Aklapper updated the task description. (Show Details)
RhinosF1 added a comment.EditedJan 2 2021, 10:25 AM

Is this not the same as T244567: Some files cannot be deleted "Error deleting file: An unknown error occurred in storage backend "local-multiwrite". " (due to missing physical file for old image entries)?

JGHowes added a comment.Jan 3 2021, 11:52 PM

Apparently now fixed ... successfully deleted at 2350 UTC tonight.

Joe triaged this task as Low priority.Jan 4 2021, 9:34 AM
Joe added a project: Performance-Team.
Joe added subscribers: aaron, Joe.

@aaron do you have any insights on what happened there? See https://logstash.wikimedia.org/goto/04e091fe7e42112a1229b9d207990d1c for the logstash data. It looks like some failures to move the object in one of the two swift clusters originally.

Setting priority to low given the problem solved itself.

• dpifke moved this task from Inbox, needs triage to Backlog: Maintenance, non-prioritized on the Performance-Team board.Jan 4 2021, 7:08 PM
gerritbot added a comment.Jan 4 2021, 11:52 PM

Change 654313 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] filebackend: add more logging to SwiftFileBackend for "conservative" sync mode

https://gerrit.wikimedia.org/r/654313

gerritbot added a project: Patch-For-Review.Jan 4 2021, 11:52 PM
gerritbot added a comment.Jan 4 2021, 11:58 PM

Change 654334 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] filebackend: fix bogus replication error log entries in FileBackendMultiWrite

https://gerrit.wikimedia.org/r/654334

aaron added a comment.Jan 4 2021, 11:58 PM

A subset of the log entries are bogus though (should be DEBUG, not ERROR).

aaron added a comment.Jan 5 2021, 12:27 AM

At first, I suspected a timeout causing a failed deferred updated, but it seems that no failure was logged likely due to NullLogger being used by sub-backends of FileBackendMultiWrite.

gerritbot added a comment.Jan 5 2021, 12:31 AM

Change 654340 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] filebackend: inject the proxy backend logger into FileBackendMultiWrite sub-backends

https://gerrit.wikimedia.org/r/654340

gerritbot added a comment.Jan 19 2021, 1:55 AM

Change 654313 merged by jenkins-bot:
[mediawiki/core@master] filebackend: add more logging to SwiftFileBackend for "conservative" sync mode

https://gerrit.wikimedia.org/r/654313

gerritbot added a comment.Jan 19 2021, 1:59 AM

Change 654334 merged by jenkins-bot:
[mediawiki/core@master] filebackend: fix bogus replication error log entries in FileBackendMultiWrite

https://gerrit.wikimedia.org/r/654334

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.28; 2021-01-26).Jan 19 2021, 2:00 AM
gerritbot added a comment.Jan 19 2021, 2:00 AM

Change 654340 merged by jenkins-bot:
[mediawiki/core@master] filebackend: inject the proxy backend logger into FileBackendMultiWrite sub-backends

https://gerrit.wikimedia.org/r/654340

Maintenance_bot removed a project: Patch-For-Review.Jan 19 2021, 2:10 AM
Krinkle edited projects, added Wikimedia-production-error, MediaWiki-File-management, Structured Data Engineering; removed Performance-Team.Mar 31 2021, 10:29 PM
Restricted Application added projects: Commons, Structured-Data-Backlog. · View Herald TranscriptMar 31 2021, 10:29 PM
Krinkle merged a task: T236246: Error restoring file: "The file … is in an inconsistent state within the internal storage backends".Mar 31 2021, 10:29 PM
Krinkle merged a task: T246374: "inconsistent state within the internal storage backends" prohibits undeletion of file versions on Wikimedia Commons.
Krinkle merged a task: T229606: Undeletion error: File is in an inconsistent state within the internal storage backends.
Krinkle added a subtask: T135318: Document how to handle 'inconsistent state within the internal storage backends' issues.
Krinkle added subscribers: MarcoAurelio, Urbanecm, Ezarate.
Krinkle added subscribers: 32X, Raymond.
Krinkle added a subscriber: Majora.
Krinkle moved this task from Untriaged to Older on the Wikimedia-production-error board.Mar 31 2021, 10:54 PM
CBogen moved this task from Triage to Multimedia on the Structured-Data-Backlog board.Apr 5 2021, 4:10 PM
Aklapper added a comment.Jun 23 2021, 8:52 AM

@aaron: All three patches have been merged. Can this task be resolved (via Add Action... → Change Status in the dropdown), or is there more to do in this task?

JGHowes closed this task as Resolved.Jun 23 2021, 11:48 AM
JGHowes claimed this task.
Aklapper reassigned this task from JGHowes to aaron.Jun 23 2021, 12:47 PM
aaron added a comment.Jul 1 2021, 5:18 PM
In T270994#7171810, @Aklapper wrote:

@aaron: All three patches have been merged. Can this task be resolved (via Add Action... → Change Status in the dropdown), or is there more to do in this task?

LGTM.

Ezarate added a comment.Jul 1 2021, 7:41 PM

I have the same problem it isn't solved

Aklapper added a comment.Jul 2 2021, 1:58 AM

@Ezarate: In that case, please follow https://www.mediawiki.org/wiki/How_to_report_a_bug - thanks a lot!

Yann subscribed.Sep 13 2021, 12:57 PM

One more : https://commons.wikimedia.org/wiki/File:Danielle_Trotta_Larry_McReynolds.jpg

MGA73 subscribed.Sep 14 2021, 5:36 PM

Another: https://commons.wikimedia.org/wiki/Special:Undelete/File:Matsudo_in_Chiba_Prefecture_Ja.svg

32X added a comment.Oct 13 2021, 5:02 PM
In T270994#7191930, @Aklapper wrote:

@Ezarate: In that case, please follow https://www.mediawiki.org/wiki/How_to_report_a_bug - thanks a lot!

Funny, because my report T246374 has been merged into this one. This one is "Closed, Resolved", but the general problem wasn't resolved for my bug. I went the more manual way (https://commons.wikimedia.org/w/index.php?title=User_talk:32X&diff=572195600&oldid=564997569), and don't feel like I would open a bug when the next problem occurs. Talking to a wall would be the easier way.

Sreejithk2000 subscribed.EditedFeb 7 2023, 5:03 AM

One more. Unable to undelete this file.

https://commons.wikimedia.org/w/index.php?title=Special:Undelete&target=File%3AWest_Enterance_of_Guangzhoubei_Railway_Station.jpg

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL