Page MenuHomePhabricator

Re-deleting a Commons file: "Error deleting file: The file "mwstore://local-multiwrite/local-deleted/..." is in an inconsistent state within the internal storage backends".
Open, LowPublic

Description

Recently, I deleted https://commons.wikimedia.org/wiki/File:Jack_Settleman_at_Super_Bowl_LIV.jpg as a copyvio. Shortly thereafter, I undeleted it to more closely examine it because of an appeal. Satisfied that this is indeed a copyvio, attempts to again delete the file are unsuccessful, consistently producing this error message:

"Error deleting file: The file "mwstore://local-multiwrite/local-deleted/e/m/0/em06ntsy84nncjbi6k7a9ca6xqk9kh3.jpg" is in an inconsistent state within the internal storage backends".

Event Timeline

Aklapper renamed this task from Bug in "internal storage backends" in re-deleting a Commons file after undelete to Re-deleting a Commons file: "Error deleting file: The file "mwstore://local-multiwrite/local-deleted/..." is in an inconsistent state within the internal storage backends"..Jan 1 2021, 4:52 PM
Aklapper updated the task description. (Show Details)

Apparently now fixed ... successfully deleted at 2350 UTC tonight.

Joe triaged this task as Low priority.Jan 4 2021, 9:34 AM
Joe added a project: Performance-Team.
Joe added subscribers: aaron, Joe.

@aaron do you have any insights on what happened there? See https://logstash.wikimedia.org/goto/04e091fe7e42112a1229b9d207990d1c for the logstash data. It looks like some failures to move the object in one of the two swift clusters originally.

Setting priority to low given the problem solved itself.

Change 654313 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] filebackend: add more logging to SwiftFileBackend for "conservative" sync mode

https://gerrit.wikimedia.org/r/654313

Change 654334 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] filebackend: fix bogus replication error log entries in FileBackendMultiWrite

https://gerrit.wikimedia.org/r/654334

A subset of the log entries are bogus though (should be DEBUG, not ERROR).

At first, I suspected a timeout causing a failed deferred updated, but it seems that no failure was logged likely due to NullLogger being used by sub-backends of FileBackendMultiWrite.

Change 654340 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] filebackend: inject the proxy backend logger into FileBackendMultiWrite sub-backends

https://gerrit.wikimedia.org/r/654340

Change 654313 merged by jenkins-bot:
[mediawiki/core@master] filebackend: add more logging to SwiftFileBackend for "conservative" sync mode

https://gerrit.wikimedia.org/r/654313

Change 654334 merged by jenkins-bot:
[mediawiki/core@master] filebackend: fix bogus replication error log entries in FileBackendMultiWrite

https://gerrit.wikimedia.org/r/654334

Change 654340 merged by jenkins-bot:
[mediawiki/core@master] filebackend: inject the proxy backend logger into FileBackendMultiWrite sub-backends

https://gerrit.wikimedia.org/r/654340