Page MenuHomePhabricator

Some files cannot be deleted "Error deleting file: An unknown error occurred in storage backend "local-multiwrite". "
Open, HighPublic

Event Timeline

Reedy renamed this task from Deletion of two files on enWikipedia fails with the error "Error deleting file: An unknown error occurred in storage backend "local-multiwrite". " to Deletion of two files on enwiki fails with the error "Error deleting file: An unknown error occurred in storage backend "local-multiwrite". ".Feb 7 2020, 10:13 PM

If it helps, it also seems that such files tend to fall off their categories. E.g these two files on Friday didn't appear on https://en.wikipedia.org/wiki/Category:Expired_proposed_deletions, once I null-edited them they appeared there, and now they don't display again.

GeoffreyT2000 added a subscriber: GeoffreyT2000.

Matching the priority of T141704.

I have added a warning template to the two files affected by this task.

From Logstash, reqId 5b36780cb13d1c18b9633f19 for mwmaint1002$ /srv/mediawiki-staging/multiversion/MWScript.php deleteBatch.php on 2020-02-07:

  • [ERROR FileOperation] FileBackendStore::ingestFreshFileStats: Could not stat file mwstore://local-swift-eqiad/local-public/archive/1/1f/
  • [ERROR FileOperation] FileBackendStore::ingestFreshFileStats: Could not stat file mwstore://local-swift-eqiad/local-public/archive/1/1f/
  • [ERROR exception] DBTransactionError: Explicit transaction still active. A caller may have caught an error. Open transactions: LocalFile::lockingTransaction
#0 /srv/mediawiki/php-1.35.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1637): Wikimedia\Rdbms\Database->assertNoOpenTransactions()
#1 /srv/mediawiki/php-1.35.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(2145): Wikimedia\Rdbms\LoadBalancer->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\DatabaseMysqli)
#2 /srv/mediawiki/php-1.35.0-wmf.16/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1656): Wikimedia\Rdbms\LoadBalancer->forEachOpenMasterConnection(Closure)
#3 /srv/mediawiki/php-1.35.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactory.php(211): Wikimedia\Rdbms\LoadBalancer->approveMasterChanges(array, string, integer)
#4 /srv/mediawiki/php-1.35.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(241): Wikimedia\Rdbms\LBFactory->Wikimedia\Rdbms\{closure}(Wikimedia\Rdbms\LoadBalancer, string, array)
#5 /srv/mediawiki/php-1.35.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactory.php(213): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure, array)
#6 /srv/mediawiki/php-1.35.0-wmf.16/includes/libs/rdbms/lbfactory/LBFactory.php(272): Wikimedia\Rdbms\LBFactory->forEachLBCallMethod(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.16/maintenance/doMaintenance.php(127): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string)
#8 /srv/mediawiki/php-1.35.0-wmf.16/maintenance/deleteBatch.php(128): require_once(string)

The same CLI process also spawned jobs (passing through the same reqId), which emitted the following errors:

  • [ERROR JobExecutor] mw1335 Failed executing job: deleteLinks File:CallipygianVenus.jpg pageId=9558870 timestamp=20200207172755 namespace=6 title=CallipygianVenus.jpg requestId=5b36780cb13d1c18b9633f19
    • deleteLinks: Page #9558870 exists
  • [ERROR JobExecutor] mw1295 Failed executing job: deleteLinks File:CallipygianVenus.jpg pageId=9558870 timestamp=20200207172755 namespace=6 title=CallipygianVenus.jpg requestId=5b36780cb13d1c18b9633f19
    • deleteLinks: Page #9558870 exists
  • [ERROR JobExecutor] mw1296 Failed executing job: deleteLinks File:CallipygianVenus.jpg pageId=9558870 timestamp=20200207172755 namespace=6 title=CallipygianVenus.jpg requestId=5b36780cb13d1c18b9633f19
    • deleteLinks: Page #9558870 exists

The FileOp errors might be related to T205567.

Just a note, this is not peculiar to enwiki as it apparently happened on Commons some few days ago. This file is due to be deleted but it cannot be deleted from the UI.

Ammarpad renamed this task from Deletion of two files on enwiki fails with the error "Error deleting file: An unknown error occurred in storage backend "local-multiwrite". " to Some files cannot be deleted "Error deleting file: An unknown error occurred in storage backend "local-multiwrite". ".Apr 3 2020, 7:38 AM

Similliar problem on idwiki, https://id.wikipedia.org/wiki/Berkas:Kementerian_Lingkungan_Hidup.svg can't be deleted with an error message: Kesalahan waktu menghapus berkas: Kesalahan yang tidak dikenal terjadi di backend penyimpanan "local-multiwrite".

From {T259443]:

…
Caused by: [Exception ApiUsageException] (/srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiBase.php:1460) An unknown error occurred in storage backend "local-multiwrite".
  #0 /srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiDelete.php(92): ApiBase->dieStatus(Status)
  #1 /srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiMain.php(1593): ApiDelete->execute()
  #2 /srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiMain.php(529): ApiMain->executeAction()
  #3 /srv/mediawiki/php-1.36.0-wmf.2/includes/api/ApiMain.php(500): ApiMain->executeActionWithErrorHandling()
  #4 /srv/mediawiki/php-1.36.0-wmf.2/api.php(90): ApiMain->execute()
  #5 /srv/mediawiki/php-1.36.0-wmf.2/api.php(45): wfApiMain()
  #6 /srv/mediawiki/w/api.php(3): require(string)
  #7 {main}

There are 35 files in https://commons.wikimedia.org/wiki/Category:Deletion_error/T244567 that are affected by this bug. If a file has multiple file revisions, all but one can be deleted. Trying to delete the offending file revision or the entire page results in the error. In some cases, like with https://commons.wikimedia.org/wiki/File:Sourabh_Sharma.jpg, that revision is duplicated (see https://quarry.wmflabs.org/query/48722). In all cases, oi_archive_name for the broken revision is null (see https://quarry.wmflabs.org/query/48730).

The [ERROR FileOperation] FileBackendStore::ingestFreshFileStats: Could not stat file mwstore://local-swift-eqiad/local-public/archive/1/1f/ that Krinkle found in the logs also shows the lack of a proper archive name, as it should appear in the file path.

I'm thinking that the missing oi_archive_name is an effect, not a cause though. The issue began to occur before many of these files were overwritten.

As was previously noted, the affected files seem to regularly fall out of categorylinks and templatelinks as well.

The both files from the task description does not have a archive name, because the file does not exists, the physical file seems lost from disc and now prevent the deletion.
The UI does see this, because there is some special code in the exists function

	/**
	 * If archive name is an empty string, then file does not "exist"
	 *
	 * This is the case for a couple files on Wikimedia servers where
	 * the old version is "lost".
	 * @return bool
	 */
	public function exists() {
		$archiveName = $this->getArchiveName();
		if ( $archiveName === '' || !is_string( $archiveName ) ) {
			return false;
		}
		return parent::exists();
	}

This does not take effect in the deletion code. Be careful when changing code to not leave the broken rows in the oldimage table by skipping the bad archive names. Not sure how to resolve the deletion of the db and not the file (to keep it for restore and for the history)

For the atomic messages I have created https://gerrit.wikimedia.org/r/c/mediawiki/core/+/632300

Linking affected files on Commons https://commons.wikimedia.org/wiki/Category:Deletion_error/T244567 so I can delete them when the bug is fixed.

I am getting the same error when moving the file https://commons.wikimedia.org/wiki/File:Hallgrimskirkja_(2).jpg to the name File:Hallgrimskirkja.jpg.

Change 666975 had a related patch set uploaded (by Itamar Givon; owner: Itamar Givon):
[mediawiki/extensions/WikibaseCirrusSearch@master] Update getLocalEntityTypes to use static method

https://gerrit.wikimedia.org/r/666975

Sorry about that, mistakenly picked up the wrong ticket number

The error I get trying to restore that is

Error undeleting file: The file "mwstore://local-multiwrite/local-public/d/d6/Zholtovsky_tarasov_house.jpg" is in an inconsistent state within the internal storage backends

Interestingly, the original is visible at https://commons.wikimedia.org/w/index.php?title=Special:Undelete&target=File%3AZholtovsky+tarasov+house.jpg&file=gafir9wwafkuafy1reuwbpsafxnbzri.jpg, unlike the un-deletable files where the file revision is broken entirely.
I tried re-uploading the file from the archive, but got the same error.
I moved the text revisions to https://commons.wikimedia.org/wiki/File:Zholtovsky_tarasov_house_(1).jpg and re-uploaded the file there, so it's at least accessible now. The unrestorable file revision remains at the original page.