Page MenuHomePhabricator
Create Task
Maniphest T205567

PHP Warning "Unable to delete stat cache" from file uploads
Open, MediumPublicPRODUCTION ERROR
Actions

Description

Error

message
PHP Warning: Unable to delete stat cache for file mwstore://local-swift-###/#######.###

Via Special:Upload:

Request ID: W5Ir5QpAADsAADmp6NYAAABF

stacktrace
#0 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1735): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(239): FileBackendStore->deleteFileCache(string)
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1161): FileBackendStore->deleteInternal(array)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendStore->doQuickOperationsInternal(array)
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendMultiWrite.php(498): FileBackend->doQuickOperations(array)
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendMultiWrite->doQuickOperationsInternal(array)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1065): FileBackend->doQuickOperations(array)
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1154): FileRepo->quickPurgeBatch(array)
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1091): LocalFile->purgeThumbList(string, array)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1789): LocalFile->purgeThumbnails()
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/AutoCommitUpdate.php(42): Closure$LocalFile::recordUpload2(Wikimedia\Rdbms\DatabaseMysqli, string)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(268): AutoCommitUpdate->doUpdate()
#12 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(214): DeferredUpdates::runUpdate(AutoCommitUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#13 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(130): DeferredUpdates::execute(array, string, integer)
#14 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(607): DeferredUpdates::doUpdates(string, integer)
#15 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(576): MediaWiki::preOutputCommit(RequestContext, Closure$MediaWiki::main;1168)
#16 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(884): MediaWiki->doPreOutputCommit(Closure$MediaWiki::main;1168)
#17 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(525): MediaWiki->main()
#18 /srv/mediawiki/php-1.32.0-wmf.20/index.php(42): MediaWiki->run()

Via web action=delete:

Request ID: W5Ir5QpAMFgAAFJ1M7IAAAAF

stacktrace
#0 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1735): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(273): FileBackendStore->deleteFileCache(string)
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/fileop/MoveFileOp.php(93): FileBackendStore->moveInternal(array)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/fileop/FileOp.php(287): MoveFileOp->doAttempt()
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileOpBatch.php(177): FileOp->attempt()
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileOpBatch.php(132): FileOpBatch::runParallelBatches(array, StatusValue)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1115): FileOpBatch::attempt(array, array, NullFileJournal)
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(432): FileBackendStore->doOperationsInternal(array, array)
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendMultiWrite.php(181): FileBackend->doOperations(array, array)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(432): FileBackendMultiWrite->doOperationsInternal(array, array)
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1465): FileBackend->doOperations(array, array)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/LocalRepo.php(593): FileRepo->deleteBatch(array)
#12 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/LocalRepo.php(576): LocalRepo->skipWriteOperationIfSha1(string, array)
#13 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(2709): LocalRepo->deleteBatch(array)
#14 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1989): LocalFileDeleteBatch->execute()
#15 /srv/mediawiki/php-1.32.0-wmf.20/includes/FileDeleteForm.php(201): LocalFile->delete(string, boolean, User)
#16 /srv/mediawiki/php-1.32.0-wmf.20/includes/FileDeleteForm.php(119): FileDeleteForm::doDelete(Title, LocalFile, string, string, boolean, User)
#17 /srv/mediawiki/php-1.32.0-wmf.20/includes/page/ImagePage.php(992): FileDeleteForm->execute()
#18 /srv/mediawiki/php-1.32.0-wmf.20/includes/actions/DeleteAction.php(46): ImagePage->delete()
#19 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(501): DeleteAction->show()
#20 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(294): MediaWiki->performAction(ImagePage, Title)
#21 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(868): MediaWiki->performRequest()
#22 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(525): MediaWiki->main()
#23 /srv/mediawiki/php-1.32.0-wmf.20/index.php(42): MediaWiki->run()

Via API :

Request ID: W5IrsgpAAEwAAAkJHZAAAACY

stacktrace
wiki: commons.wikimedia.org
url: /w/api.php?origin=https%3A%2F%2Fen%2Ewikipedia%2Eorg 

PHP Warning: Unable to delete stat cache for file mwstore://local-swift-codfw/local-temp/a/a0/20180907074139!###.jpg.


#0 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1735): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(170): FileBackendStore->deleteFileCache(string)
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1161): FileBackendStore->storeInternal(array)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendStore->doQuickOperationsInternal(array)
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendMultiWrite.php(514): FileBackend->doQuickOperations(array)
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendMultiWrite->doQuickOperationsInternal(array)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1042): FileBackend->doQuickOperations(array)
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(968): FileRepo->quickImportBatch(array)
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1088): FileRepo->quickImport(string, string)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadStash.php(256): FileRepo->storeTemp(string, string)
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadBase.php(1124): UploadStash->stashFile(string, string)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadFromChunks.php(124): UploadBase->doStashFile(User)
#12 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadBase.php(1070): UploadFromChunks->doStashFile(User)
#13 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadFromChunks.php(76): UploadBase->tryStashFile(User, boolean)
#14 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(315): UploadFromChunks->tryStashFile(User, boolean)
#15 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(178): ApiUpload->performStash(string, array)
#16 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(129): ApiUpload->getWarningsResult(array)
#17 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(104): ApiUpload->getContextResult()
#18 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiMain.php(1587): ApiUpload->execute()
#19 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiMain.php(531): ApiMain->executeAction()
#20 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#21 /srv/mediawiki/php-1.32.0-wmf.20/api.php(87): ApiMain->execute()

Notes

Details

SubjectRepoBranchLines +/-
filebackend: Add normalization for stat errorsmediawiki/corewmf/1.32.0-wmf.24+12 -4
filebackend: Add normalization for stat errorsmediawiki/coremaster+12 -4
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Sep 26 2018, 6:45 PM
Restricted Application added projects: Commons, Multimedia. · View Herald TranscriptSep 26 2018, 6:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle added a project: SRE.Sep 26 2018, 6:46 PM
Krinkle added a project: SRE-swift-storage.

Tagging Operations/media-storage as well given this is a fairly unusual backend error. Might be due to an issue Swift rather than MediaWiki.

Krinkle updated the task description. (Show Details)Sep 26 2018, 6:48 PM
Tgr subscribed.Sep 26 2018, 7:05 PM

That's thrown in response of memcache error. IIRC most BagOStuff subclasses do not have any usefuk error logging, so that's probably the place to start.

• MoritzMuehlenhoff added a subscriber: fgiunchedi.Sep 27 2018, 7:51 AM
Krinkle added a comment.EditedOct 4 2018, 6:51 PM

There's a similar error that appears to be a variation of this that I'll map to the same task for now. Let me know if this should be separate.

channel: FileOperation
level: WARNING
message:

FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-codfw/timeline-render/637509851a7e6035db769e7c120f9d63.err.

Seen about 2,000 times in the last 30 days, of which the vast majority were only in the last 24 hours - which could be a coincidence but perhaps worth checking on.

Maybe related: T205567

gerritbot subscribed.Oct 4 2018, 6:55 PM

Change 464622 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464622

gerritbot added a project: Patch-For-Review.Oct 4 2018, 6:55 PM
gerritbot added a comment.Oct 4 2018, 11:22 PM

Change 464622 merged by jenkins-bot:
[mediawiki/core@master] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464622

gerritbot added a comment.Oct 4 2018, 11:58 PM

Change 464735 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@wmf/1.32.0-wmf.24] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464735

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)).Oct 5 2018, 12:00 AM
gerritbot added a comment.Oct 5 2018, 3:11 AM

Change 464735 merged by jenkins-bot:
[mediawiki/core@wmf/1.32.0-wmf.24] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464735

ReleaseTaggerBot edited projects, added MW-1.32-notes (WMF-deploy-2018-10-02 (1.32.0-wmf.24)); removed MW-1.32-notes (WMF-deploy-2018-10-16 (1.32.0-wmf.26)).Oct 5 2018, 4:00 AM
Stashbot subscribed.Oct 5 2018, 4:18 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-05T04:18:03Z] <krinkle@deploy1001> Synchronized php-1.32.0-wmf.24/includes/libs/filebackend/FileBackendStore.php: T205567 - I75f1eb6dc2cb (duration: 00m 56s)

Stashbot added a comment.Oct 10 2018, 12:15 AM

Mentioned in SAL (#wikimedia-operations) [2018-10-10T00:15:48Z] <krinkle@deploy1001> sync-file aborted: T205567 - I75f1eb6dc2cb (duration: 00m 01s)

Krinkle removed a project: Patch-For-Review.Oct 10 2018, 9:00 PM
• Ramsey-WMF moved this task from Untriaged to Tracking on the Multimedia board.Oct 17 2018, 5:12 PM
jijiki triaged this task as Medium priority.Oct 23 2018, 12:18 PM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 10:26 PM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM
Krinkle mentioned this in T71760: Error "An unknown error occurred in storage backend" when deleting files.Mar 12 2020, 1:42 AM
Krinkle mentioned this in T244567: Some files cannot be deleted "Error deleting file: An unknown error occurred in storage backend "local-multiwrite". " (due to missing physical file for old image entries).
Krinkle mentioned this in T240281: Clarify confusion between Multimedia and Structured Data team and update docs accordingly.Jun 23 2020, 7:02 PM
TheDJ subscribed.Jun 4 2024, 10:12 PM

In the last 7 days there were 85 log entries for this warning.

48 of these were on labswiki, triggered by /srv/mediawiki/multiversion/MWScript.php maintenance/cleanupUploadStash.php --wiki=labswiki

Unable to delete stat cache for file mwstore://local-swift-eqiad/local-temp/3/3e/20240603133035!chunkedupload_f27cc7b142cf.png.

All remaining 37 were from commonswiki

  • One Unable to delete stat cache for file mwstore://local-swift-eqiad/local-temp/6/64/1aysso4sxrnk.bqskoh.7498417.tif.9 triggered on Special:UploadWizard
  • And 36 on a canary host for the url /w/api.php?action=purge&assert=user&forcerecursivelinkupdate=1&format=json with entries like:
Unable to delete stat cache for file mwstore://local-swift-codfw/local-thumb/3/39/Mallows_Bay_(49054523096).jpg/{width}px-Mallows_Bay_(49054523096).jpg
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits