Page MenuHomePhabricator

PHP Warning "Unable to delete stat cache" from file uploads
Open, MediumPublicPRODUCTION ERROR

Description

Error

message
PHP Warning: Unable to delete stat cache for file mwstore://local-swift-###/#######.###

Via Special:Upload:

Request ID: W5Ir5QpAADsAADmp6NYAAABF

stacktrace
#0 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1735): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(239): FileBackendStore->deleteFileCache(string)
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1161): FileBackendStore->deleteInternal(array)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendStore->doQuickOperationsInternal(array)
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendMultiWrite.php(498): FileBackend->doQuickOperations(array)
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendMultiWrite->doQuickOperationsInternal(array)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1065): FileBackend->doQuickOperations(array)
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1154): FileRepo->quickPurgeBatch(array)
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1091): LocalFile->purgeThumbList(string, array)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1789): LocalFile->purgeThumbnails()
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/AutoCommitUpdate.php(42): Closure$LocalFile::recordUpload2(Wikimedia\Rdbms\DatabaseMysqli, string)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(268): AutoCommitUpdate->doUpdate()
#12 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(214): DeferredUpdates::runUpdate(AutoCommitUpdate, Wikimedia\Rdbms\LBFactoryMulti, string, integer)
#13 /srv/mediawiki/php-1.32.0-wmf.20/includes/deferred/DeferredUpdates.php(130): DeferredUpdates::execute(array, string, integer)
#14 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(607): DeferredUpdates::doUpdates(string, integer)
#15 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(576): MediaWiki::preOutputCommit(RequestContext, Closure$MediaWiki::main;1168)
#16 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(884): MediaWiki->doPreOutputCommit(Closure$MediaWiki::main;1168)
#17 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(525): MediaWiki->main()
#18 /srv/mediawiki/php-1.32.0-wmf.20/index.php(42): MediaWiki->run()

Via web action=delete:

Request ID: W5Ir5QpAMFgAAFJ1M7IAAAAF

stacktrace
#0 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1735): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(273): FileBackendStore->deleteFileCache(string)
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/fileop/MoveFileOp.php(93): FileBackendStore->moveInternal(array)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/fileop/FileOp.php(287): MoveFileOp->doAttempt()
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileOpBatch.php(177): FileOp->attempt()
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileOpBatch.php(132): FileOpBatch::runParallelBatches(array, StatusValue)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1115): FileOpBatch::attempt(array, array, NullFileJournal)
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(432): FileBackendStore->doOperationsInternal(array, array)
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendMultiWrite.php(181): FileBackend->doOperations(array, array)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(432): FileBackendMultiWrite->doOperationsInternal(array, array)
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1465): FileBackend->doOperations(array, array)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/LocalRepo.php(593): FileRepo->deleteBatch(array)
#12 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/LocalRepo.php(576): LocalRepo->skipWriteOperationIfSha1(string, array)
#13 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(2709): LocalRepo->deleteBatch(array)
#14 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/file/LocalFile.php(1989): LocalFileDeleteBatch->execute()
#15 /srv/mediawiki/php-1.32.0-wmf.20/includes/FileDeleteForm.php(201): LocalFile->delete(string, boolean, User)
#16 /srv/mediawiki/php-1.32.0-wmf.20/includes/FileDeleteForm.php(119): FileDeleteForm::doDelete(Title, LocalFile, string, string, boolean, User)
#17 /srv/mediawiki/php-1.32.0-wmf.20/includes/page/ImagePage.php(992): FileDeleteForm->execute()
#18 /srv/mediawiki/php-1.32.0-wmf.20/includes/actions/DeleteAction.php(46): ImagePage->delete()
#19 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(501): DeleteAction->show()
#20 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(294): MediaWiki->performAction(ImagePage, Title)
#21 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(868): MediaWiki->performRequest()
#22 /srv/mediawiki/php-1.32.0-wmf.20/includes/MediaWiki.php(525): MediaWiki->main()
#23 /srv/mediawiki/php-1.32.0-wmf.20/index.php(42): MediaWiki->run()

Via API :

Request ID: W5IrsgpAAEwAAAkJHZAAAACY

stacktrace
wiki: commons.wikimedia.org
url: /w/api.php?origin=https%3A%2F%2Fen%2Ewikipedia%2Eorg 

PHP Warning: Unable to delete stat cache for file mwstore://local-swift-codfw/local-temp/a/a0/20180907074139!###.jpg.


#0 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1735): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(170): FileBackendStore->deleteFileCache(string)
#2 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendStore.php(1161): FileBackendStore->storeInternal(array)
#3 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendStore->doQuickOperationsInternal(array)
#4 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackendMultiWrite.php(514): FileBackend->doQuickOperations(array)
#5 /srv/mediawiki/php-1.32.0-wmf.20/includes/libs/filebackend/FileBackend.php(670): FileBackendMultiWrite->doQuickOperationsInternal(array)
#6 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1042): FileBackend->doQuickOperations(array)
#7 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(968): FileRepo->quickImportBatch(array)
#8 /srv/mediawiki/php-1.32.0-wmf.20/includes/filerepo/FileRepo.php(1088): FileRepo->quickImport(string, string)
#9 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadStash.php(256): FileRepo->storeTemp(string, string)
#10 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadBase.php(1124): UploadStash->stashFile(string, string)
#11 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadFromChunks.php(124): UploadBase->doStashFile(User)
#12 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadBase.php(1070): UploadFromChunks->doStashFile(User)
#13 /srv/mediawiki/php-1.32.0-wmf.20/includes/upload/UploadFromChunks.php(76): UploadBase->tryStashFile(User, boolean)
#14 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(315): UploadFromChunks->tryStashFile(User, boolean)
#15 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(178): ApiUpload->performStash(string, array)
#16 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(129): ApiUpload->getWarningsResult(array)
#17 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiUpload.php(104): ApiUpload->getContextResult()
#18 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiMain.php(1587): ApiUpload->execute()
#19 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiMain.php(531): ApiMain->executeAction()
#20 /srv/mediawiki/php-1.32.0-wmf.20/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#21 /srv/mediawiki/php-1.32.0-wmf.20/api.php(87): ApiMain->execute()

Notes

Event Timeline

Krinkle created this task.Sep 26 2018, 6:45 PM
Restricted Application added projects: Commons, Multimedia. · View Herald TranscriptSep 26 2018, 6:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Krinkle added a project: SRE-swift-storage.

Tagging Operations/media-storage as well given this is a fairly unusual backend error. Might be due to an issue Swift rather than MediaWiki.

Krinkle updated the task description. (Show Details)Sep 26 2018, 6:48 PM
Tgr added a subscriber: Tgr.Sep 26 2018, 7:05 PM

That's thrown in response of memcache error. IIRC most BagOStuff subclasses do not have any usefuk error logging, so that's probably the place to start.

Krinkle added a comment.EditedOct 4 2018, 6:51 PM

There's a similar error that appears to be a variation of this that I'll map to the same task for now. Let me know if this should be separate.

channel: FileOperation
level: WARNING
message:

FileBackendStore::getFileStat: Could not stat file mwstore://local-swift-codfw/timeline-render/637509851a7e6035db769e7c120f9d63.err.

Seen about 2,000 times in the last 30 days, of which the vast majority were only in the last 24 hours - which could be a coincidence but perhaps worth checking on.

Maybe related: T205567

Change 464622 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464622

Change 464622 merged by jenkins-bot:
[mediawiki/core@master] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464622

Change 464735 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@wmf/1.32.0-wmf.24] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464735

Change 464735 merged by jenkins-bot:
[mediawiki/core@wmf/1.32.0-wmf.24] filebackend: Add normalization for stat errors

https://gerrit.wikimedia.org/r/464735

Mentioned in SAL (#wikimedia-operations) [2018-10-05T04:18:03Z] <krinkle@deploy1001> Synchronized php-1.32.0-wmf.24/includes/libs/filebackend/FileBackendStore.php: T205567 - I75f1eb6dc2cb (duration: 00m 56s)

Mentioned in SAL (#wikimedia-operations) [2018-10-10T00:15:48Z] <krinkle@deploy1001> sync-file aborted: T205567 - I75f1eb6dc2cb (duration: 00m 01s)

Ramsey-WMF moved this task from Untriaged to Tracking on the Multimedia board.Oct 17 2018, 5:12 PM
jijiki triaged this task as Medium priority.Oct 23 2018, 12:18 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM