Page MenuHomePhabricator

Some edits fail to save due to BlobAccessException: "Unable to store text to external storage"
Closed, ResolvedPublic

Description

  1. I viewed https://www.mediawiki.org/wiki/CDB
  2. Pressed Edit, leading to https://www.mediawiki.org/w/index.php?title=CDB&action=edit
  3. Made a minor change and clicked Publish changes.

The response was

[XmkhwgpAAEMAAAzY7RAAAADH] 2020-03-11 17:37:06: Fatal exception of type "MediaWiki\Storage\BlobAccessException"
Error message
[{exMediaWiki\Storage\BlobAccessException: Unable to store text to external storage
 from line 261 of /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php
exception.previous.trace
#0 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php(230): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(947): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#2 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(684): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(619): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#4 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(516): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#5 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#6 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#7 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(518): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#9 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#10 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.23/includes/page/WikiPage.php(1905): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(2388): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#13 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(1659): EditPage->internalAttemptSave(array, boolean)
#14 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(704): EditPage->attemptSave(array)
#15 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/EditAction.php(60): EditPage->edit()
#16 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/SubmitAction.php(38): EditAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(519): SubmitAction->show()
#18 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#19 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(972): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(535): MediaWiki->main()
#21 /srv/mediawiki/php-1.35.0-wmf.23/index.php(47): MediaWiki->run()
#22 /srv/mediawiki/w/index.php(3): require(string)
#23 {main}
exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(947): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(684): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#2 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(619): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(516): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#4 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#5 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#6 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(518): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#8 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#9 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#10 /srv/mediawiki/php-1.35.0-wmf.23/includes/page/WikiPage.php(1905): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(2388): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(1659): EditPage->internalAttemptSave(array, boolean)
#13 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(704): EditPage->attemptSave(array)
#14 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/EditAction.php(60): EditPage->edit()
#15 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/SubmitAction.php(38): EditAction->show()
#16 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(519): SubmitAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#18 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(972): MediaWiki->performRequest()
#19 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(535): MediaWiki->main()
#20 /srv/mediawiki/php-1.35.0-wmf.23/index.php(47): MediaWiki->run()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}
Impact

Saving or publishing edits on the wikis sometimes fails, the abckend system appears to reject them sometimes.

The error is a fatal error with HTTP 5xx, which monitoring interpets as decreased system stabillity. This can results in aborted deployments and alerts.

Notes

May be related:

Details

Request ID
XmkhwgpAAEMAAAzY7RAAAADH

Event Timeline

Krinkle created this task.Mar 11 2020, 5:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2020, 5:44 PM
Krinkle set Request ID to XmkhwgpAAEMAAAzY7RAAAADH.Mar 11 2020, 5:46 PM

Also seen elsewhere in prod, for example:

`
[Xmq93ApAEJsAA-ZeZA8AAAAF] /w/index.php?title= … &action=submit   MediaWiki\Storage\BlobAccessException from line 261 of /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/SqlBlobStore.php: Unable to store text to external storage
#0 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/SqlBlobStore.php(230): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(948): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#2 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(685): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(620): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#4 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(517): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#5 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#6 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#7 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(519): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#9 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#10 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.22/includes/page/WikiPage.php(1942): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(2385): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#13 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(1657): EditPage->internalAttemptSave(array, boolean)
#14 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(704): EditPage->attemptSave(array)
#15 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/EditAction.php(60): EditPage->edit()
#16 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/SubmitAction.php(38): EditAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(519): SubmitAction->show()
#18 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#19 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(972): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(535): MediaWiki->main()
#21 /srv/mediawiki/php-1.35.0-wmf.22/index.php(47): MediaWiki->run()
#22 /srv/mediawiki/w/index.php(3): require(string)
#23 {main}
t exception.trace 	
	#0 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(948): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(685): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#2 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(620): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(517): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#4 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#5 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#6 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(519): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#8 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#9 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#10 /srv/mediawiki/php-1.35.0-wmf.22/includes/page/WikiPage.php(1942): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(2385): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(1657): EditPage->internalAttemptSave(array, boolean)
#13 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(704): EditPage->attemptSave(array)
#14 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/EditAction.php(60): EditPage->edit()
#15 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/SubmitAction.php(38): EditAction->show()
#16 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(519): SubmitAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#18 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(972): MediaWiki->performRequest()
#19 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(535): MediaWiki->main()
#20 /srv/mediawiki/php-1.35.0-wmf.22/index.php(47): MediaWiki->run()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}

This seems quite serious that edits are being aborted fatally at the edit save due to some kind of ExternalStore failure.

Tagging CPT for investigation.

Note that there are also a dozen reports about this failure specific to the Beta Cluster, which could be unrelated, or it might be the same root cause in a way that perhaps is just more amplified there for some reason. See T228088 and associated tasks.

Krinkle updated the task description. (Show Details)

@Krinkle can you update the priority for this task?

daniel triaged this task as High priority.Mar 30 2020, 2:37 PM
AMooney assigned this task to daniel.Mar 30 2020, 3:25 PM

Change 584622 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] ExternalStore: report cause of non-exception failure

https://gerrit.wikimedia.org/r/584622

From looking at ExternalStoreAccess::insert(), it seems like this may be caused by the database being in readonly state. The error modes and reporting in that method should be cleaned up. At the very least, the ExternalStoreException thrown at the very end of the method should include $msg.

Logically, we should be throwing a ReadOnlyError if a blob can't be written because the DB is in read only mode.

Anomie added a subscriber: Anomie.Mar 30 2020, 4:25 PM

From looking at ExternalStoreAccess::insert(), it seems like this may be caused by the database being in readonly state. The error modes and reporting in that method should be cleaned up. At the very least, the ExternalStoreException thrown at the very end of the method should include $msg.

I see there's already logging of the failures to the ExternalStore channel. Over the past 30 days, all the relevant messages to that channel have had the reason as "read only".

I made a patch to throw ReadOnlyError in such a case.

Change 584622 merged by jenkins-bot:
[mediawiki/core@master] ExternalStore: report cause of non-exception failure

https://gerrit.wikimedia.org/r/584622

daniel closed this task as Resolved.Apr 17 2020, 8:57 PM