Page MenuHomePhabricator
Create Task
Maniphest T247429

Some edits fail to save due to BlobAccessException: "Unable to store text to external storage"
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

  1. I viewed https://www.mediawiki.org/wiki/CDB
  2. Pressed Edit, leading to https://www.mediawiki.org/w/index.php?title=CDB&action=edit
  3. Made a minor change and clicked Publish changes.

The response was

[XmkhwgpAAEMAAAzY7RAAAADH] 2020-03-11 17:37:06: Fatal exception of type "MediaWiki\Storage\BlobAccessException"
Error message
[{exMediaWiki\Storage\BlobAccessException: Unable to store text to external storage
 from line 261 of /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php
exception.previous.trace
#0 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/SqlBlobStore.php(230): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(947): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#2 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(684): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(619): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#4 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(516): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#5 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#6 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#7 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(518): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#9 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#10 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.23/includes/page/WikiPage.php(1905): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(2388): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#13 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(1659): EditPage->internalAttemptSave(array, boolean)
#14 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(704): EditPage->attemptSave(array)
#15 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/EditAction.php(60): EditPage->edit()
#16 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/SubmitAction.php(38): EditAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(519): SubmitAction->show()
#18 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#19 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(972): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(535): MediaWiki->main()
#21 /srv/mediawiki/php-1.35.0-wmf.23/index.php(47): MediaWiki->run()
#22 /srv/mediawiki/w/index.php(3): require(string)
#23 {main}
exception.trace
#0 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(947): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(684): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#2 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(619): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(516): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#4 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#5 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#6 /srv/mediawiki/php-1.35.0-wmf.23/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.23/includes/Revision/RevisionStore.php(518): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#8 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#9 /srv/mediawiki/php-1.35.0-wmf.23/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#10 /srv/mediawiki/php-1.35.0-wmf.23/includes/page/WikiPage.php(1905): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(2388): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(1659): EditPage->internalAttemptSave(array, boolean)
#13 /srv/mediawiki/php-1.35.0-wmf.23/includes/EditPage.php(704): EditPage->attemptSave(array)
#14 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/EditAction.php(60): EditPage->edit()
#15 /srv/mediawiki/php-1.35.0-wmf.23/includes/actions/SubmitAction.php(38): EditAction->show()
#16 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(519): SubmitAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#18 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(972): MediaWiki->performRequest()
#19 /srv/mediawiki/php-1.35.0-wmf.23/includes/MediaWiki.php(535): MediaWiki->main()
#20 /srv/mediawiki/php-1.35.0-wmf.23/index.php(47): MediaWiki->run()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}
Impact

Saving or publishing edits on the wikis sometimes fails, the abckend system appears to reject them sometimes.

The error is a fatal error with HTTP 5xx, which monitoring interpets as decreased system stabillity. This can results in aborted deployments and alerts.

Notes

May be related:

Details

Request ID
XmkhwgpAAEMAAAzY7RAAAADH
SubjectRepoBranchLines +/-
ExternalStore: report cause of non-exception failuremediawiki/coremaster+45 -23
Customize query in gerrit

Related Objects

Event Timeline

Krinkle created this task.Mar 11 2020, 5:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2020, 5:44 PM
Krinkle set Request ID to XmkhwgpAAEMAAAzY7RAAAADH.Mar 11 2020, 5:46 PM
Krinkle moved this task from Untriaged to Mar 2020 on the Wikimedia-production-error board.Mar 12 2020, 1:42 AM
Krinkle added a comment.Mar 12 2020, 11:12 PM

Also seen elsewhere in prod, for example:

`
[Xmq93ApAEJsAA-ZeZA8AAAAF] /w/index.php?title= … &action=submit   MediaWiki\Storage\BlobAccessException from line 261 of /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/SqlBlobStore.php: Unable to store text to external storage
#0 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/SqlBlobStore.php(230): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(948): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#2 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(685): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(620): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#4 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(517): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#5 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#6 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#7 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#8 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(519): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#9 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#10 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.22/includes/page/WikiPage.php(1942): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(2385): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#13 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(1657): EditPage->internalAttemptSave(array, boolean)
#14 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(704): EditPage->attemptSave(array)
#15 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/EditAction.php(60): EditPage->edit()
#16 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/SubmitAction.php(38): EditAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(519): SubmitAction->show()
#18 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#19 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(972): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(535): MediaWiki->main()
#21 /srv/mediawiki/php-1.35.0-wmf.22/index.php(47): MediaWiki->run()
#22 /srv/mediawiki/w/index.php(3): require(string)
#23 {main}
t exception.trace 	
	#0 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(948): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#1 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(685): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#2 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(620): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(517): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#4 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/Database.php(4045): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#5 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#6 /srv/mediawiki/php-1.35.0-wmf.22/includes/libs/rdbms/database/DBConnRef.php(650): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.35.0-wmf.22/includes/Revision/RevisionStore.php(519): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure)
#8 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(996): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
#9 /srv/mediawiki/php-1.35.0-wmf.22/includes/Storage/PageUpdater.php(766): MediaWiki\Storage\PageUpdater->doModify(CommentStoreComment, User, integer)
#10 /srv/mediawiki/php-1.35.0-wmf.22/includes/page/WikiPage.php(1942): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
#11 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(2385): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
#12 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(1657): EditPage->internalAttemptSave(array, boolean)
#13 /srv/mediawiki/php-1.35.0-wmf.22/includes/EditPage.php(704): EditPage->attemptSave(array)
#14 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/EditAction.php(60): EditPage->edit()
#15 /srv/mediawiki/php-1.35.0-wmf.22/includes/actions/SubmitAction.php(38): EditAction->show()
#16 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(519): SubmitAction->show()
#17 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(305): MediaWiki->performAction(Article, Title)
#18 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(972): MediaWiki->performRequest()
#19 /srv/mediawiki/php-1.35.0-wmf.22/includes/MediaWiki.php(535): MediaWiki->main()
#20 /srv/mediawiki/php-1.35.0-wmf.22/index.php(47): MediaWiki->run()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}

This seems quite serious that edits are being aborted fatally at the edit save due to some kind of ExternalStore failure.

Tagging CPT for investigation.

Note that there are also a dozen reports about this failure specific to the Beta Cluster, which could be unrelated, or it might be the same root cause in a way that perhaps is just more amplified there for some reason. See T228088 and associated tasks.

Krinkle moved this task from Untriaged to ExternalStore on the MediaWiki-Core-Revision-backend board.Mar 12 2020, 11:12 PM
Krinkle updated the task description. (Show Details)
• Pchelolo edited projects, added Platform Team Workboards (Clinic Duty Team); removed Platform Engineering.Mar 13 2020, 9:50 AM
Krinkle mentioned this in T228088: BetaCluster: ExternalStoreException - Unable to store text to external storage.Mar 13 2020, 3:47 PM
• AMooney subscribed.Mar 27 2020, 12:07 PM

@Krinkle can you update the priority for this task?

daniel triaged this task as High priority.Mar 30 2020, 2:37 PM
daniel moved this task from Inbox to Later on the Platform Team Workboards (Clinic Duty Team) board.
• AMooney assigned this task to daniel.Mar 30 2020, 3:25 PM
• AMooney moved this task from Later to Ready (WIP:5) on the Platform Team Workboards (Clinic Duty Team) board.
gerritbot added a comment.Mar 30 2020, 3:43 PM

Change 584622 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] ExternalStore: report cause of non-exception failure

https://gerrit.wikimedia.org/r/584622

gerritbot added a project: Patch-For-Review.Mar 30 2020, 3:43 PM
daniel added a comment.Mar 30 2020, 3:44 PM

From looking at ExternalStoreAccess::insert(), it seems like this may be caused by the database being in readonly state. The error modes and reporting in that method should be cleaned up. At the very least, the ExternalStoreException thrown at the very end of the method should include $msg.

daniel added a comment.Mar 30 2020, 3:51 PM

Logically, we should be throwing a ReadOnlyError if a blob can't be written because the DB is in read only mode.

Anomie subscribed.Mar 30 2020, 4:25 PM
In T247429#6010996, @daniel wrote:

From looking at ExternalStoreAccess::insert(), it seems like this may be caused by the database being in readonly state. The error modes and reporting in that method should be cleaned up. At the very least, the ExternalStoreException thrown at the very end of the method should include $msg.

I see there's already logging of the failures to the ExternalStore channel. Over the past 30 days, all the relevant messages to that channel have had the reason as "read only".

daniel added a comment.Mar 30 2020, 5:07 PM

I made a patch to throw ReadOnlyError in such a case.

daniel moved this task from Ready (WIP:5) to Waiting for Review on the Platform Team Workboards (Clinic Duty Team) board.Mar 30 2020, 5:13 PM
gerritbot added a comment.Apr 17 2020, 4:15 PM

Change 584622 merged by jenkins-bot:
[mediawiki/core@master] ExternalStore: report cause of non-exception failure

https://gerrit.wikimedia.org/r/584622

DannyS712 subscribed.Apr 17 2020, 4:23 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 17 2020, 5:11 PM
daniel closed this task as Resolved.Apr 17 2020, 8:57 PM
daniel moved this task from Waiting for Review to Done on the Platform Team Workboards (Clinic Duty Team) board.
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:40 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits