Page MenuHomePhabricator

Many "Unable to store text to external storage" exception on beta (read-only)
Closed, DuplicatePublic

Description

When attempting to edit a page on beta (commonswiki on beta, in this case, but probably equally true for other wikis), there will be an "Unable to store text to external storage" exception every now and then.
I get it like at about 1/3rd of my attempts, so relatively easy to reproduce (although I'm not sure if that's similarly divided throughout the rest of the day), and more than common enough to be a massive annoyance.

Here's one example's ExternalStore.log entries:

2019-09-02 08:44:00 [XWzWUKwQBHcAAH0j@8IAAABI] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore DEBUG: ExternalStoreAccess::insert: trying DB://cluster1

2019-09-02 08:44:00 [XWzWUKwQBHcAAH0j@8IAAABI] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"}

And this was the exception trace, from exception.log.

2019-09-02 08:44:00 [XWzWUKwQBHcAAH0j@8IAAABI] deployment-mediawiki-07 commonswiki 1.34.0-alpha exception ERROR: [XWzWUKwQBHcAAH0j@8IAAABI] /w/index.php?title=TestPage_6513304179583546&action=submit   MediaWiki\Storage\BlobAccessException from line 263 of /srv/mediawiki/php-master/includes/Storage/SqlBlobStore.php: Unable to store text to external storage {"exception_id":"XWzWUKwQBHcAAH0j@8IAAABI","exception_url":"/w/index.php?title=TestPage_6513304179583546&action=submit","caught_by":"mwe_handler"}
[Exception MediaWiki\Storage\BlobAccessException] (/srv/mediawiki/php-master/includes/Storage/SqlBlobStore.php:263) Unable to store text to external storage
  #0 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(932): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
  #1 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(668): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
  #2 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(603): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
  #3 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(501): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
  #4 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(3962): Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn(Wikimedia\Rdbms\DatabaseMysqli, string)
  #5 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn;4548)
  #6 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(635): Wikimedia\Rdbms\DBConnRef->__call(string, array)
  #7 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(503): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn;4548)
  #8 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(1120): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
  #9 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(758): MediaWiki\Storage\PageUpdater->doCreate(CommentStoreComment, User, integer)
  #10 /srv/mediawiki/php-master/includes/page/WikiPage.php(1930): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
  #11 /srv/mediawiki/php-master/includes/EditPage.php(2331): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
  #12 /srv/mediawiki/php-master/includes/EditPage.php(1612): EditPage->internalAttemptSave(array, boolean)
  #13 /srv/mediawiki/php-master/includes/EditPage.php(682): EditPage->attemptSave(array)
  #14 /srv/mediawiki/php-master/includes/actions/EditAction.php(55): EditPage->edit()
  #15 /srv/mediawiki/php-master/includes/actions/SubmitAction.php(38): EditAction->show()
  #16 /srv/mediawiki/php-master/includes/MediaWiki.php(507): SubmitAction->show()
  #17 /srv/mediawiki/php-master/includes/MediaWiki.php(302): MediaWiki->performAction(Article, Title)
  #18 /srv/mediawiki/php-master/includes/MediaWiki.php(892): MediaWiki->performRequest()
  #19 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
  #20 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
  #21 /srv/mediawiki/w/index.php(3): include(string)
  #22 {main}
Caused by: [Exception ExternalStoreException] (/srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php:140) Unable to store text to external storage
  #0 /srv/mediawiki/php-master/includes/Storage/SqlBlobStore.php(232): ExternalStoreAccess->insert(string, array)
  #1 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(932): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
  #2 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(668): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
  #3 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(603): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
  #4 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(501): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
  #5 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(3962): Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn(Wikimedia\Rdbms\DatabaseMysqli, string)
  #6 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn;4548)
  #7 /srv/mediawiki/php-master/includes/libs/rdbms/database/DBConnRef.php(635): Wikimedia\Rdbms\DBConnRef->__call(string, array)
  #8 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(503): Wikimedia\Rdbms\DBConnRef->doAtomicSection(string, Closure$MediaWiki\Revision\RevisionStore::insertRevisionOn;4548)
  #9 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(1120): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DBConnRef)
  #10 /srv/mediawiki/php-master/includes/Storage/PageUpdater.php(758): MediaWiki\Storage\PageUpdater->doCreate(CommentStoreComment, User, integer)
  #11 /srv/mediawiki/php-master/includes/page/WikiPage.php(1930): MediaWiki\Storage\PageUpdater->saveRevision(CommentStoreComment, integer)
  #12 /srv/mediawiki/php-master/includes/EditPage.php(2331): WikiPage->doEditContent(WikitextContent, CommentStoreComment, integer, boolean, User, string, array, integer)
  #13 /srv/mediawiki/php-master/includes/EditPage.php(1612): EditPage->internalAttemptSave(array, boolean)
  #14 /srv/mediawiki/php-master/includes/EditPage.php(682): EditPage->attemptSave(array)
  #15 /srv/mediawiki/php-master/includes/actions/EditAction.php(55): EditPage->edit()
  #16 /srv/mediawiki/php-master/includes/actions/SubmitAction.php(38): EditAction->show()
  #17 /srv/mediawiki/php-master/includes/MediaWiki.php(507): SubmitAction->show()
  #18 /srv/mediawiki/php-master/includes/MediaWiki.php(302): MediaWiki->performAction(Article, Title)
  #19 /srv/mediawiki/php-master/includes/MediaWiki.php(892): MediaWiki->performRequest()
  #20 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
  #21 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
  #22 /srv/mediawiki/w/index.php(3): include(string)
  #23 {main}

There are a few other similar tickets that I expect to have the same cause: T228088: BetaCluster: ExternalStoreException - Unable to store text to external storage, T227648: Not possible to move a page to a redirect you created: "Unable to store text to external storage"