Page MenuHomePhabricator

BetaCluster: ExternalStoreException - Unable to store text to external storage
Closed, ResolvedPublic

Description

In the context of testing something else, I tried to create https://en.wikipedia.beta.wmflabs.org/wiki/User talk:DannyS712/aoe with some random text (I think it was "test") and got the following error:

[XSzFKKwQBHcAAFUheyYAAAAO] /w/index.php?title=User_talk:DannyS712/aoe&action=submit ExternalStoreException from line 140 of /srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php: Unable to store text to external storage

Backtrace:

#0 /srv/mediawiki/php-master/includes/externalstore/ExternalStore.php(136): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilter.php(1601): ExternalStore::insertToDefault(string)
#2 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilter.php(1445): AbuseFilter::storeVarDump(array)
#3 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilter.php(1281): AbuseFilter::addLogEntries(array, array, AbuseFilterVariableHolder, string)
#4 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(138): AbuseFilter::filterAction(AbuseFilterVariableHolder, Title, string, User)
#5 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(60): AbuseFilterHooks::filterEdit(DerivativeContext, WikitextContent, string, Status, string, string)
#6 /srv/mediawiki/php-master/includes/Hooks.php(174): AbuseFilterHooks::onEditFilterMergedContent(DerivativeContext, WikitextContent, Status, string, User, boolean)
#7 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#8 /srv/mediawiki/php-master/includes/EditPage.php(1766): Hooks::run(string, array)
#9 /srv/mediawiki/php-master/includes/EditPage.php(2096): EditPage->runPostMergeFilters(WikitextContent, Status, User)
#10 /srv/mediawiki/php-master/includes/EditPage.php(1596): EditPage->internalAttemptSave(NULL, boolean)
#11 /srv/mediawiki/php-master/includes/EditPage.php(681): EditPage->attemptSave(NULL)
#12 /srv/mediawiki/php-master/includes/actions/EditAction.php(55): EditPage->edit()
#13 /srv/mediawiki/php-master/includes/actions/SubmitAction.php(38): EditAction->show()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(507): SubmitAction->show()
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(302): MediaWiki->performAction(Article, Title)
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(892): MediaWiki->performRequest()
#17 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
#18 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#19 /srv/mediawiki/w/index.php(3): include(string)
#20 {main}

I tried again with the text "test2", and it worked properly that time


A similar problem is also seen in production, which might be related:

Details

Related Objects

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Looks like a temporary error to me.

Looks like a temporary error to me.

Yes, the second attempt to save the edit is successful. The error(s) happened again today - some examples from Fatal monitor)

[XTdcGKwQBGoAAHzN5-sAAAAL] [no req]   ExternalStoreException from line 140 of /srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php: Unable to store text to external storage
[XTdX3awQBGoAAHzN5sEAAAAO] Exception caught: Unable to store text to external storage

Another error on Create account:

[XUBvxawQBHcAABhPC58AAAAG] /w/index.php?title=Special:CreateAccount&returnto=User:ET240 ExternalStoreException from line 140 of /srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php: Unable to store text to external storage

Backtrace:

#0 /srv/mediawiki/php-master/includes/externalstore/ExternalStore.php(136): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilter.php(835): ExternalStore::insertToDefault(string)
#2 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(1091): AbuseFilter::storeVarDump(array)
#3 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(173): AbuseFilterRunner->addLogEntries(array)
#4 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterPreAuthenticationProvider.php(59): AbuseFilterRunner->run()
#5 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterPreAuthenticationProvider.php(14): AbuseFilterPreAuthenticationProvider->testUser(User, User, boolean)
#6 /srv/mediawiki/php-master/includes/auth/AuthManager.php(1285): AbuseFilterPreAuthenticationProvider->testForAccountCreation(User, User, array)
#7 /srv/mediawiki/php-master/includes/auth/AuthManager.php(1147): MediaWiki\Auth\AuthManager->continueAccountCreation(array)
#8 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(358): MediaWiki\Auth\AuthManager->beginAccountCreation(User, array, string)
#9 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(482): AuthManagerSpecialPage->performAuthenticationStep(string, array)
#10 /srv/mediawiki/php-master/includes/htmlform/HTMLForm.php(689): AuthManagerSpecialPage->handleFormSubmit(array, VFormHTMLForm)
#11 /srv/mediawiki/php-master/includes/specialpage/AuthManagerSpecialPage.php(416): HTMLForm->trySubmit()
#12 /srv/mediawiki/php-master/includes/specialpage/LoginSignupSpecialPage.php(313): AuthManagerSpecialPage->trySubmit()
#13 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(571): LoginSignupSpecialPage->execute(NULL)
#14 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(296): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(892): MediaWiki->performRequest()
#17 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
#18 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#19 /srv/mediawiki/w/index.php(3): include(string)
#20 {main}
[XWXdl6wQBHcAAEPsZDEAAABL] /w/index.php?title=MediaWiki:Guidedtour-tour-firstscript.js&action=submit ExternalStoreException from line 140 of /srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php: Unable to store text to external storage

Backtrace:

#0 /srv/mediawiki/php-master/includes/externalstore/ExternalStore.php(136): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilter.php(769): ExternalStore::insertToDefault(string)
#2 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(1204): AbuseFilter::storeVarDump(array)
#3 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(180): AbuseFilterRunner->addLogEntries(array)
#4 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(219): AbuseFilterRunner->run()
#5 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(108): AbuseFilterHooks::filterEdit(DerivativeContext, JavaScriptContent, string, Status, string, string)
#6 /srv/mediawiki/php-master/includes/Hooks.php(174): AbuseFilterHooks::onEditFilterMergedContent(DerivativeContext, JavaScriptContent, Status, string, User, boolean)
#7 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#8 /srv/mediawiki/php-master/includes/EditPage.php(1768): Hooks::run(string, array)
#9 /srv/mediawiki/php-master/includes/EditPage.php(2222): EditPage->runPostMergeFilters(JavaScriptContent, Status, User)
#10 /srv/mediawiki/php-master/includes/EditPage.php(1598): EditPage->internalAttemptSave(NULL, boolean)
#11 /srv/mediawiki/php-master/includes/EditPage.php(682): EditPage->attemptSave(NULL)
#12 /srv/mediawiki/php-master/includes/actions/EditAction.php(55): EditPage->edit()
#13 /srv/mediawiki/php-master/includes/actions/SubmitAction.php(38): EditAction->show()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(507): SubmitAction->show()
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(302): MediaWiki->performAction(Article, Title)
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(892): MediaWiki->performRequest()
#17 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
#18 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#19 /srv/mediawiki/w/index.php(3): include(string)
#20 {main}

and again

[XWXeIqwQBHcAAEPsZSIAAABA] /w/index.php?title=MediaWiki:Guidedtour-tour-firstscript.js&action=submit ExternalStoreException from line 140 of /srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php: Unable to store text to external storage

Backtrace:

#0 /srv/mediawiki/php-master/includes/externalstore/ExternalStore.php(136): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilter.php(769): ExternalStore::insertToDefault(string)
#2 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(1204): AbuseFilter::storeVarDump(array)
#3 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(180): AbuseFilterRunner->addLogEntries(array)
#4 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(219): AbuseFilterRunner->run()
#5 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(108): AbuseFilterHooks::filterEdit(DerivativeContext, JavaScriptContent, string, Status, string, string)
#6 /srv/mediawiki/php-master/includes/Hooks.php(174): AbuseFilterHooks::onEditFilterMergedContent(DerivativeContext, JavaScriptContent, Status, string, User, boolean)
#7 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#8 /srv/mediawiki/php-master/includes/EditPage.php(1768): Hooks::run(string, array)
#9 /srv/mediawiki/php-master/includes/EditPage.php(2222): EditPage->runPostMergeFilters(JavaScriptContent, Status, User)
#10 /srv/mediawiki/php-master/includes/EditPage.php(1598): EditPage->internalAttemptSave(NULL, boolean)
#11 /srv/mediawiki/php-master/includes/EditPage.php(682): EditPage->attemptSave(NULL)
#12 /srv/mediawiki/php-master/includes/actions/EditAction.php(55): EditPage->edit()
#13 /srv/mediawiki/php-master/includes/actions/SubmitAction.php(38): EditAction->show()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(507): SubmitAction->show()
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(302): MediaWiki->performAction(Article, Title)
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(892): MediaWiki->performRequest()
#17 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
#18 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#19 /srv/mediawiki/w/index.php(3): include(string)
#20 {main}

and again

[XWXe3awQBHcAAEPsZf8AAABQ] /w/index.php?title=MediaWiki:Guidedtour-tour-firstscript.js&action=submit ExternalStoreException from line 140 of /srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php: Unable to store text to external storage

Backtrace:

#0 /srv/mediawiki/php-master/includes/externalstore/ExternalStore.php(136): ExternalStoreAccess->insert(string, array)
#1 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilter.php(769): ExternalStore::insertToDefault(string)
#2 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(1204): AbuseFilter::storeVarDump(array)
#3 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterRunner.php(180): AbuseFilterRunner->addLogEntries(array)
#4 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(219): AbuseFilterRunner->run()
#5 /srv/mediawiki/php-master/extensions/AbuseFilter/includes/AbuseFilterHooks.php(108): AbuseFilterHooks::filterEdit(DerivativeContext, JavaScriptContent, string, Status, string, string)
#6 /srv/mediawiki/php-master/includes/Hooks.php(174): AbuseFilterHooks::onEditFilterMergedContent(DerivativeContext, JavaScriptContent, Status, string, User, boolean)
#7 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook(string, array, array, NULL)
#8 /srv/mediawiki/php-master/includes/EditPage.php(1768): Hooks::run(string, array)
#9 /srv/mediawiki/php-master/includes/EditPage.php(2222): EditPage->runPostMergeFilters(JavaScriptContent, Status, User)
#10 /srv/mediawiki/php-master/includes/EditPage.php(1598): EditPage->internalAttemptSave(NULL, boolean)
#11 /srv/mediawiki/php-master/includes/EditPage.php(682): EditPage->attemptSave(NULL)
#12 /srv/mediawiki/php-master/includes/actions/EditAction.php(55): EditPage->edit()
#13 /srv/mediawiki/php-master/includes/actions/SubmitAction.php(38): EditAction->show()
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(507): SubmitAction->show()
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(302): MediaWiki->performAction(Article, Title)
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(892): MediaWiki->performRequest()
#17 /srv/mediawiki/php-master/includes/MediaWiki.php(523): MediaWiki->main()
#18 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#19 /srv/mediawiki/w/index.php(3): include(string)
#20 {main}

Just got [XczI46wQBGoAAGq15@gAAAAD] Exception caught: Unable to store text to external storage on the beta cluster when trying to edit. Tried again, and it worked.

A few seconds later tried to move and delete target, and got another error:

[XczJJ6wQBHcAACLx-5oAAAAH] /w/index.php?title=Special:MovePage&action=submit MediaWiki\Storage\BlobAccessException from line 261 of /srv/mediawiki/php-master/includes/Storage/SqlBlobStore.php: Unable to store text to external storage

Backtrace:

#0 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(926): MediaWiki\Storage\SqlBlobStore->storeBlob(string, array)
#1 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(665): MediaWiki\Revision\RevisionStore->storeContentBlob(MediaWiki\Revision\SlotRecord, Title, array)
#2 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(600): MediaWiki\Revision\RevisionStore->insertSlotOn(Wikimedia\Rdbms\DatabaseMysqli, integer, MediaWiki\Revision\SlotRecord, Title, array)
#3 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(497): MediaWiki\Revision\RevisionStore->insertRevisionInternal(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli, User, CommentStoreComment, Title, integer, integer)
#4 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(3990): MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(Wikimedia\Rdbms\DatabaseMysqli, string)
#5 /srv/mediawiki/php-master/includes/Revision/RevisionStore.php(499): Wikimedia\Rdbms\Database->doAtomicSection(string, Closure)
#6 /srv/mediawiki/php-master/includes/Revision.php(963): MediaWiki\Revision\RevisionStore->insertRevisionOn(MediaWiki\Revision\MutableRevisionRecord, Wikimedia\Rdbms\DatabaseMysqli)
#7 /srv/mediawiki/php-master/includes/MovePage.php(878): Revision->insertOn(Wikimedia\Rdbms\DatabaseMysqli)
#8 /srv/mediawiki/php-master/includes/MovePage.php(518): MovePage->moveToInternal(User, Title, string, boolean, array)
#9 /srv/mediawiki/php-master/includes/MovePage.php(361): MovePage->moveUnsafe(User, string, boolean, array)
#10 /srv/mediawiki/php-master/includes/specials/SpecialMovepage.php(620): MovePage->moveIfAllowed(User, string, boolean)
#11 /srv/mediawiki/php-master/includes/specials/SpecialMovepage.php(130): MovePageForm->doSubmit()
#12 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(575): MovePageForm->execute(NULL)
#13 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(607): SpecialPage->run(NULL)
#14 /srv/mediawiki/php-master/includes/MediaWiki.php(298): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#15 /srv/mediawiki/php-master/includes/MediaWiki.php(967): MediaWiki->performRequest()
#16 /srv/mediawiki/php-master/includes/MediaWiki.php(530): MediaWiki->main()
#17 /srv/mediawiki/php-master/index.php(46): MediaWiki->run()
#18 /srv/mediawiki/w/index.php(3): require(string)
#19 {main}
Anomie subscribed.

Unfortunately due to T233134: logstash-beta.wmflabs.org does not receive any mediawiki events I can't seem to find any logs since September that might point to an underlying issue. There are still a few logs from before that error began, though, that suggest the database is (intermittently) read-only.

2019-08-29 11:49:39 [XWe70qwQBGoAAFptYSYAAADP] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-08-29 18:05:13 [XWgT2awQBHcAADxvoWcAAABJ] deployment-mediawiki-07 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-08-30 05:05:38 [XWiuoawQBGoAABnRQVEAAAAU] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-08-30 14:17:21 [XWkv8KwQBGoAAHHCz2MAAAAV] deployment-mediawiki-09 zhwiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-08-30 14:17:36 [XWkv-6wQBGoAAHHCz2wAAAAX] deployment-mediawiki-09 zhwiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-08-30 18:04:44 [XWllPKwQBGoAADEr17sAAAAW] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-08-30 18:05:03 [XWllTqwQBGoAADEr1@AAAAAT] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-08-31 18:05:27 [XWq25qwQBGoAAEd7HZsAAAAC] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-01 13:52:59 [XWvNO6wQBGoAAAHMo4oAAAAG] deployment-mediawiki-09 zhwiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-01 13:53:07 [XWvNQ6wQBGoAAAHMo5IAAAAV] deployment-mediawiki-09 zhwiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-01 13:53:44 [XWvNZ6wQBHcAACs5iJQAAABL] deployment-mediawiki-07 zhwiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-01 13:53:49 [XWvNbawQBHcAACs5iLYAAABX] deployment-mediawiki-07 zhwiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-01 18:04:35 [XWwIM6wQBGoAAGgU1KoAAAAL] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 08:44:00 [XWzWUKwQBHcAAH0j@8IAAABI] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 09:15:33 [XWzdtKwQBHcAAAi-@WsAAAAE] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 09:22:13 [XWzfRawQBHcAAAi-@r4AAAAA] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 09:22:19 [XWzfS6wQBHcAAAi-@tkAAAAG] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 09:40:03 [XWzjcqwQBGoAACBKs5gAAAAV] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 09:44:04 [XWzkZKwQBHcAABg@PLUAAAAK] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 09:51:32 [XWzmJKwQBHcAABg@QPUAAAAR] deployment-mediawiki-07 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 10:00:02 [XWzoIqwQBGoAAClNZOsAAABN] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 10:06:30 [XWzppqwQBGoAAClNZV8AAABP] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-02 10:09:25 [XWzqVKwQBGoAAClNZcoAAABB] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-03 04:14:29 [XW3opawQBHcAAA1vQmUAAAAQ] deployment-mediawiki-07 metawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-03 04:15:43 [XW3o7qwQBHcAAA1vQsIAAAAX] deployment-mediawiki-07 metawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-03 05:01:47 [XW3zu6wQBGoAAClNslkAAABN] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-03 18:07:08 [XW6rzKwQBGoAAFwf9-YAAAAC] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-03 18:09:32 [XW6sXKwQBGoAAFwf@aAAAAAI] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-03 18:09:47 [XW6sa6wQBGoAAFwf@coAAAAP] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 09:27:52 [XW@DmKwQBGoAAAFzkX0AAABI] deployment-mediawiki-09 metawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 09:28:34 [XW@DwawQBHcAAH8iOS4AAABA] deployment-mediawiki-07 metawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 09:30:53 [XW@ETawQBGoAAAFzkgwAAABW] deployment-mediawiki-09 metawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 09:30:58 [XW@EUawQBGoAAAFzkg8AAABW] deployment-mediawiki-09 metawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 10:05:17 [XW@MXawQBHcAAH8iQzsAAABK] deployment-mediawiki-07 jawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 10:05:23 [XW@MY6wQBHcAAH8iQ0sAAABU] deployment-mediawiki-07 jawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 10:05:28 [XW@MaKwQBHcAAH8iQ1IAAABY] deployment-mediawiki-07 jawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 18:04:50 [XW-8wqwQBHcAAGlRBDkAAAAR] deployment-mediawiki-07 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-04 18:05:31 [XW-866wQBGoAAC-4LIYAAAAF] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-05 13:37:56 [XXEPs6wQBGoAAC5M-0IAAAAF] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-05 13:43:15 [XXEQ8qwQBGoAAC5M-6QAAAAX] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-05 13:43:27 [XXEQ-6wQBGoAAC5M-7EAAAAE] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-05 13:43:55 [XXERG6wQBGoAAC5M-8sAAAAF] deployment-mediawiki-09 commonswiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-06 18:07:22 [XXKgWqwQBGoAAFTKeWcAAADY] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-07 11:48:42 [XXOZGawQBGoAAAoxCdwAAABM] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-07 11:49:13 [XXOZOawQBGoAAAoxCgMAAABA] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-07 18:06:52 [XXPxvKwQBGoAACa5Cn8AAAAX] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"} 
2019-09-07 18:07:14 [XXPx0qwQBGoAACa5Cq8AAAAB] deployment-mediawiki-09 wikidatawiki 1.34.0-alpha ExternalStore ERROR: Unable to store text to external storage DB://cluster1 (read only) {"store_path":"DB://cluster1","failure":"read only"}

I'm going to guess this is a problem more related to Beta-Cluster-Infrastructure than to MediaWiki itself.

Just got [XhVJzawQBGoAAFZofegAAAAO] Exception caught: Unable to store text to external storage again

@Krinkle look at the history of https://commons.wikimedia.beta.wmflabs.org/w/index.php?title=User_talk:AJ&action=history starting with https://commons.wikimedia.beta.wmflabs.org/w/index.php?title=User_talk:AJ&oldid=217967.

I added the first 20 sections reasonably quickly and only number 17 failed. Sections 21-40 were added more quickly (still done by hand but just clicking faster) and #24, #25 and #29 failed.

Starting 14:58 I clicked as fast as I could, starting at https://commons.wikimedia.beta.wmflabs.org/w/index.php?title=User_talk:AJ&oldid=218049. #11, #39 and #43 failed.

So I did 80 edits using the mobile interface, hoping that would be faster, but I can't click much faster. #11 failed, #12 failed, 14 failed due to error conflict (?), #41 failed and #49 failed.

Unless stated otherwise, "failed" means "Unable to store text to external storage".

My gut says T247207, T247617 and "Upload error Unable to modify the file "6f2dbfewe.jpg" because the file repository "local" is in read-only mode.
The system administrator who locked it offered this explanation: "".
"could also be related.

From looking at ExternalStoreAccess::insert(), it seems like this may be caused by the database being in readonly state. The error modes and reporting in that method should be cleaned up. At the very least, the ExternalStoreException thrown at the very end of the method should include $msg.

Change 584622 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] ExternalStore: report cause of non-exception failure

https://gerrit.wikimedia.org/r/584622

Change 584622 merged by jenkins-bot:
[mediawiki/core@master] ExternalStore: report cause of non-exception failure

https://gerrit.wikimedia.org/r/584622

daniel claimed this task.