Page MenuHomePhabricator

[betalabs-regression] 'Unable to store text to external storage' - cannot publish any content on Flow-based pages
Closed, ResolvedPublic

Description

On Flow-based page in betalabs when publishing posted content the following error will be displayed:

[XRv73KwQBGoAAG2wBOQAAAAD] /w/api.php   ExternalStoreException from line 140 of /srv/mediawiki/php-master/includes/externalstore/ExternalStoreAccess.php: Unable to store text to external storage

#0 /srv/mediawiki/php-master/includes/externalstore/ExternalStore.php(155): ExternalStoreAccess->insert(string, array, array)
#1 /srv/mediawiki/php-master/extensions/Flow/includes/Data/Storage/RevisionStorage.php(463): ExternalStore::insertWithFallback(array, string)
#2 /srv/mediawiki/php-master/extensions/Flow/includes/Data/Storage/RevisionStorage.php(446): Flow\Data\Storage\RevisionStorage->insertExternalStore(array)
#3 /srv/mediawiki/php-master/extensions/Flow/includes/Data/Storage/RevisionStorage.php(383): Flow\Data\Storage\RevisionStorage->processExternalStore(array)
#4 /srv/mediawiki/php-master/extensions/Flow/includes/Data/ObjectManager.php(248): Flow\Data\Storage\RevisionStorage->insert(array)
#5 /srv/mediawiki/php-master/extensions/Flow/includes/Data/ObjectManager.php(173): Flow\Data\ObjectManager->insert(array, array)
#6 /srv/mediawiki/php-master/extensions/Flow/includes/Data/ObjectManager.php(146): Flow\Data\ObjectManager->multiPut(array, array)
#7 /srv/mediawiki/php-master/extensions/Flow/includes/Data/ManagerGroup.php(81): Flow\Data\ObjectManager->put(Flow\Model\PostRevision, array)
#8 /srv/mediawiki/php-master/extensions/Flow/includes/Block/Topic.php(442): Flow\Data\ManagerGroup->put(Flow\Model\PostRevision, array)
#9 /srv/mediawiki/php-master/extensions/Flow/includes/SubmissionHandler.php(160): Flow\Block\TopicBlock->commit()
#10 /srv/mediawiki/php-master/extensions/Flow/includes/WorkflowLoader.php(66): Flow\SubmissionHandler->commit(Flow\Model\Workflow, array)
#11 /srv/mediawiki/php-master/extensions/Flow/includes/Api/ApiFlowBasePost.php(35): Flow\WorkflowLoader->commit(array)
#12 /srv/mediawiki/php-master/extensions/Flow/includes/Api/ApiFlow.php(98): Flow\Api\ApiFlowBasePost->execute()
#13 /srv/mediawiki/php-master/includes/api/ApiMain.php(1583): Flow\Api\ApiFlow->execute()
#14 /srv/mediawiki/php-master/includes/api/ApiMain.php(531): ApiMain->executeAction()
#15 /srv/mediawiki/php-master/includes/api/ApiMain.php(502): ApiMain->executeActionWithErrorHandling()
#16 /srv/mediawiki/php-master/api.php(87): ApiMain->execute()
#17 /srv/mediawiki/w/api.php(3): include(string)
#18 {main}

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 3 2019, 12:59 AM
JTannerWMF edited projects, added Growth-Team (Current Sprint); removed Growth-Team.
Catrope added a subscriber: aaron.Jul 16 2019, 6:27 PM

This is caused by rMWbaafb5adb400: Make ExternalStore wrap ExternalStoreFactory and create access class (cc @aaron)

In beta (but not yet in production), Flow uses a separate external store cluster (flow_cluster1) from the rest of MediaWiki (cluster1). We want to set it up this way in production too, but haven't yet, see T107610. What this means in practice is that db-labs.php configures flow_cluster1 as an external cluster with a blobs table, and sets $wgFlowExternalStore = [ 'DB://flow_cluster1' ]; $wgDefaultExternalStore = [ 'DB://cluster1' ];. This means Flow will only write to flow_cluster1, and other ExternalStore users (like MW core revision text) will only write to cluster1.

However, rMWbaafb5adb400: Make ExternalStore wrap ExternalStoreFactory and create access class changes the ExternalStore infrastructure such that a store that's not in $wgDefaultExternalStore is not considered writable, and so all attempts to write to flow_cluster1 fail with Unable to store text to external storage DB://flow_cluster1 (read only). The fundamental problem appears to be that $wgDefaultExternalStore (and the writeBaseUrls setting in ExternalStoreFactory that it maps to) serves two functions: it's both the set of external store clusters that you're allowed to write to, and the set of external store clusters that you should try to write to. And we don't want revision text storage to write to flow_cluster1. Maybe these two concepts (writeable stores, and default stores to write to) could be separated?

As a temporary workaround, I considered adding flow_cluster1 to $wgDefaultExternalStore, but that won't work either. It'll unbreak Flow, but it'll also cause MW core revision text writes to be split 50-50 between cluster1 and flow_cluster1, which is not what I want.

I think we'll have to separate the concepts of "list of writable external store clusters" and "default list of clusters to write to", right now $wgDefaultExternalStore does both.

Change 523803 had a related patch set uploaded (by Catrope; owner: Catrope):
[mediawiki/core@master] Allow writable, non-default external store clusters to be configured

https://gerrit.wikimedia.org/r/523803

Change 523848 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] externalstore: unbreak writes to non-default storage clusters due to isReadOnly()

https://gerrit.wikimedia.org/r/523848

Change 523803 abandoned by Catrope:
Allow writable, non-default external store clusters to be configured

Reason:
Superseded by https://gerrit.wikimedia.org/r/c/mediawiki/core/ /523848 ; thanks Aaron!

https://gerrit.wikimedia.org/r/523803

Change 523848 merged by jenkins-bot:
[mediawiki/core@master] externalstore: unbreak writes to non-default storage clusters due to isReadOnly()

https://gerrit.wikimedia.org/r/523848

Etonkovidova closed this task as Resolved.Jul 19 2019, 7:54 PM

Tested in betalabs - no error is present; questions got posted and replies (with a mention) trigger correct notifications.