Page MenuHomePhabricator
Create Task
Maniphest T207054

Some Commons pages fatal with IncompleteRevisionException: "Uninitialized field: content_address"
Closed, ResolvedPublic
Actions

Description

Error

Request ID: W8S@4QpAICEAAEQQ4SwAAABC

message
MediaWiki\Storage\IncompleteRevisionException: Uninitialized field: content_address
trace
#0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(361): MediaWiki\Storage\SlotRecord->getField(string)
#1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(501): MediaWiki\Storage\SlotRecord->getStringField(string)
#2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1435): MediaWiki\Storage\SlotRecord->getAddress()
#3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1354): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
#4 [internal function]: Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2(MediaWiki\Storage\SlotRecord)
#5 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(308): call_user_func(Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2;523, MediaWiki\Storage\SlotRecord)
#6 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionRecord.php(174): MediaWiki\Storage\SlotRecord->getContent()
#7 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(206): MediaWiki\Storage\RevisionRecord->getContent(string, integer, NULL)
#8 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(170): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#9 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(123): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#10 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#11 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(177): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;791, MediaWiki\Revision\RenderedRevision, array)
#12 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#13 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#14 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/Article.php(774): PoolCounterWork->execute()
#15 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/CategoryPage.php(69): Article->view()
#16 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/ViewAction.php(68): CategoryPage->view()
#17 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(501): ViewAction->show()
#18 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(294): MediaWiki->performAction(CategoryTreeCategoryPage, Title)
#19 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(868): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(525): MediaWiki->main()
#21 /srv/mediawiki/php-1.32.0-wmf.24/index.php(42): MediaWiki->run()

Impact

Some users are unable to view some pages on Commons.

Notes

924 exceptions recorded so far. Started less than 4 hours ago. No instances of this error in the 30 days prior.

Related Objects

Event Timeline

Krinkle created this task.Oct 15 2018, 4:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2018, 4:29 PM
Krinkle added a project: MediaWiki-ResourceLoader.Oct 15 2018, 4:35 PM

The following error while seemingly unrelated, also started around the same time, and is coming from the same server-wiki combinations. Will track under the same task for now.

exception_id: W8TAggpAICEAAEllqHIAAACT
url: /w/load.php?....

Failed to load content of JS/CSS page "{title}" in ResourceLoaderWikiModule::getContentObj

Failed to load content of JS/CSS page "MediaWiki:Common.js" in ResourceLoaderWikiModule::getContentObj
Failed to load content of JS/CSS page "MediaWiki:Gadget-UploadWizard.js" in ResourceLoaderWikiModule::getContentObj
Failed to load content of JS/CSS page "MediaWiki:Group-user.css" in ResourceLoaderWikiModule::getContentObj
Impact

Various site scripts and gadgets are likely intermittently unavailable to users.

Restricted Application added a project: Performance-Team. · View Herald TranscriptOct 15 2018, 4:35 PM
Krinkle added a comment.EditedOct 15 2018, 5:28 PM

A third error popped up at 15,000 hits per minute. Again from the same server (mw1331) and code path (ViewAction/PoolWorkArticleView/../RevisionStorage), so I'll map it to the same task for now until we know otherwise.

message
PHP Notice: Undefined variable: logArgs

#0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1369): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1660): MediaWiki\Storage\RevisionStore->emulateMainSlot_1_29(stdClass, integer, Title)
#2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1788): MediaWiki\Storage\RevisionStore->newRevisionSlots(string, stdClass, integer, Title)
#3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(2753): MediaWiki\Storage\RevisionStore->newRevisionFromRow(stdClass, integer, Title)
#4 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision.php(1305): MediaWiki\Storage\RevisionStore->getKnownCurrentRevision(Title, integer)
#5 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3607): Revision::newKnownCurrent(Wikimedia\Rdbms\DatabaseMysqli, Title)
#6 [internal function]: Parser::statelessFetchRevision(Title, Parser)
#7 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(344): call_user_func(array, Title, Parser)
#8 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3592): Closure$MediaWiki\Revision\RenderedRevision::setRevisionInternal(Title, Parser)
#9 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3684): Parser->fetchCurrentRevisionOfTitle(Title)
#10 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3620): Parser::statelessFetchTemplate(Title, Parser)
#11 /srv/mediawiki/php-1.32.0-wmf.24/extensions/Scribunto/includes/common/Base.php(168): Parser->fetchTemplateAndTitle(Title)
#12 /srv/mediawiki/php-1.32.0-wmf.24/extensions/Scribunto/includes/common/Hooks.php(110): ScribuntoEngineBase->fetchModuleFromParser(Title)
#13 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3495): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#14 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3202): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#15 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Preprocessor_Hash.php(1114): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#16 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3376): PPFrame_Hash->expand(PPNode_Hash_Tree)
#17 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Preprocessor_Hash.php(1114): Parser->braceSubstitution(array, PPFrame_Hash)
#18 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3016): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#19 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(1352): Parser->replaceVariables(string)
#20 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(476): Parser->internalParse(string)
#21 /srv/mediawiki/php-1.32.0-wmf.24/includes/StubObject.php(112): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#22 /srv/mediawiki/php-1.32.0-wmf.24/includes/StubObject.php(138): StubObject->_call(string, array)
#23 /srv/mediawiki/php-1.32.0-wmf.24/includes/content/WikitextContent.php(341): StubObject->__call(string, array)
#24 /srv/mediawiki/php-1.32.0-wmf.24/includes/content/AbstractContent.php(517): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#25 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(245): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#26 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(213): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#27 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(170): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#28 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(123): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#29 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#30 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(177): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;791, MediaWiki\Revision\RenderedRevision, array)
#31 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#32 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#33 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/Article.php(774): PoolCounterWork->execute()
#34 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/ViewAction.php(68): Article->view()
#35 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(501): ViewAction->show()

This particular sample was from /zh-hant/ on zh.wikipedia.org, limited to requests handled by mw1331.

Anomie added a subscriber: Anomie.Oct 15 2018, 5:30 PM

The logArgs one was me live-hacking trying to get a better backtrace. It should be gone now.

Anomie added a comment.Oct 15 2018, 6:09 PM

From that live-hacking, I got the following:

2018-10-15 17:28:22 [W8TONgpAICEAAFxS48oAAAAO] mw1331 commonswiki 1.32.0-wmf.24 exception ERROR: [W8TONgpAICEAAFxS48oAAAAO] /w/index.php?title=Image:(a)Original_Image_(b)Image_Generated_using_equation(1)_(c)Image_generated_using_equation(2)_(d)_Image_generated_using_equation(3)_(e)Image_generated_using_equation(4).jpg&action=render&uselang=en   MediaWiki\Storage\IncompleteRevisionException from line 334 of /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php: Uninitialized field: content_address {"exception_id":"W8TONgpAICEAAFxS48oAAAAO","exception_url":"/w/index.php?title=Image:(a)Original_Image_(b)Image_Generated_using_equation(1)_(c)Image_generated_using_equation(2)_(d)_Image_generated_using_equation(3)_(e)Image_generated_using_equation(4).jpg&action=render&uselang=en","caught_by":"mwe_handler"}
[Exception MediaWiki\Storage\IncompleteRevisionException] (/srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php:334) Uninitialized field: content_address
  #0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(361): MediaWiki\Storage\SlotRecord->getField(string)
  #1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(501): MediaWiki\Storage\SlotRecord->getStringField(string)
  #2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1449): MediaWiki\Storage\SlotRecord->getAddress()
  #3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1362): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
  #4 [internal function]: Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2(MediaWiki\Storage\SlotRecord)
  #5 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(308): call_user_func(Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2;2313, MediaWiki\Storage\SlotRecord)
  #6 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionRecord.php(174): MediaWiki\Storage\SlotRecord->getContent()
  #7 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(206): MediaWiki\Storage\RevisionRecord->getContent(string, integer, NULL)
  #8 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(170): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
  #9 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(123): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
  #10 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
  #11 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(177): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;791, MediaWiki\Revision\RenderedRevision, array)
  #12 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
  #13 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
  #14 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/Article.php(774): PoolCounterWork->execute()
  #15 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/ImagePage.php(95): Article->view()
  #16 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/RenderAction.php(44): ImagePage->render()
  #17 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(501): RenderAction->show()
  #18 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(294): MediaWiki->performAction(ImagePage, Title)
  #19 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(868): MediaWiki->performRequest()
  #20 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(525): MediaWiki->main()
  #21 /srv/mediawiki/php-1.32.0-wmf.24/index.php(42): MediaWiki->run()
  #22 /srv/mediawiki/w/index.php(3): include(string)
  #23 {main}

and

2018-10-15 17:28:22 [W8TONgpAICEAAFxS48oAAAAO] mw1331 commonswiki 1.32.0-wmf.24 RevisionStore INFO: T207054 {"row":"O:8:\"stdClass\":21:{s:6:\"rev_id\";s:9:\"210955399\";s:8:\"rev_page\";s:8:\"37281922\";s:13:\"rev_timestamp\";s:14:\"20161026191615\";s:14:\"rev_minor_edit\";s:1:\"1\";s:11:\"rev_deleted\";s:1:\"0\";s:7:\"rev_len\";s:3:\"502\";s:13:\"rev_parent_id\";s:9:\"184769585\";s:8:\"rev_sha1\";s:31:\"rehfbcgwa2jhauf8edb4q1nuba6eqz4\";s:16:\"rev_comment_text\";s:83:\"[[Help:Cat-a-lot|Cat-a-lot]]: Removing from [[Category:Uploaded with UploadWizard]]\";s:16:\"rev_comment_data\";N;s:15:\"rev_comment_cid\";N;s:8:\"rev_user\";s:6:\"505543\";s:13:\"rev_user_text\";s:9:\"INeverCry\";s:9:\"rev_actor\";N;s:14:\"page_namespace\";s:1:\"6\";s:10:\"page_title\";s:170:\"(a)Original_Image_(b)Image_Generated_using_equation(1)_(c)Image_generated_using_equation(2)_(d)_Image_generated_using_equation(3)_(e)Image_generated_using_equation(4).jpg\";s:7:\"page_id\";s:8:\"37281922\";s:11:\"page_latest\";s:9:\"210955399\";s:16:\"page_is_redirect\";s:1:\"0\";s:8:\"page_len\";s:3:\"502\";s:9:\"user_name\";s:9:\"INeverCry\";}","mainSlotRow":"O:8:\"stdClass\":9:{s:9:\"role_name\";s:4:\"main\";s:10:\"model_name\";N;s:16:\"slot_revision_id\";i:210955399;s:15:\"slot_content_id\";N;s:15:\"content_address\";N;s:11:\"slot_origin\";i:210955399;s:12:\"content_size\";i:502;s:12:\"content_sha1\";s:31:\"rehfbcgwa2jhauf8edb4q1nuba6eqz4\";s:11:\"format_name\";N;}","blobData":"NULL"}
[Exception Exception] (/srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php:1346) T207054
  #0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1660): MediaWiki\Storage\RevisionStore->emulateMainSlot_1_29(stdClass, integer, Title)
  #1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1788): MediaWiki\Storage\RevisionStore->newRevisionSlots(string, stdClass, integer, Title)
  #2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(2753): MediaWiki\Storage\RevisionStore->newRevisionFromRow(stdClass, integer, Title)
  #3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision.php(1305): MediaWiki\Storage\RevisionStore->getKnownCurrentRevision(Title, integer)
  #4 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(744): Revision::newKnownCurrent(Wikimedia\Rdbms\DatabaseMysqli, Title, integer)
  #5 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(766): WikiPage->loadLastEdit()
  #6 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(638): WikiPage->getRevision()
  #7 /srv/mediawiki/php-1.32.0-wmf.24/includes/libs/objectcache/WANObjectCache.php(1277): Closure$WikiPage::getContentModel(boolean, integer, array, NULL)
  #8 /srv/mediawiki/php-1.32.0-wmf.24/includes/libs/objectcache/WANObjectCache.php(1150): WANObjectCache->doGetWithSetCallback(string, integer, Closure$WikiPage::getContentModel;586, array)
  #9 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(648): WANObjectCache->getWithSetCallback(string, integer, Closure$WikiPage::getContentModel;586)
  #10 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(269): WikiPage->getContentModel()
  #11 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(256): WikiPage->getContentHandler()
  #12 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/Action.php(98): WikiPage->getActionOverrides()
  #13 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/Action.php(156): Action::factory(string, WikiFilePage, RequestContext)
  #14 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(155): Action::getActionName(RequestContext)
  #15 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(785): MediaWiki->getAction()
  #16 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(525): MediaWiki->main()
  #17 /srv/mediawiki/php-1.32.0-wmf.24/index.php(42): MediaWiki->run()
  #18 /srv/mediawiki/w/index.php(3): include(string)
  #19 {main}

The WTF part is #0 in that last trace, that line shouldn't be reached now that we enabled READ_NEW mode.

As far as I can tell, the scap sync-file this morning for InitialiseSettings.php somehow or other wasn't picked up properly on mw1331:

anomie@mw1331:/srv/mediawiki$ ls --full-time wmf-config/InitialiseSettings.php /tmp/mw-cache-1.32.0-wmf.24/conf-commonswiki-hhvm
-rw------- 1 www-data www-data  60599 2018-10-15 13:35:42.359630632 +0000 /tmp/mw-cache-1.32.0-wmf.24/conf-commonswiki-hhvm
-rwxr-xr-x 1 mwdeploy mwdeploy 702602 2018-10-15 13:35:42.403630552 +0000 wmf-config/InitialiseSettings.php
anomie@mw1331:/srv/mediawiki$ grep -A8 MultiContentRevisionSchemaMigrationStage wmf-config/InitialiseSettings.php 
'wgMultiContentRevisionSchemaMigrationStage' => [
	'default' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD,
	'testwiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'mediawikiwiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'igwiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'eswikivoyage' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'ltwikisource' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'commonswiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
],
anomie@mw1331:/srv/mediawiki$ sudo -u www-data grep -o 's:42:"wgMultiContentRevisionSchemaMigrationStage";i:[0-9]*' /tmp/mw-cache-1.32.0-wmf.24/conf-commonswiki-hhvm
s:42:"wgMultiContentRevisionSchemaMigrationStage";i:19

That should be "i:49" at the end. I get the same results on mw1312.

But that still doesn't explain where this error is coming from. Using READ_OLD is what it has been since rOMWCafa1353239c7: Enable MCR migration stage "write both, read old" (the default) on remaining… back in August... Ah ha. RevisionStore::getKnownCurrentRevision() is using a shared cache for the revision row data, so probably this is only happening when some other app server populated the cache (leaving out the 'rev_text_id' field) and then mw1331 or mw1312 get that cache entry and break.

Krinkle closed this task as Resolved.EditedOct 15 2018, 7:30 PM
Krinkle assigned this task to Anomie.
Krinkle added a subscriber: Addshore.

Looks its fixed itself now. Presumably by the scap command that @Addshore ran in the same minute for /wmf-config/Wikibase-production, which also had scap touch InitialiseSettings.php.

As for why it happened, we still don't know. But I suspect T181833 may be related.

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL