Page MenuHomePhabricator
Create Task
Maniphest T207054

Some Commons pages fatal with IncompleteRevisionException: "Uninitialized field: content_address"
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error

Request ID: W8S@4QpAICEAAEQQ4SwAAABC

message
MediaWiki\Storage\IncompleteRevisionException: Uninitialized field: content_address
trace
#0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(361): MediaWiki\Storage\SlotRecord->getField(string)
#1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(501): MediaWiki\Storage\SlotRecord->getStringField(string)
#2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1435): MediaWiki\Storage\SlotRecord->getAddress()
#3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1354): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
#4 [internal function]: Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2(MediaWiki\Storage\SlotRecord)
#5 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(308): call_user_func(Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2;523, MediaWiki\Storage\SlotRecord)
#6 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionRecord.php(174): MediaWiki\Storage\SlotRecord->getContent()
#7 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(206): MediaWiki\Storage\RevisionRecord->getContent(string, integer, NULL)
#8 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(170): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#9 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(123): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#10 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#11 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(177): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;791, MediaWiki\Revision\RenderedRevision, array)
#12 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#13 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#14 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/Article.php(774): PoolCounterWork->execute()
#15 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/CategoryPage.php(69): Article->view()
#16 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/ViewAction.php(68): CategoryPage->view()
#17 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(501): ViewAction->show()
#18 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(294): MediaWiki->performAction(CategoryTreeCategoryPage, Title)
#19 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(868): MediaWiki->performRequest()
#20 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(525): MediaWiki->main()
#21 /srv/mediawiki/php-1.32.0-wmf.24/index.php(42): MediaWiki->run()

Impact

Some users are unable to view some pages on Commons.

Notes

924 exceptions recorded so far. Started less than 4 hours ago. No instances of this error in the 30 days prior.

Related Objects

Event Timeline

Krinkle created this task.Oct 15 2018, 4:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 15 2018, 4:29 PM
Paladox added a subscriber: Paladox.Oct 15 2018, 4:31 PM
Krinkle added a project: MediaWiki-ResourceLoader.Oct 15 2018, 4:35 PM

The following error while seemingly unrelated, also started around the same time, and is coming from the same server-wiki combinations. Will track under the same task for now.

exception_id: W8TAggpAICEAAEllqHIAAACT
url: /w/load.php?....

Failed to load content of JS/CSS page "{title}" in ResourceLoaderWikiModule::getContentObj

Failed to load content of JS/CSS page "MediaWiki:Common.js" in ResourceLoaderWikiModule::getContentObj
Failed to load content of JS/CSS page "MediaWiki:Gadget-UploadWizard.js" in ResourceLoaderWikiModule::getContentObj
Failed to load content of JS/CSS page "MediaWiki:Group-user.css" in ResourceLoaderWikiModule::getContentObj
Impact

Various site scripts and gadgets are likely intermittently unavailable to users.

Restricted Application added a project: Performance-Team. · View Herald TranscriptOct 15 2018, 4:35 PM
Krinkle moved this task from Inbox to Confirmed Problem on the MediaWiki-ResourceLoader board.Oct 15 2018, 4:35 PM
Krinkle moved this task from Confirmed Problem to External on the MediaWiki-ResourceLoader board.
Krinkle added a comment.EditedOct 15 2018, 5:28 PM

A third error popped up at 15,000 hits per minute. Again from the same server (mw1331) and code path (ViewAction/PoolWorkArticleView/../RevisionStorage), so I'll map it to the same task for now until we know otherwise.

message
PHP Notice: Undefined variable: logArgs

#0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1369): MWExceptionHandler::handleError(integer, string, string, integer, array, array)
#1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1660): MediaWiki\Storage\RevisionStore->emulateMainSlot_1_29(stdClass, integer, Title)
#2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1788): MediaWiki\Storage\RevisionStore->newRevisionSlots(string, stdClass, integer, Title)
#3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(2753): MediaWiki\Storage\RevisionStore->newRevisionFromRow(stdClass, integer, Title)
#4 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision.php(1305): MediaWiki\Storage\RevisionStore->getKnownCurrentRevision(Title, integer)
#5 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3607): Revision::newKnownCurrent(Wikimedia\Rdbms\DatabaseMysqli, Title)
#6 [internal function]: Parser::statelessFetchRevision(Title, Parser)
#7 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(344): call_user_func(array, Title, Parser)
#8 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3592): Closure$MediaWiki\Revision\RenderedRevision::setRevisionInternal(Title, Parser)
#9 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3684): Parser->fetchCurrentRevisionOfTitle(Title)
#10 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3620): Parser::statelessFetchTemplate(Title, Parser)
#11 /srv/mediawiki/php-1.32.0-wmf.24/extensions/Scribunto/includes/common/Base.php(168): Parser->fetchTemplateAndTitle(Title)
#12 /srv/mediawiki/php-1.32.0-wmf.24/extensions/Scribunto/includes/common/Hooks.php(110): ScribuntoEngineBase->fetchModuleFromParser(Title)
#13 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3495): ScribuntoHooks::invokeHook(Parser, PPTemplateFrame_Hash, array)
#14 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3202): Parser->callParserFunction(PPTemplateFrame_Hash, string, array)
#15 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Preprocessor_Hash.php(1114): Parser->braceSubstitution(array, PPTemplateFrame_Hash)
#16 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3376): PPFrame_Hash->expand(PPNode_Hash_Tree)
#17 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Preprocessor_Hash.php(1114): Parser->braceSubstitution(array, PPFrame_Hash)
#18 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(3016): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#19 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(1352): Parser->replaceVariables(string)
#20 /srv/mediawiki/php-1.32.0-wmf.24/includes/parser/Parser.php(476): Parser->internalParse(string)
#21 /srv/mediawiki/php-1.32.0-wmf.24/includes/StubObject.php(112): Parser->parse(string, Title, ParserOptions, boolean, boolean, integer)
#22 /srv/mediawiki/php-1.32.0-wmf.24/includes/StubObject.php(138): StubObject->_call(string, array)
#23 /srv/mediawiki/php-1.32.0-wmf.24/includes/content/WikitextContent.php(341): StubObject->__call(string, array)
#24 /srv/mediawiki/php-1.32.0-wmf.24/includes/content/AbstractContent.php(517): WikitextContent->fillParserOutput(Title, integer, ParserOptions, boolean, ParserOutput)
#25 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(245): AbstractContent->getParserOutput(Title, integer, ParserOptions, boolean)
#26 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(213): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#27 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(170): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
#28 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(123): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
#29 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
#30 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(177): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;791, MediaWiki\Revision\RenderedRevision, array)
#31 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#32 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#33 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/Article.php(774): PoolCounterWork->execute()
#34 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/ViewAction.php(68): Article->view()
#35 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(501): ViewAction->show()

This particular sample was from /zh-hant/ on zh.wikipedia.org, limited to requests handled by mw1331.

Anomie added a subscriber: Anomie.Oct 15 2018, 5:30 PM

The logArgs one was me live-hacking trying to get a better backtrace. It should be gone now.

Anomie added a comment.Oct 15 2018, 6:09 PM

From that live-hacking, I got the following:

2018-10-15 17:28:22 [W8TONgpAICEAAFxS48oAAAAO] mw1331 commonswiki 1.32.0-wmf.24 exception ERROR: [W8TONgpAICEAAFxS48oAAAAO] /w/index.php?title=Image:(a)Original_Image_(b)Image_Generated_using_equation(1)_(c)Image_generated_using_equation(2)_(d)_Image_generated_using_equation(3)_(e)Image_generated_using_equation(4).jpg&action=render&uselang=en   MediaWiki\Storage\IncompleteRevisionException from line 334 of /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php: Uninitialized field: content_address {"exception_id":"W8TONgpAICEAAFxS48oAAAAO","exception_url":"/w/index.php?title=Image:(a)Original_Image_(b)Image_Generated_using_equation(1)_(c)Image_generated_using_equation(2)_(d)_Image_generated_using_equation(3)_(e)Image_generated_using_equation(4).jpg&action=render&uselang=en","caught_by":"mwe_handler"}
[Exception MediaWiki\Storage\IncompleteRevisionException] (/srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php:334) Uninitialized field: content_address
  #0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(361): MediaWiki\Storage\SlotRecord->getField(string)
  #1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(501): MediaWiki\Storage\SlotRecord->getStringField(string)
  #2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1449): MediaWiki\Storage\SlotRecord->getAddress()
  #3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1362): MediaWiki\Storage\RevisionStore->loadSlotContent(MediaWiki\Storage\SlotRecord, NULL, NULL, NULL, integer)
  #4 [internal function]: Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2(MediaWiki\Storage\SlotRecord)
  #5 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/SlotRecord.php(308): call_user_func(Closure$MediaWiki\Storage\RevisionStore::emulateMainSlot_1_29#2;2313, MediaWiki\Storage\SlotRecord)
  #6 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionRecord.php(174): MediaWiki\Storage\SlotRecord->getContent()
  #7 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(206): MediaWiki\Storage\RevisionRecord->getContent(string, integer, NULL)
  #8 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(170): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string)
  #9 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RevisionRenderer.php(123): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, array)
  #10 [internal function]: Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2(MediaWiki\Revision\RenderedRevision, array)
  #11 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision/RenderedRevision.php(177): call_user_func(Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2;791, MediaWiki\Revision\RenderedRevision, array)
  #12 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
  #13 /srv/mediawiki/php-1.32.0-wmf.24/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
  #14 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/Article.php(774): PoolCounterWork->execute()
  #15 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/ImagePage.php(95): Article->view()
  #16 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/RenderAction.php(44): ImagePage->render()
  #17 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(501): RenderAction->show()
  #18 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(294): MediaWiki->performAction(ImagePage, Title)
  #19 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(868): MediaWiki->performRequest()
  #20 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(525): MediaWiki->main()
  #21 /srv/mediawiki/php-1.32.0-wmf.24/index.php(42): MediaWiki->run()
  #22 /srv/mediawiki/w/index.php(3): include(string)
  #23 {main}

and

2018-10-15 17:28:22 [W8TONgpAICEAAFxS48oAAAAO] mw1331 commonswiki 1.32.0-wmf.24 RevisionStore INFO: T207054 {"row":"O:8:\"stdClass\":21:{s:6:\"rev_id\";s:9:\"210955399\";s:8:\"rev_page\";s:8:\"37281922\";s:13:\"rev_timestamp\";s:14:\"20161026191615\";s:14:\"rev_minor_edit\";s:1:\"1\";s:11:\"rev_deleted\";s:1:\"0\";s:7:\"rev_len\";s:3:\"502\";s:13:\"rev_parent_id\";s:9:\"184769585\";s:8:\"rev_sha1\";s:31:\"rehfbcgwa2jhauf8edb4q1nuba6eqz4\";s:16:\"rev_comment_text\";s:83:\"[[Help:Cat-a-lot|Cat-a-lot]]: Removing from [[Category:Uploaded with UploadWizard]]\";s:16:\"rev_comment_data\";N;s:15:\"rev_comment_cid\";N;s:8:\"rev_user\";s:6:\"505543\";s:13:\"rev_user_text\";s:9:\"INeverCry\";s:9:\"rev_actor\";N;s:14:\"page_namespace\";s:1:\"6\";s:10:\"page_title\";s:170:\"(a)Original_Image_(b)Image_Generated_using_equation(1)_(c)Image_generated_using_equation(2)_(d)_Image_generated_using_equation(3)_(e)Image_generated_using_equation(4).jpg\";s:7:\"page_id\";s:8:\"37281922\";s:11:\"page_latest\";s:9:\"210955399\";s:16:\"page_is_redirect\";s:1:\"0\";s:8:\"page_len\";s:3:\"502\";s:9:\"user_name\";s:9:\"INeverCry\";}","mainSlotRow":"O:8:\"stdClass\":9:{s:9:\"role_name\";s:4:\"main\";s:10:\"model_name\";N;s:16:\"slot_revision_id\";i:210955399;s:15:\"slot_content_id\";N;s:15:\"content_address\";N;s:11:\"slot_origin\";i:210955399;s:12:\"content_size\";i:502;s:12:\"content_sha1\";s:31:\"rehfbcgwa2jhauf8edb4q1nuba6eqz4\";s:11:\"format_name\";N;}","blobData":"NULL"}
[Exception Exception] (/srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php:1346) T207054
  #0 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1660): MediaWiki\Storage\RevisionStore->emulateMainSlot_1_29(stdClass, integer, Title)
  #1 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(1788): MediaWiki\Storage\RevisionStore->newRevisionSlots(string, stdClass, integer, Title)
  #2 /srv/mediawiki/php-1.32.0-wmf.24/includes/Storage/RevisionStore.php(2753): MediaWiki\Storage\RevisionStore->newRevisionFromRow(stdClass, integer, Title)
  #3 /srv/mediawiki/php-1.32.0-wmf.24/includes/Revision.php(1305): MediaWiki\Storage\RevisionStore->getKnownCurrentRevision(Title, integer)
  #4 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(744): Revision::newKnownCurrent(Wikimedia\Rdbms\DatabaseMysqli, Title, integer)
  #5 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(766): WikiPage->loadLastEdit()
  #6 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(638): WikiPage->getRevision()
  #7 /srv/mediawiki/php-1.32.0-wmf.24/includes/libs/objectcache/WANObjectCache.php(1277): Closure$WikiPage::getContentModel(boolean, integer, array, NULL)
  #8 /srv/mediawiki/php-1.32.0-wmf.24/includes/libs/objectcache/WANObjectCache.php(1150): WANObjectCache->doGetWithSetCallback(string, integer, Closure$WikiPage::getContentModel;586, array)
  #9 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(648): WANObjectCache->getWithSetCallback(string, integer, Closure$WikiPage::getContentModel;586)
  #10 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(269): WikiPage->getContentModel()
  #11 /srv/mediawiki/php-1.32.0-wmf.24/includes/page/WikiPage.php(256): WikiPage->getContentHandler()
  #12 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/Action.php(98): WikiPage->getActionOverrides()
  #13 /srv/mediawiki/php-1.32.0-wmf.24/includes/actions/Action.php(156): Action::factory(string, WikiFilePage, RequestContext)
  #14 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(155): Action::getActionName(RequestContext)
  #15 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(785): MediaWiki->getAction()
  #16 /srv/mediawiki/php-1.32.0-wmf.24/includes/MediaWiki.php(525): MediaWiki->main()
  #17 /srv/mediawiki/php-1.32.0-wmf.24/index.php(42): MediaWiki->run()
  #18 /srv/mediawiki/w/index.php(3): include(string)
  #19 {main}

The WTF part is #0 in that last trace, that line shouldn't be reached now that we enabled READ_NEW mode.

As far as I can tell, the scap sync-file this morning for InitialiseSettings.php somehow or other wasn't picked up properly on mw1331:

anomie@mw1331:/srv/mediawiki$ ls --full-time wmf-config/InitialiseSettings.php /tmp/mw-cache-1.32.0-wmf.24/conf-commonswiki-hhvm
-rw------- 1 www-data www-data  60599 2018-10-15 13:35:42.359630632 +0000 /tmp/mw-cache-1.32.0-wmf.24/conf-commonswiki-hhvm
-rwxr-xr-x 1 mwdeploy mwdeploy 702602 2018-10-15 13:35:42.403630552 +0000 wmf-config/InitialiseSettings.php
anomie@mw1331:/srv/mediawiki$ grep -A8 MultiContentRevisionSchemaMigrationStage wmf-config/InitialiseSettings.php 
'wgMultiContentRevisionSchemaMigrationStage' => [
	'default' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD,
	'testwiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'mediawikiwiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'igwiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'eswikivoyage' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'ltwikisource' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
	'commonswiki' => SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_NEW,
],
anomie@mw1331:/srv/mediawiki$ sudo -u www-data grep -o 's:42:"wgMultiContentRevisionSchemaMigrationStage";i:[0-9]*' /tmp/mw-cache-1.32.0-wmf.24/conf-commonswiki-hhvm
s:42:"wgMultiContentRevisionSchemaMigrationStage";i:19

That should be "i:49" at the end. I get the same results on mw1312.

But that still doesn't explain where this error is coming from. Using READ_OLD is what it has been since rOMWCafa1353239c7: Enable MCR migration stage "write both, read old" (the default) on remaining… back in August... Ah ha. RevisionStore::getKnownCurrentRevision() is using a shared cache for the revision row data, so probably this is only happening when some other app server populated the cache (leaving out the 'rev_text_id' field) and then mw1331 or mw1312 get that cache entry and break.

Tgr added a subscriber: Tgr.Oct 15 2018, 7:14 PM

Aside: was the file File:(a)Original Image (b)Image Generated using equation(1) (c)Image generated using equation(2) (d) Image generated using equation(3) (e)Image generated using equation(4).jpg created by some tool? Seems like the name and caption got mixed up.

Krinkle closed this task as Resolved.EditedOct 15 2018, 7:30 PM
Krinkle assigned this task to Anomie.
Krinkle added a subscriber: Addshore.

Looks its fixed itself now. Presumably by the scap command that @Addshore ran in the same minute for /wmf-config/Wikibase-production, which also had scap touch InitialiseSettings.php.

As for why it happened, we still don't know. But I suspect T181833 may be related.

CCicalese_WMF edited projects, added Platform Team Workboards (Done with CPT), Platform Engineering (MCR), Multi-Content-Revisions (Reactive); removed Platform Engineering, Multi-Content-Revisions.Oct 26 2018, 5:26 PM
CCicalese_WMF moved this task from Reactive to Done on the Multi-Content-Revisions board.Apr 1 2019, 1:16 PM
CCicalese_WMF edited projects, added Multi-Content-Revisions; removed Multi-Content-Revisions (Reactive).
CCicalese_WMF edited projects, added Platform Team Initiatives (MCR); removed Platform Engineering (MCR).Jul 30 2019, 7:46 PM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:08 PM
Aklapper removed a subscriber: Anomie.Oct 16 2020, 5:38 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL