Page MenuHomePhabricator

includes/Revision/RevisionStore.php: Main slot of revision (number) not found in database!
Open, MediumPublic

Description

Noticed in mediawiki-errors for both wmf.8 an wmf.9

[XBuw4QpAICMAALTXkHkAAABG] /w/index.php?title=Speci%C3%A1lis:Lap_%C3%A1tnevez%C3%A9se&action=submit   MediaWiki\Revision\RevisionAccessException from line 1643 of /srv/mediawiki/php-1.33.0-wmf.8/includes/Revision/RevisionStore.php: Main slot of revision 20798212 not found in database!
[XBuxngpAIDgAAJFITF8AAADO] /w/api.php?rvprop=userid%7Cuser%7Cids%7Ccontent%7Csize%7Ctimestamp%7Ccontentmodel%7Ccomment&revids=815943409&prop=revisions&format=json&rvslots=main&action=query   MediaWiki\Revision\RevisionAccessException from line 1643 of /srv/mediawiki/php-1.33.0-wmf.9/includes/Revision/RevisionStore.php: Main slot of revision 815943409 not found in database!

Impact

Through the API, querying information about some pages results in an internal_api_error response.

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

this just happened on lij.wikisource, trying to import https://wikisource.org/wiki/Index:Bacigal%C3%B4.Prose.rim%C3%A6.1891.pdf (by file). If that helps with anything?

Error occurs with Special:Import, too, for import both from wiki and file.

Error occurs with Special:Import, too, for import both from wiki and file.

Can you provide back stack trace or an error ID? Does it happen every time, or sporadically? Does it happen for any page, or only specific pages?

I tried to import https://de.wikipedia.org/wiki/R%C3%B6merstra%C3%9Fe_Neckar-Alb-Aare (w:de:Römerstraße Neckar-Alb-Aare) from German Wikipedia to German Wikivoyage. I got only the error message "Import fehlgeschlagen (= import failed): Main slot of revision not found in database. See T212428." but nothing else. This failure happened all the time with this page, but we did not tested it with other pages.

I found some log entries that look like they are related to this issue. Here is the relevant info:

_id: AXN7hz9h3_NNwgAUthTp
mwversion: 1.36.0-wmf.1
reqId: 30d52c41-ee69-4991-a1ba-40ceed794137
message: MediaWiki\Revision\RevisionStore::constructSlotRecords falling back to READ_LATEST.
trace:<ul>
<li>RevisionStore.php line 1274 calls wfBacktrace()</li>
<li>RevisionStore.php line 1208 calls MediaWiki\Revision\RevisionStore->constructSlotRecords()</li>
<li>RevisionStore.php line 1330 calls MediaWiki\Revision\RevisionStore->loadSlotRecords()</li>
<li>- line - calls MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()</li>
<li>RevisionSlots.php line 175 calls call_user_func()</li>
<li>RevisionSlots.php line 134 calls MediaWiki\Revision\RevisionSlots->getSlots()</li>
<li>RevisionRecord.php line 209 calls MediaWiki\Revision\RevisionSlots->hasSlot()</li>
<li>ImportableOldRevisionImporter.php line 165 calls MediaWiki\Revision\RevisionRecord->hasSlot()</li>
<li>WikiRevision.php line 669 calls ImportableOldRevisionImporter->import()</li>
<li>WikiImporter.php line 371 calls WikiRevision->importOldRevision()</li>
<li>WikiImporter.php line 507 calls WikiImporter->importRevision()</li>
<li>WikiImporter.php line 999 calls WikiImporter->revisionCallback()</li>
<li>WikiImporter.php line 864 calls WikiImporter->processRevision()</li>
<li>WikiImporter.php line 802 calls WikiImporter->handleRevision()</li>
<li>WikiImporter.php line 612 calls WikiImporter->handlePage()</li>
<li>SpecialImport.php line 235 calls WikiImporter->doImport()</li>
<li>SpecialImport.php line 118 calls SpecialImport->doImport()</li>
<li>SpecialPage.php line 600 calls SpecialImport->execute()</li>
<li>SpecialPageFactory.php line 635 calls SpecialPage->run()</li>
<li>MediaWiki.php line 307 calls MediaWiki\SpecialPage\SpecialPageFactory->executePath()</li>
<li>MediaWiki.php line 940 calls MediaWiki->performRequest()</li>
<li>MediaWiki.php line 543 calls MediaWiki->main()</li>
<li>index.php line 53 calls MediaWiki->run()</li>
<li>index.php line 46 calls wfIndexMain()</li>
<li>index.php line 3 calls require()</li>
</ul>

From another log entry:
_id: AXN7hzzDNoG2jwpwShrO
message: MediaWiki\Revision\RevisionStore::constructSlotRecords: Main slot of revision 1316093 not found in database. See T212428.
mwversion: 1.36.0-wmf.1
reqId: 30d52c41-ee69-4991-a1ba-40ceed794137
revid: 1316093

From the database table:
$ select * from slots join slot_roles on role_id = slot_role_id join content on content_id = slot_content_id where slot_revision_id = 1316093;

slot_revision_idslot_role_idslot_content_idslot_originrole_idrole_namecontent_idcontent_sizecontent_sha1content_modelcontent_address
13160931128652813160931main128652831979zx2cqynjxesuh5835hhv6637x7kca81tt:1295133

So the main slot is there.

I assume the revision was just created right before we again try to read it. So I can see how it would be missing from a replica, but
the fact that it's not found even after falling back to the master database is a mystery to me. Maybe it's due to transaction behavior? Perhaps a DBA has an idea...

Note that the importer code was changed in a way that is likely to aggravate this issue: previously, we were looking on at the data in the revision table, now we are also looking at data in the slots table (T220525). This however makes this more of a mystery. If the problem was that we are seeing old data in the database query, why are we getting an up to date view of the revision table, but a stale view of the slots table (or the content table, which is joined in)?

The only thing I could think of is that _all_ slaves were lagging a bit (which would be really weird) and due to semi-sync replication the master didn't commit the transaction until at least one replica reported the transaction as committed. But I think the odds of that happening would be very low.

Looking at other log messages from the same request, it seems that replication lag was high at the time:

_id: AXN7hz7_MQ_08tQavmlc
message: Wikimedia\Rdbms\LoadBalancer::waitForMasterPos: timed out waiting on 10.64.16.7 pos 0-171966669-4075108480,171966669-171966669-4196523483,171974792-171974792-378345284,171978787-171978787-1768880104,180363367-180363367-134174373 [3.2E-5s]

Could be related.

I just realized that the fallback to master happens in constructSlotRecords(), and only applies to loading slot blobs. If we already failed to fetch the slot rows, this will not help. We'd need the same kind of logic in loadSlotRecords()! So that explains why the fallback doesn't help.

Looking at other log messages from the same request, it seems that replication lag was high at the time:

_id: AXN7hz7_MQ_08tQavmlc
message: Wikimedia\Rdbms\LoadBalancer::waitForMasterPos: timed out waiting on 10.64.16.7 pos 0-171966669-4075108480,171966669-171966669-4196523483,171974792-171974792-378345284,171978787-171978787-1768880104,180363367-180363367-134174373 [3.2E-5s]

Could be related.

That could be the false positive we get logged on MW quite often
I doubt all the slaves were lagging but happy to double check! do you have a timestamp so I can check all of them? Which wiki was this?

Change 615753 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/core@master] RevisionSTore: fall back to master if no revision rows were found

https://gerrit.wikimedia.org/r/615753

That could be the false positive we get logged on MW quite often
I doubt all the slaves were lagging but happy to double check! do you have a timestamp so I can check all of them? Which wiki was this?

Timestamp: 2020-07-23T11:55:26
Wiki: dewikivoyage
_id: AXN7hz7_MQ_08tQavmlc
reqId: 30d52c41-ee69-4991-a1ba-40ceed794137

Note that T258666 looks like it could be an expression of this problem that was exacerbated by MediaWiki 1.36/wmf.1 (which is, I think, scheduled to hit the Wikipedias today). And as I commented there: I've never before seen this problem, and I've now seen 3 reports from 3 different projects in the last 24 hours. If so, this is not just a log spamming problem with the occasional user-visible weirdness any more.

One way this can happen is the issue I just fixed for T258666:

Revision rows are selected from the master DB, and then RevisionStore::newRevisionFromRow is used to create a RevisionObject.
In that situation, newRevisionFromRow() needs to be passed the READ_LATEST flag, so associated information in the slots and content tables is also read from the master database.
Otherwise, it will default to reading from a replica, which may not yet have the revision we found on the master db.

I didn't see any other instances of this in core, but we should watch out for this issue when analyzing stack traces related to this.

A new case just happened on lij.wikisource at https://lij.wikisource.org/w/index.php?title=Laude_zeneize/19&action=history - Special:Import showed the error, but the page was still imported (though without a log entry).

Edit: Or rather, several cases - I was importing a file with pages numbered /1 to /25. It happened for every page after 18.

That could be the false positive we get logged on MW quite often
I doubt all the slaves were lagging but happy to double check! do you have a timestamp so I can check all of them? Which wiki was this?

Timestamp: 2020-07-23T11:55:26
Wiki: dewikivoyage
_id: AXN7hz7_MQ_08tQavmlc
reqId: 30d52c41-ee69-4991-a1ba-40ceed794137

No lag reported around those times

Change 615753 merged by jenkins-bot:
[mediawiki/core@master] RevisionStore: fall back to master if no revision rows were found.

https://gerrit.wikimedia.org/r/615753

Change 616608 had a related patch set uploaded (by Cicalese; owner: Daniel Kinzler):
[mediawiki/core@REL1_35] RevisionStore: fall back to master if no revision rows were found.

https://gerrit.wikimedia.org/r/616608

Change 616608 merged by jenkins-bot:
[mediawiki/core@REL1_35] RevisionStore: fall back to master if no revision rows were found.

https://gerrit.wikimedia.org/r/616608

BPirkle added a subscriber: BPirkle.

Removing myself as assignee as I am not actively working on this. I'm happy to help, but others have been more active on this task.

daniel raised the priority of this task from Medium to High.Aug 26 2020, 10:04 AM

So this is still ongoing, but not very high frequency. I'm seeing about 100 incidents over the last week:
https://logstash.wikimedia.org/goto/89f0103da0f86a76470a3003a684f2d0

Bumping to "high", since we should at least understand when and why this is happening.

Here's a logstash link for all RevisionStore errors: https://logstash.wikimedia.org/goto/57ff2c50e5c662f862b897e08720f3ee

The instances I find now all come from JobRunner, and go via Scribunto:

 	RevisionStore.php line 1322 calls wfBacktrace()
RevisionStore.php line 1254 calls MediaWiki\Revision\RevisionStore->constructSlotRecords()
RevisionStore.php line 1250 calls MediaWiki\Revision\RevisionStore->loadSlotRecords()
RevisionStore.php line 1365 calls MediaWiki\Revision\RevisionStore->loadSlotRecords()
- line - calls MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
RevisionSlots.php line 175 calls call_user_func()
RevisionSlots.php line 117 calls MediaWiki\Revision\RevisionSlots->getSlots()
RevisionRecord.php line 192 calls MediaWiki\Revision\RevisionSlots->getSlot()
RevisionRecord.php line 175 calls MediaWiki\Revision\RevisionRecord->getSlot()
Parser.php line 3641 calls MediaWiki\Revision\RevisionRecord->getContent()
Parser.php line 3526 calls Parser->statelessFetchTemplate()
ScribuntoEngineBase.php line 167 calls Parser->fetchTemplateAndTitle()
LuaEngine.php line 566 calls ScribuntoEngineBase->fetchModuleFromParser()
LuaSandboxCallback.php line 26 calls Scribunto_LuaEngine->loadPackage()
- line - calls Scribunto_LuaSandboxCallback->__call()
LuaSandboxInterpreter.php line 113 calls LuaSandboxFunction->call()
LuaEngine.php line 266 calls Scribunto_LuaSandboxInterpreter->callFunction()
LuaModule.php line 55 calls Scribunto_LuaEngine->executeModule()
Hooks.php line 128 calls Scribunto_LuaModule->invoke()
Parser.php line 3335 calls ScribuntoHooks::invokeHook()
Parser.php line 3042 calls Parser->callParserFunction()
PPFrame_Hash.php line 253 calls Parser->braceSubstitution()
Parser.php line 3220 calls PPFrame_Hash->expand()
PPFrame_Hash.php line 253 calls Parser->braceSubstitution()
Parser.php line 2882 calls PPFrame_Hash->expand()
Parser.php line 1555 calls Parser->replaceVariables()
Parser.php line 650 calls Parser->internalParse()
WikitextContent.php line 374 calls Parser->parse()
AbstractContent.php line 590 calls WikitextContent->fillParserOutput()
RenderedRevision.php line 266 calls AbstractContent->getParserOutput()
RenderedRevision.php line 235 calls MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
RevisionRenderer.php line 215 calls MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
RevisionRenderer.php line 152 calls MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
- line - calls MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
RenderedRevision.php line 197 calls call_user_func()
RefreshLinksJob.php line 259 calls MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
RefreshLinksJob.php line 179 calls RefreshLinksJob->getParserOutput()
RefreshLinksJob.php line 126 calls RefreshLinksJob->runForTitle()
JobExecutor.php line 79 calls RefreshLinksJob->run()
RunSingleJob.php line 76 calls MediaWiki\Extension\EventBus\JobExecutor->execute()

The solution is probably to introduce a fallback to master in constructSlotRecords(), in the same way we have done it in loadSlotRecords().

Observation:
Some of the revisions for which this problem is being reported are missing from the labs replica. E.g.:
reqId: 61a4914d-39ef-4fcf-a169-19a4435c2a09
revid: 977912002
wiki: enwiki

wikiadmin@10.192.48.87(enwiki)> select * from revision where rev_id = 977912002;
Empty set (0.00 sec)

MariaDB [enwiki_p]> select * from slots join content on slot_content_id = content_id where slot_revision_id = 977912002 \G
Empty set (0.01 sec)

The revision is present master, and is not deleted:

> select * from revision where rev_id = 977912002 \G
*************************** 1. row ***************************
        rev_id: 977912002
      rev_page: 48255424
rev_comment_id: 0
     rev_actor: 0
 rev_timestamp: 20200911184757
rev_minor_edit: 0
   rev_deleted: 0
       rev_len: 1545
 rev_parent_id: 977911774
      rev_sha1: qqugtkzso9kanir9k8py66op3x4h1tv
1 row in set (0.00 sec)

wikiadmin@10.192.48.87(enwiki)> select * from slots join content on slot_content_id = content_id where slot_revision_id = 977912002 \G
*************************** 1. row ***************************
slot_revision_id: 977912002
    slot_role_id: 1
 slot_content_id: 968890405
     slot_origin: 977912002
      content_id: 968890405
    content_size: 1545
    content_sha1: qqugtkzso9kanir9k8py66op3x4h1tv
   content_model: 1
 content_address: tt:990025728
1 row in set (0.00 sec)

Why would this revision be filtered out?

Or is replication somehow flaky? Are some rows (revision, slots, or content) missing from some replicas? That would epxlain the errors.

As we're referencing this task in error messages... The description should really continue something more... substantial on how people should be "fixing" it, rather than having to dig through the comments

Just like to note that I encountered this error while upgrading from 1.29.1 to 1.35.0, and I was able to work around it by running the maintenance script populateContentTables.php, which failed with a duplicate entry for key "PRIMARY" error but still allowed the update.php script to continue. It seems like the updater tried to work on the "slots" table before it was even populated, hence the slot not found error.

Just like to note that I encountered this error while upgrading from 1.29.1 to 1.35.0, and I was able to work around it by running the maintenance script populateContentTables.php, which failed with a duplicate entry for key "PRIMARY" error but still allowed the update.php script to continue. It seems like the updater tried to work on the "slots" table before it was even populated, hence the slot not found error.

That problem seems to be specific for updating from 1.29 to 1.35, and would have a different root casue. While the symptom is the same, it would probably be useful to have a separate task for it. Can you file it separately? Thank you!

As we're referencing this task in error messages... The description should really continue something more... substantial on how people should be "fixing" it, rather than having to dig through the comments

Do you know how to fix it? I don't even know what causes this. I can't reproduce it, and looking at the code, I can't think of any condition that would trigger this.

So since March people are unable to update from old MW versions?

I just get this error after update from 1.30.0 to 1.35.0.

MediaWiki\Revision\RevisionAccessException from line 1296 of /usr/share/mediawiki/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
#0 /usr/share/mediawiki/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords('811', Object(Wikimedia\Rdbms\ResultWrapper), 1, Object(Title))
#1 /usr/share/mediawiki/includes/Revision/RevisionStore.php(1220): MediaWiki\Revision\RevisionStore->loadSlotRecords('811', 1, Object(Title))
#2 /usr/share/mediawiki/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords('811', 0, Object(Title))

There is no even an revision number.

I got a similar error halfway running update.php when upgrading from 1.31 to 1.35. It seems that the slots table was created in 1.31.10 (minor version) and that the populateContentTables.php script normally should have been executed automatically by the update.php script. Luckily by running the populateContentTables.php manually I could run update.php a 2nd time to execute the rest of the upgrade (without requiring to restore the database to the initial backup). Here is the log tail from the 1st run of update.php

Modifying table site_stats ...done.^M
Populating ar_rev_id.^M
Populating ar_rev_id...^M
MediaWiki\Revision\RevisionAccessException from line 1296 of /data/html/mediawiki-1.35.0/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.^M

I got a similar error halfway running update.php when upgrading from 1.31 to 1.35. It seems that the slots table was created in 1.31.10 (minor version) and that the populateContentTables.php script normally should have been executed automatically by the update.php script. Luckily by running the populateContentTables.php manually I could run update.php a 2nd time to execute the rest of the upgrade (without requiring to restore the database to the initial backup). Here is the log tail from the 1st run of update.php

Modifying table site_stats ...done.^M
Populating ar_rev_id.^M
Populating ar_rev_id...^M
MediaWiki\Revision\RevisionAccessException from line 1296 of /data/html/mediawiki-1.35.0/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.^M

This worked for me - thanks

I hate to +1, but I hit this upgrading from 1.27 to 1.35 and Geertivp's fix worked for me as well. Trace:

Populating ar_rev_id.
Populating ar_rev_id...
MediaWiki\Revision\RevisionAccessException from line 1296 of /var/www/html/w/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
#0 /var/www/html/w/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords()
#1 /var/www/html/w/includes/Revision/RevisionStore.php(1220): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#2 /var/www/html/w/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/w/includes/Revision/RevisionSlots.php(175): call_user_func()
#5 /var/www/html/w/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/w/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#7 /var/www/html/w/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\RevisionRecord->getSlot()
#8 /var/www/html/w/includes/cache/MessageCache.php(1185): MediaWiki\Revision\RevisionRecord->getContent()
#9 /var/www/html/w/includes/libs/objectcache/wancache/WANObjectCache.php(1529): MessageCache->{closure}()
#10 /var/www/html/w/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate()
#11 /var/www/html/w/includes/cache/MessageCache.php(1205): WANObjectCache->getWithSetCallback()
#12 /var/www/html/w/includes/libs/objectcache/BagOStuff.php(149): MessageCache->{closure}()
#13 /var/www/html/w/includes/cache/MessageCache.php(1207): BagOStuff->getWithSetCallback()
#14 /var/www/html/w/includes/cache/MessageCache.php(1109): MessageCache->loadCachedMessagePageEntry()
#15 /var/www/html/w/includes/cache/MessageCache.php(1017): MessageCache->getMsgFromNamespace()
#16 /var/www/html/w/includes/cache/MessageCache.php(988): MessageCache->getMessageForLang()
#17 /var/www/html/w/includes/cache/MessageCache.php(930): MessageCache->getMessageFromFallbackChain()
#18 /var/www/html/w/includes/language/Message.php(1304): MessageCache->get()
#19 /var/www/html/w/includes/language/Message.php(862): Message->fetchMessage()
#20 /var/www/html/w/includes/language/Message.php(954): Message->toString()
#21 /var/www/html/w/includes/Title.php(661): Message->text()
#22 /var/www/html/w/maintenance/populateArchiveRevId.php(213): Title::newMainPage()
#23 /var/www/html/w/maintenance/populateArchiveRevId.php(118): PopulateArchiveRevId::makeDummyRevisionRow()
#24 /var/www/html/w/maintenance/populateArchiveRevId.php(63): PopulateArchiveRevId::checkMysqlAutoIncrementBug()
#25 /var/www/html/w/maintenance/includes/LoggedUpdateMaintenance.php(45): PopulateArchiveRevId->doDBUpdates()
#26 /var/www/html/w/includes/installer/DatabaseUpdater.php(1419): LoggedUpdateMaintenance->execute()
#27 /var/www/html/w/includes/installer/DatabaseUpdater.php(554): DatabaseUpdater->populateArchiveRevId()
#28 /var/www/html/w/includes/installer/DatabaseUpdater.php(517): DatabaseUpdater->runUpdates()
#29 /var/www/html/w/maintenance/update.php(181): DatabaseUpdater->doUpdates()
#30 /var/www/html/w/maintenance/doMaintenance.php(107): UpdateMediaWiki->execute()
#31 /var/www/html/w/maintenance/update.php(253): require_once('/var/www/html/w...')
#32 {main}

Not sure if this indicates a problem with the order of operations in update.php.

Hello. I'm getting this error after a trial upgrade from 1.34.1 to 1.35.1, not during upgrade but when accessing some but not all pages. Previously I patched us to 1.33.2 from 1.33.1. We were on an earlier version before that, so various migration flags were set along the way. Am running in a docker container which is on PHP 7.3.25. A separate container is running MySQL 5.7.32.

I've tried running populateContentTables.php which outputs this.

Any suggestions? Or debugging that I could try?

Thanks in advance.

Populating revision...
No need to populate, revision.rev_text_id field does not exist
Populating archive...
No need to populate, archive.ar_text_id field does not exist
Done. Processed 0 rows in 0.010226964950562 seconds
Original exception: [e7f342cbae2e2797607e2a9b] /Contact_Details MediaWiki\Revision\RevisionAccessException from line 1296 of /var/www/html/includes/Revision/RevisionStore.php: Main slot of revision not found in database. See T212428.
Backtrace:
#0 /var/www/html/includes/Revision/RevisionStore.php(1224): MediaWiki\Revision\RevisionStore->constructSlotRecords(string, Wikimedia\Rdbms\ResultWrapper, integer, Title)
#1 /var/www/html/includes/Revision/RevisionStore.php(1220): MediaWiki\Revision\RevisionStore->loadSlotRecords(string, integer, Title)
#2 /var/www/html/includes/Revision/RevisionStore.php(1335): MediaWiki\Revision\RevisionStore->loadSlotRecords(string, integer, Title)
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#4 /var/www/html/includes/Revision/RevisionSlots.php(175): call_user_func(Closure)
#5 /var/www/html/includes/Revision/RevisionSlots.php(117): MediaWiki\Revision\RevisionSlots->getSlots()
#6 /var/www/html/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot(string)
#7 /var/www/html/includes/page/WikiPage.php(643): MediaWiki\Revision\RevisionRecord->getSlot(string, integer)
#8 /var/www/html/includes/libs/objectcache/wancache/WANObjectCache.php(1529): WikiPage->{closure}(boolean, integer, array, NULL, array)
#9 /var/www/html/includes/libs/objectcache/wancache/WANObjectCache.php(1376): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#10 /var/www/html/includes/page/WikiPage.php(651): WANObjectCache->getWithSetCallback(string, integer, Closure)
#11 /var/www/html/includes/page/WikiPage.php(311): WikiPage->getContentModel()
#12 /var/www/html/includes/page/WikiPage.php(297): WikiPage->getContentHandler()
#13 /var/www/html/includes/page/Article.php(2547): WikiPage->getActionOverrides()
#14 /var/www/html/includes/actions/Action.php(130): Article->getActionOverrides()
#15 /var/www/html/includes/actions/Action.php(189): Action::factory(string, Article, RequestContext)
#16 /var/www/html/includes/MediaWiki.php(166): Action::getActionName(RequestContext)
#17 /var/www/html/includes/MediaWiki.php(903): MediaWiki->getAction()
#18 /var/www/html/includes/MediaWiki.php(543): MediaWiki->main()
#19 /var/www/html/index.php(53): MediaWiki->run()
#20 /var/www/html/index.php(46): wfIndexMain()
#21 {main}

I reverted the database and ran populateContentTables.php before the main update and that fixed the problem.

daniel lowered the priority of this task from High to Medium.Jan 26 2021, 12:08 PM
daniel changed the subtype of this task from "Production Error" to "Task".

The remaining reported instances are issues during upgrades of 3rd party installs. No longer a production error.

We are facing this error as well. Our setup is not master/slave DB.

We were at 1.31.10 and trying to upgrade to 1.35 we faced a myriad of DB inconsistency issues pertaining to actor creation which takes place as part of the schema migration that prevented update.php from completing. So we decided to make intermediate upgrades instead.

We are now at version MW1.33.4. update.php completes successfully.
However all pages other than Special pages output a variation on a stacktrace:


MediaWiki internal error.

Original exception: [70dd072ddcab266f70e2b671] /wiki/%D9%83%D9%8A%D9%81%D9%8A%D8%A9_%D8%A5%D9%86%D8%B4%D8%A7%D8%A1_%D8%B5%D9%81%D8%AD%D8%A9_%D9%81%D8%B1%D8%B9%D9%8A%D8%A9 MediaWiki\Revision\RevisionAccessException from line 1643 of /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionStore.php: Main slot of revision 42340 not found in database!
Backtrace:
#0 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionStore.php(1680): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#1 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#2 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionSlots.php(165): call_user_func()
#3 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionSlots.php(107): MediaWiki\Revision\RevisionSlots->getSlots()
#4 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#5 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision.php(689): MediaWiki\Revision\RevisionRecord->getSlot()
#6 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision.php(956): Revision->getMainSlotRaw()
#7 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(659): Revision->getContentModel()
#8 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/libs/objectcache/WANObjectCache.php(1414): WikiPage->{closure}()
#9 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/libs/objectcache/WANObjectCache.php(1275): WANObjectCache->doGetWithSetCallback()
#10 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(665): WANObjectCache->getWithSetCallback()
#11 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(287): WikiPage->getContentModel()
#12 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(274): WikiPage->getContentHandler()
#13 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/actions/Action.php(98): WikiPage->getActionOverrides()
#14 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/actions/Action.php(155): Action::factory()
#15 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(155): Action::getActionName()
#16 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(782): MediaWiki->getAction()
#17 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(515): MediaWiki->main()
#18 /var/www/adef.xyz/wiki/mediawiki-1.33/index.php(42): MediaWiki->run()
#19 {main}

Exception caught inside exception handler: [70dd072ddcab266f70e2b671] /wiki/%D9%83%D9%8A%D9%81%D9%8A%D8%A9_%D8%A5%D9%86%D8%B4%D8%A7%D8%A1_%D8%B5%D9%81%D8%AD%D8%A9_%D9%81%D8%B1%D8%B9%D9%8A%D8%A9 MediaWiki\Revision\RevisionAccessException from line 1643 of /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionStore.php: Main slot of revision 42340 not found in database!
Backtrace:
#0 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionStore.php(1680): MediaWiki\Revision\RevisionStore->loadSlotRecords()
#1 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#2 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionSlots.php(165): call_user_func()
#3 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionSlots.php(107): MediaWiki\Revision\RevisionSlots->getSlots()
#4 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionRecord.php(192): MediaWiki\Revision\RevisionSlots->getSlot()
#5 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision.php(689): MediaWiki\Revision\RevisionRecord->getSlot()
#6 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision.php(956): Revision->getMainSlotRaw()
#7 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(659): Revision->getContentModel()
#8 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/libs/objectcache/WANObjectCache.php(1414): WikiPage->{closure}()
#9 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/libs/objectcache/WANObjectCache.php(1275): WANObjectCache->doGetWithSetCallback()
#10 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(665): WANObjectCache->getWithSetCallback()
#11 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(287): WikiPage->getContentModel()
#12 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/WikiPage.php(274): WikiPage->getContentHandler()
#13 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/actions/Action.php(98): WikiPage->getActionOverrides()
#14 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/actions/Action.php(155): Action::factory()
#15 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/skins/SkinTemplate.php(256): Action::getActionName()
#16 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/skins/SkinTemplate.php(452): SkinTemplate->wrapHTML()
#17 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/skins/SkinTemplate.php(228): SkinTemplate->prepareQuickTemplate()
#18 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/OutputPage.php(2723): SkinTemplate->outputPage()
#19 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/exception/MWExceptionRenderer.php(134): OutputPage->output()
#20 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/exception/MWExceptionRenderer.php(53): MWExceptionRenderer::reportHTML()
#21 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/exception/MWExceptionHandler.php(98): MWExceptionRenderer::output()
#22 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/exception/MWExceptionHandler.php(172): MWExceptionHandler::report()
#23 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(542): MWExceptionHandler::handleException()
#24 /var/www/adef.xyz/wiki/mediawiki-1.33/index.php(42): MediaWiki->run()

#25 {main}

Looking at reports of similar mistakes we found this ticket, which seems to mostly have to do with master/slave DB setups, as well as a reference to $wgMultiContentRevisionSchemaMigrationStage the application of which changes the internal error to something like:


[76cf7e5794422e8a1eec25d8] /wiki/%D8%A7%D9%84%D8%B5%D9%81%D8%AD%D8%A9_%D8%A7%D9%84%D8%B1%D8%A6%D9%8A%D8%B3%D9%8A%D8%A9 MediaWiki\Revision\IncompleteRevisionException from line 333 of /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/SlotRecord.php: Uninitialized field: content_address

Backtrace:

#0 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/SlotRecord.php(360): MediaWiki\Revision\SlotRecord->getField()
#1 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/SlotRecord.php(500): MediaWiki\Revision\SlotRecord->getStringField()
#2 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionStore.php(1459): MediaWiki\Revision\SlotRecord->getAddress()
#3 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionStore.php(1373): MediaWiki\Revision\RevisionStore->loadSlotContent()
#4 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}()
#5 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/SlotRecord.php(307): call_user_func()
#6 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionRecord.php(175): MediaWiki\Revision\SlotRecord->getContent()
#7 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RenderedRevision.php(226): MediaWiki\Revision\RevisionRecord->getContent()
#8 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionRenderer.php(193): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#9 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RevisionRenderer.php(142): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#10 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}()
#11 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/Revision/RenderedRevision.php(197): call_user_func()
#12 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#13 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#14 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/page/Article.php(773): PoolCounterWork->execute()
#15 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/actions/ViewAction.php(68): Article->view()
#16 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(499): ViewAction->show()
#17 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(294): MediaWiki->performAction()
#18 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(865): MediaWiki->performRequest()
#19 /var/www/adef.xyz/wiki/mediawiki-1.33/includes/MediaWiki.php(515): MediaWiki->main()
#20 /var/www/adef.xyz/wiki/mediawiki-1.33/index.php(42): MediaWiki->run()

#21 {main}

What does this error actually mean?
If we were to try to bring the database to a consistent state, what tables and/or columns will be looking at?

Nikerabbit added a subscriber: Nikerabbit.

I'm seeing Main slot of revision not found in database. See T212428. in translatewiki.net logs. I'm not sure what I should see here or what should I do to resolve it.

I'm seeing Main slot of revision not found in database. See T212428. in translatewiki.net logs. I'm not sure what I should see here or what should I do to resolve it.

The root cause of this issue is unknown, and I suspect there are multiple causes.

The error message points here to we can collect more information about how and when the error occurrs.

This is a known bug in the update script. The slots table was created in 1.31.10 (minor version). Normally there are only database changes in a major version. The update.php script should have automatically run the populateContentTables.php script during the upgrade, but it does not.

After having run the first part of the update.php script, and getting the error "Main slot of revision not found in database", you can run the populateContentTables.php manually, and immediately run the update.php a 2nd time to execute the remaining upgrade instructions (without restoring the database to the initial backup).

This is a known bug in the update script. The slots table was created in 1.31.10 (minor version). Normally there are only database changes in a major version. The update.php script should have automatically run the populateContentTables.php script during the upgrade, but it does not.

I recall there being a problem with the installer at the time, but as far as I know, updating to 1.32 or later should work fine. Is this not the case?

In any case, translatewiki.net is running 1.37-alpha, like the WMF sites. It ran the schema update in question a long time ago. So the cause has to be something else entirely... Or maybe some pages were missed in the update at the time for some reason, and have been producing errors ever since?

This is a known bug in the update script. The slots table was created in 1.31.10 (minor version). Normally there are only database changes in a major version. The update.php script should have automatically run the populateContentTables.php script during the upgrade, but it does not.

After having run the first part of the update.php script, and getting the error "Main slot of revision not found in database", you can run the populateContentTables.php manually, and immediately run the update.php a 2nd time to execute the remaining upgrade instructions (without restoring the database to the initial backup).

It wasn't 1.31.10... https://github.com/wikimedia/mediawiki/compare/1.31.9...1.31.10

Slots was there in 1.31.0-rc.0 as per rMW943c724198f1: MCR database schema

I get really confused now.

I do not understand then how I could possibly have encountered the error when migrating from 1.31.0 to 1.35.0?

In addition to that, I read now that https://www.mediawiki.org/wiki/Manual:PopulateContentTables.php was only introduced with MediaWiki 1.32.0?

Can any developer shed a light on this strange error? Or is "slot(s)" just used in two different contexts?

I get really confused now.

I do not understand then how I could possibly have encountered the error when migrating from 1.31.0 to 1.35.0?

In addition to that, I read now that https://www.mediawiki.org/wiki/Manual:PopulateContentTables.php was only introduced with MediaWiki 1.32.0?

Can any developer shed a light on this strange error? Or is "slot(s)" just used in two different contexts?

I wrote the relevant schema change. When and why the error occurs is still not clear to me.

As to the versions: if I recall correctly, we added the tables to the schema in 1.31, but using them was experimental. In 1.32, we made the installer perform the migration to the new schema.

I am not aware of a bug that would prevent an update from 1.31 to 1.35, but I have not tested this recently. There were some people who had issues because the migration script choked on data corruption in the database left over from previous bugs or buggy extensions, and were left with a partially migrated database.

This ticket is still open because we are still trying to understand the circumstances that lead to this issue.