MWException: Internal error in ApiQueryRevisionsBase::getRevisionRecords: RevisionStore does not return record for [n]
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	brennen
	May 4 2023, 8:14 PM

Description

Error

service.version: 1.41.0-wmf.7
trace.id: c1b1da31-afeb-4ae2-aeb8-baf6981e262d
Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   MWException: Internal error in ApiQueryRevisionsBase::getRevisionRecords: RevisionStore does not return record for [redacted]

error.stack_trace

from /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiBase.php(1702)
#0 /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiQueryRevisionsBase.php(331): ApiBase::dieDebug(string, string)
#1 /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiQueryRevisions.php(407): ApiQueryRevisionsBase->getRevisionRecords(Wikimedia\Rdbms\MysqliResultWrapper)
#2 /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiQueryRevisionsBase.php(137): ApiQueryRevisions->run()
#3 /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiQuery.php(679): ApiQueryRevisionsBase->execute()
#4 /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiMain.php(1908): ApiQuery->execute()
#5 /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiMain.php(884): ApiMain->executeAction()
#6 /srv/mediawiki/php-1.41.0-wmf.7/includes/api/ApiMain.php(855): ApiMain->executeActionWithErrorHandling()
#7 /srv/mediawiki/php-1.41.0-wmf.7/api.php(91): ApiMain->execute()
#8 /srv/mediawiki/php-1.41.0-wmf.7/api.php(46): wfApiMain()
#9 /srv/mediawiki/w/api.php(3): require(string)
#10 {main}

Impact

Notes

Looks like ~50 of these in 1.41.0-wmf.7 (T330213).

Details

Request URL: https://de.wikipedia.org/w/api.php

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
Revert "api: Use RevisionStore::newRevisionsFromBatch to fetch revision records"	mediawiki/core	wmf/1.41.0-wmf.7	+4 -56
api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords	mediawiki/core	master	+1 -1
api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords	mediawiki/core	wmf/1.41.0-wmf.7	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T336022: 1.41.0-wmf.7 increases Memcached call rate by +100%
T330213: 1.41.0-wmf.7 deployment blockers
Mentioned Here: T336022: 1.41.0-wmf.7 increases Memcached call rate by +100%
T330213: 1.41.0-wmf.7 deployment blockers

Event Timeline

brennen created this task.May 4 2023, 8:14 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 4 2023, 8:14 PM

brennen updated the task description. (Show Details)May 4 2023, 8:15 PM

brennen mentioned this in T330213: 1.41.0-wmf.7 deployment blockers.May 4 2023, 8:17 PM

brennen added projects: MediaWiki-Core-Revision-backend, Platform Engineering.

RhinosF1 subscribed.May 4 2023, 8:22 PM

brennen added a parent task: T330213: 1.41.0-wmf.7 deployment blockers.May 4 2023, 8:31 PM

It is new code in wmf/1.41.0-wmf.7 from https://gerrit.wikimedia.org/r/c/mediawiki/core/+/910808

From the class name it is a action=query&prop=revisions

Change 915845 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@master] api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords

https://gerrit.wikimedia.org/r/915845

gerritbot added a project: Patch-For-Review.May 4 2023, 9:10 PM

For me the old code would also throw when there are problems with revisions. It's now just from a new location, so there could be some RevisionAccessException previous and now showing as internal errors.
I have uploaded a patch set to get better error messages from the ealier check in the same function (hopefully).

I have uploaded a patch set to get better error messages from the ealier check in the same function (hopefully).

Am I correct in thinking that this is just for debugging purposes, but will not unblock the train?

In T336008#8828350, @brennen wrote:

I have uploaded a patch set to get better error messages from the ealier check in the same function (hopefully).

Am I correct in thinking that this is just for debugging purposes, but will not unblock the train?

Yes, it can give hopefully some more information about what RevisionStore does not like, but it is also possible that is already known by some other production error task.
But it can also show that I am totally wrong and there is a real issue with the code.

Ok, let's go ahead and backport that one.

Change 915710 had a related patch set uploaded (by Brennen Bearnes; author: Umherirrender):

[mediawiki/core@wmf/1.41.0-wmf.7] api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords

https://gerrit.wikimedia.org/r/915710

Change 915710 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.7] api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords

https://gerrit.wikimedia.org/r/915710

Mentioned in SAL (#wikimedia-operations) [2023-05-04T22:03:03Z] <brennen@deploy1002> Started scap: Backport for [[gerrit:915710|api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords (T336008)]]

Change 915845 merged by jenkins-bot:

[mediawiki/core@master] api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords

https://gerrit.wikimedia.org/r/915845

Mentioned in SAL (#wikimedia-operations) [2023-05-04T22:04:32Z] <brennen@deploy1002> brennen: Backport for [[gerrit:915710|api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords (T336008)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug2002.codfw.wmnet, mwdebug1002.eqiad.wmnet

Maintenance_bot removed a project: Patch-For-Review.May 4 2023, 10:10 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-04T22:12:11Z] <brennen@deploy1002> Finished scap: Backport for [[gerrit:915710|api: Use Status::isGood in ApiQueryRevisionsBase::getRevisionRecords (T336008)]] (duration: 09m 07s)

Krinkle mentioned this in T336022: 1.41.0-wmf.7 increases Memcached call rate by +100%.May 5 2023, 2:05 AM

Change 915719 had a related patch set uploaded (by Umherirrender; author: Umherirrender):

[mediawiki/core@wmf/1.41.0-wmf.7] Revert "api: Use RevisionStore::newRevisionsFromBatch to fetch revision records"

https://gerrit.wikimedia.org/r/915719

gerritbot added a project: Patch-For-Review.May 5 2023, 5:24 PM

Mentioned in SAL (#wikimedia-operations) [2023-05-05T18:25:20Z] <brennen> train 1.41.0-wmf.7 (T330213): trying revert for T336008, T336022

Change 915719 merged by jenkins-bot:

[mediawiki/core@wmf/1.41.0-wmf.7] Revert "api: Use RevisionStore::newRevisionsFromBatch to fetch revision records"

https://gerrit.wikimedia.org/r/915719

Mentioned in SAL (#wikimedia-operations) [2023-05-05T18:42:44Z] <brennen@deploy1002> Started scap: Backport for [[gerrit:915719|Revert "api: Use RevisionStore::newRevisionsFromBatch to fetch revision records" (T336008 T336022)]]

Mentioned in SAL (#wikimedia-operations) [2023-05-05T18:44:17Z] <brennen@deploy1002> umherirrender and brennen: Backport for [[gerrit:915719|Revert "api: Use RevisionStore::newRevisionsFromBatch to fetch revision records" (T336008 T336022)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-05-05T18:57:05Z] <brennen@deploy1002> Finished scap: Backport for [[gerrit:915719|Revert "api: Use RevisionStore::newRevisionsFromBatch to fetch revision records" (T336008 T336022)]] (duration: 14m 21s)

Maintenance_bot removed a project: Patch-For-Review.May 5 2023, 7:10 PM

Removing as wmf.7 blocker since logspam seems fixed with the revert.

The warnings will be occur also on next train when nothing is done, it is possible to merge the revert into master (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/915718 feel free to remove -2) or trying to work on the warnings showing up.

But it is possible the warnings are not technical warnings, more warnings from RevisionStore about problems how the revisions are stored currently in the database, like other RevisionAccessException already exists (tasks with Wikimedia-database-issue (Bad data) for example).

Both code using RevisionStore::newRevisionFromRowAndSlots where I would assume the exceptions are from.
Possible the exceptions happens also for the old code, but that log spam possible is just hidden by the exception text, due to new or same text in different context now the issue is shown again. But in both code the api would give a error to the user.

The warnings will be occur also on next train when nothing is done, it is possible to merge the revert into master (https://gerrit.wikimedia.org/r/c/mediawiki/core/+/915718 feel free to remove -2) or trying to work on the warnings showing up.

For now, I think I would prefer to merge the revert to avoid logspam. If these result from the same problem as an existing source of logspam, could they be made to throw the existing more-generic exception?

cc: ~~@jeena~~ @hashar as train conductor this week.

brennen added a subscriber: hashar.May 8 2023, 3:41 PM

brennen moved this task from Backlog to Logs/Train on the User-brennen board.May 8 2023, 8:40 PM

Umherirrender closed this task as Resolved.May 8 2023, 8:49 PM

Umherirrender claimed this task.

ReleaseTaggerBot added a project: MW-1.41-notes (1.41.0-wmf.10; 2023-05-23).May 16 2023, 11:01 AM