Page MenuHomePhabricator

ApiUsageException: There is no revision with ID [REDACTED].
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   ApiUsageException: There is no revision with ID [REDACTED].
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiUsageException.php(71)
#0 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiBase.php(1383): ApiUsageException::newWithMessage(ApiParse, array, NULL, NULL, NULL)
#1 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiParse.php(156): ApiBase->dieWithError(array)
#2 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiMain.php(1721): ApiParse->execute()
#3 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiMain.php(671): ApiMain->executeAction()
#4 /srv/mediawiki/php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(45): ApiMain->execute()
#5 /srv/mediawiki/php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(94): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::getParsedRevision(MediaWiki\Revision\RevisionStoreRecord)
#6 /srv/mediawiki/php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Hooks/EchoHooks.php(91): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::generateEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord)
#7 /srv/mediawiki/php-1.37.0-wmf.11/includes/HookContainer/HookContainer.php(338): MediaWiki\Extension\DiscussionTools\Hooks\EchoHooks::onEchoGetEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord, boolean)
#8 /srv/mediawiki/php-1.37.0-wmf.11/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#9 /srv/mediawiki/php-1.37.0-wmf.11/includes/Hooks.php(137): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#10 /srv/mediawiki/php-1.37.0-wmf.11/extensions/Echo/includes/DiscussionParser.php(165): Hooks::run(string, array)
#11 /srv/mediawiki/php-1.37.0-wmf.11/extensions/Echo/includes/EchoHooks.php(546): EchoDiscussionParser::generateEventsForRevision(MediaWiki\Revision\RevisionStoreRecord, boolean)
#12 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/MWCallableUpdate.php(38): EchoHooks::{closure}()
#13 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(513): MWCallableUpdate->doUpdate()
#14 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(390): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#15 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#16 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(MWCallableUpdate, integer)
#17 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#18 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#19 /srv/mediawiki/php-1.37.0-wmf.11/includes/MediaWiki.php(1136): DeferredUpdates::doUpdates(string)
#20 /srv/mediawiki/php-1.37.0-wmf.11/includes/MediaWiki.php(846): MediaWiki->restInPeace()
#21 /srv/mediawiki/php-1.37.0-wmf.11/api.php(125): MediaWiki->doPostOutputShutdown()
#22 /srv/mediawiki/php-1.37.0-wmf.11/api.php(45): wfApiMain()
#23 /srv/mediawiki/w/api.php(3): require(string)
#24 {main}
Impact

Significant number of error messages for 1.37.0-wmf.11.

Notes

Event Timeline

matmarex subscribed.

The error is coming from DiscussionTools' topic subscription feature, enabled on different wikis on Monday and Tuesday (T274280 and T284491).

You can disable that code in config using $wgDiscussionToolsEnableTopicSubscriptionBackend = false. I don't know what is the error rate, and whether it warrants that?

The impact of the error is that some Echo notifications may not be delivered for the edits where the exception occurs (e.g. mentions). I hope each deferred update is isolated, but if they're not, then other deferred updates may also not happen (e.g. updates to page links).

I suspect the issue is caused by querying replica databases, when the data we need may not have been replicated yet. We had this problem earlier on the beta cluster, but we thought we fixed it: T284175.

We need to do stuff on the HTML of pages, and we use the action=parse API internally to do that (that's why there's a ApiUsageException), and there's no way to make it return data from the primary database – I thought that MediaWiki would take care of that (because it ordinarily ensures that users see the effects of their actions), but I guess it does not. We should get the HTML in a different way.

I don't know what is the error rate, and whether it warrants that?

311 instances so far over 2 days, around 15 per hour recently, if my search query is correct: https://logstash.wikimedia.org/goto/aedc4ef716be7625340871dad3f6d2a9

image.png (535×2 px, 78 KB)

Change 702457 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] [WIP] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702457

Change 702457 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702457

Change 702708 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.11] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702708

Change 702709 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.12] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702709

Change 702708 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.11] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702708

Change 702709 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.12] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702709

Mentioned in SAL (#wikimedia-operations) [2021-07-01T18:18:44Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: 6d9043087ec421e1321cd6797934928e2651b1c1: EventDispatcher: Ensure we fetch page content from the primary database (T285895) (duration: 01m 14s)

Mentioned in SAL (#wikimedia-operations) [2021-07-01T18:20:17Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.12/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: 654877f92fa18ae766d693630025c69400cad3ac: EventDispatcher: Ensure we fetch page content from the primary database (T285895) (duration: 01m 12s)

matmarex added subscribers: Urbanecm, Pchelolo.

The patch seems to have fixed the issue, with the last exception in the query https://logstash.wikimedia.org/goto/aedc4ef716be7625340871dad3f6d2a9 occurring at 18:17 UTC (3 minutes before the deployment, 1.5 hour ago).

image.png (527×2 px, 81 KB)

There also aren't any new exceptions on the mediawiki-new-errors dashboard https://logstash.wikimedia.org/app/kibana#/dashboard/0a9ecdc0-b6dc-11e8-9d8f-dbc23b470465 (errors like "Could not load revision for notifications" or "Revision had no page" could be expected if the code change didn't fix the underlying issue).

And of course, our notifications are still being delivered, according to a quick manual check I did on https://test2.wikipedia.org/wiki/Talk:Main_Page.

Thanks @dduvall for reporting, @Pchelolo for review help, and @Urbanecm for actually deploying the changes.

ppelberg claimed this task.

Thanks @dduvall for reporting, @Pchelolo for review help, and @Urbanecm for actually deploying the changes.

+1. Thanks, y'all.