ApiUsageException: There is no revision with ID [REDACTED].
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	dduvall
	Jun 30 2021, 6:46 PM

Description

Error

mwversion: 1.37.0-wmf.11
reqId: 9b7198fa-9420-40f9-98eb-51e85aff28f4
Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   ApiUsageException: There is no revision with ID [REDACTED].

exception.trace

from /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiUsageException.php(71)
#0 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiBase.php(1383): ApiUsageException::newWithMessage(ApiParse, array, NULL, NULL, NULL)
#1 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiParse.php(156): ApiBase->dieWithError(array)
#2 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiMain.php(1721): ApiParse->execute()
#3 /srv/mediawiki/php-1.37.0-wmf.11/includes/api/ApiMain.php(671): ApiMain->executeAction()
#4 /srv/mediawiki/php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(45): ApiMain->execute()
#5 /srv/mediawiki/php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(94): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::getParsedRevision(MediaWiki\Revision\RevisionStoreRecord)
#6 /srv/mediawiki/php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Hooks/EchoHooks.php(91): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::generateEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord)
#7 /srv/mediawiki/php-1.37.0-wmf.11/includes/HookContainer/HookContainer.php(338): MediaWiki\Extension\DiscussionTools\Hooks\EchoHooks::onEchoGetEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord, boolean)
#8 /srv/mediawiki/php-1.37.0-wmf.11/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#9 /srv/mediawiki/php-1.37.0-wmf.11/includes/Hooks.php(137): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#10 /srv/mediawiki/php-1.37.0-wmf.11/extensions/Echo/includes/DiscussionParser.php(165): Hooks::run(string, array)
#11 /srv/mediawiki/php-1.37.0-wmf.11/extensions/Echo/includes/EchoHooks.php(546): EchoDiscussionParser::generateEventsForRevision(MediaWiki\Revision\RevisionStoreRecord, boolean)
#12 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/MWCallableUpdate.php(38): EchoHooks::{closure}()
#13 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(513): MWCallableUpdate->doUpdate()
#14 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(390): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#15 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#16 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(MWCallableUpdate, integer)
#17 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#18 /srv/mediawiki/php-1.37.0-wmf.11/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#19 /srv/mediawiki/php-1.37.0-wmf.11/includes/MediaWiki.php(1136): DeferredUpdates::doUpdates(string)
#20 /srv/mediawiki/php-1.37.0-wmf.11/includes/MediaWiki.php(846): MediaWiki->restInPeace()
#21 /srv/mediawiki/php-1.37.0-wmf.11/api.php(125): MediaWiki->doPostOutputShutdown()
#22 /srv/mediawiki/php-1.37.0-wmf.11/api.php(45): wfApiMain()
#23 /srv/mediawiki/w/api.php(3): require(string)
#24 {main}

Impact

Significant number of error messages for 1.37.0-wmf.11.

Notes

Details

Request URL: https://www.wikidata.org/w/api.php

Project	Branch	Lines +/-	Subject
mediawiki/extensions/DiscussionTools	wmf/1.37.0-wmf.12	+21 -16	EventDispatcher: Ensure we fetch page content from the primary database
mediawiki/extensions/DiscussionTools	wmf/1.37.0-wmf.11	+21 -16	EventDispatcher: Ensure we fetch page content from the primary database
mediawiki/extensions/DiscussionTools	master	+21 -16	EventDispatcher: Ensure we fetch page content from the primary database

Customize query in gerrit

Related Objects

Mentioned In: T289717: Wikimedia\Assert\PostconditionException: Postcondition failed: Revision had no page
Mentioned Here: rEDTO654877f92fa1: EventDispatcher: Ensure we fetch page content from the primary database
rEDTO6d9043087ec4: EventDispatcher: Ensure we fetch page content from the primary database
T284175: Topic subscription notifications are not sent on the beta cluster
T274280: Deploy config change to make Topic Subscriptions available as beta feature
T284491: Make topic subscriptions available via URL parameter

Event Timeline

dduvall created this task.Jun 30 2021, 6:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 30 2021, 6:46 PM

The error is coming from DiscussionTools' topic subscription feature, enabled on different wikis on Monday and Tuesday (T274280 and T284491).

You can disable that code in config using $wgDiscussionToolsEnableTopicSubscriptionBackend = false. I don't know what is the error rate, and whether it warrants that?

The impact of the error is that some Echo notifications may not be delivered for the edits where the exception occurs (e.g. mentions). I hope each deferred update is isolated, but if they're not, then other deferred updates may also not happen (e.g. updates to page links).

I suspect the issue is caused by querying replica databases, when the data we need may not have been replicated yet. We had this problem earlier on the beta cluster, but we thought we fixed it: T284175.

We need to do stuff on the HTML of pages, and we use the action=parse API internally to do that (that's why there's a ApiUsageException), and there's no way to make it return data from the primary database – I thought that MediaWiki would take care of that (because it ordinarily ensures that users see the effects of their actions), but I guess it does not. We should get the HTML in a different way.

I don't know what is the error rate, and whether it warrants that?

311 instances so far over 2 days, around 15 per hour recently, if my search query is correct: https://logstash.wikimedia.org/goto/aedc4ef716be7625340871dad3f6d2a9

matmarex edited projects, added Editing-team (Kanban Board); removed Growth-Team.Jun 30 2021, 8:10 PM

Restricted Application added a project: Growth-Team. · View Herald TranscriptJun 30 2021, 8:10 PM

Change 702457 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] [WIP] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702457

gerritbot added a project: Patch-For-Review.Jun 30 2021, 8:25 PM

matmarex moved this task from Incoming to Code Review on the Editing-team (Kanban Board) board.Jun 30 2021, 8:48 PM

ppelberg added a subscriber: ppelberg.Jun 30 2021, 9:11 PM

matmarex moved this task from Backlog to Triaged on the DiscussionTools board.Jul 1 2021, 2:55 PM

Change 702457 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702457

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.14; 2021-07-12).Jul 1 2021, 5:00 PM

Change 702708 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.11] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702708

Change 702709 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.12] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702709

matmarex moved this task from Code Review to Doing on the Editing-team (Kanban Board) board.Jul 1 2021, 5:04 PM

Change 702708 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.11] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702708

Change 702709 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.12] EventDispatcher: Ensure we fetch page content from the primary database

https://gerrit.wikimedia.org/r/702709

Maintenance_bot removed a project: Patch-For-Review.Jul 1 2021, 6:10 PM

Mentioned in SAL (#wikimedia-operations) [2021-07-01T18:18:44Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.11/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: 6d9043087ec421e1321cd6797934928e2651b1c1: EventDispatcher: Ensure we fetch page content from the primary database (T285895) (duration: 01m 14s)

Mentioned in SAL (#wikimedia-operations) [2021-07-01T18:20:17Z] <urbanecm@deploy1002> Synchronized php-1.37.0-wmf.12/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: 654877f92fa18ae766d693630025c69400cad3ac: EventDispatcher: Ensure we fetch page content from the primary database (T285895) (duration: 01m 12s)

ReleaseTaggerBot edited projects, added MW-1.37-notes (1.37.0-wmf.11; 2021-06-21); removed MW-1.37-notes (1.37.0-wmf.14; 2021-07-12).Jul 1 2021, 7:00 PM

The patch seems to have fixed the issue, with the last exception in the query https://logstash.wikimedia.org/goto/aedc4ef716be7625340871dad3f6d2a9 occurring at 18:17 UTC (3 minutes before the deployment, 1.5 hour ago).

There also aren't any new exceptions on the mediawiki-new-errors dashboard https://logstash.wikimedia.org/app/kibana#/dashboard/0a9ecdc0-b6dc-11e8-9d8f-dbc23b470465 (errors like "Could not load revision for notifications" or "Revision had no page" could be expected if the code change didn't fix the underlying issue).

And of course, our notifications are still being delivered, according to a quick manual check I did on https://test2.wikipedia.org/wiki/Talk:Main_Page.

Thanks @dduvall for reporting, @Pchelolo for review help, and @Urbanecm for actually deploying the changes.

In T285895#7191116, @matmarex wrote:

Thanks @dduvall for reporting, @Pchelolo for review help, and @Urbanecm for actually deploying the changes.

+1. Thanks, y'all.

matmarex mentioned this in T289717: Wikimedia\Assert\PostconditionException: Postcondition failed: Revision had no page.Aug 25 2021, 7:22 PM

Krinkle moved this task from Untriaged to June 2021 on the Wikimedia-production-error board.Sep 5 2021, 12:26 AM

	F34534703: image.png
	Jul 1 2021, 7:59 PM

ApiUsageException: There is no revision with ID [REDACTED].Closed, ResolvedPublicPRODUCTION ERRORActions