Page MenuHomePhabricator
Create Task
Maniphest T288775

InvariantException: Invariant failed: Comments are always preceded by headings
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Assert\InvariantException: Invariant failed: Comments are always preceded by headings
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.18/vendor/wikimedia/assert/src/Assert.php(224)
#0 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(119): Wikimedia\Assert\Assert::invariant(boolean, string)
#1 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(144): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::groupCommentsByThreadAndName(array)
#2 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(102): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::generateEventsFromParsers(array, MediaWiki\Extension\DiscussionTools\CommentParser, MediaWiki\Extension\DiscussionTools\CommentParser, MediaWiki\Revision\RevisionStoreRecord, Title, User)
#3 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Hooks/EchoHooks.php(91): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::generateEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord)
#4 /srv/mediawiki/php-1.37.0-wmf.18/includes/HookContainer/HookContainer.php(338): MediaWiki\Extension\DiscussionTools\Hooks\EchoHooks::onEchoGetEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord, boolean)
#5 /srv/mediawiki/php-1.37.0-wmf.18/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#6 /srv/mediawiki/php-1.37.0-wmf.18/includes/Hooks.php(137): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#7 /srv/mediawiki/php-1.37.0-wmf.18/extensions/Echo/includes/DiscussionParser.php(165): Hooks::run(string, array)
#8 /srv/mediawiki/php-1.37.0-wmf.18/extensions/Echo/includes/EchoHooks.php(494): EchoDiscussionParser::generateEventsForRevision(MediaWiki\Revision\RevisionStoreRecord, boolean)
#9 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/MWCallableUpdate.php(38): EchoHooks::{closure}()
#10 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(515): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(391): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#12 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#13 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(MWCallableUpdate, integer)
#14 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#15 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#16 /srv/mediawiki/php-1.37.0-wmf.18/includes/MediaWiki.php(1136): DeferredUpdates::doUpdates(string)
#17 /srv/mediawiki/php-1.37.0-wmf.18/includes/MediaWiki.php(846): MediaWiki->restInPeace()
#18 /srv/mediawiki/php-1.37.0-wmf.18/includes/MediaWiki.php(584): MediaWiki->doPostOutputShutdown()
#19 /srv/mediawiki/php-1.37.0-wmf.18/index.php(53): MediaWiki->run()
#20 /srv/mediawiki/php-1.37.0-wmf.18/index.php(46): wfIndexMain()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}
Impact
  1. Echo notifications will not be sent for some edits where they should have been
  2. logspam.
Notes

Not sure if there is visible breakage but there is quite a lot of logspam from this.

Details

Request URL
https://commons.wikimedia.org/w/index.php?title=*&action=submit
ProjectBranchLines +/-Subject
mediawiki/extensions/DiscussionToolsmaster+86 -5EventDispatcher: Fix ignoring level 3+ headings
mediawiki/extensions/DiscussionToolsmaster+2 -2EventDispatcher: Remove failing invariant check
mediawiki/extensions/DiscussionToolswmf/1.37.0-wmf.18+2 -2EventDispatcher: Remove failing invariant check
Customize query in gerrit

Related Objects

Event Timeline

mmodell created this task.Aug 12 2021, 8:17 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptAug 12 2021, 8:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
mmodell added a comment.Aug 12 2021, 8:18 PM

This may be a train blocker (T281159) if the error rate goes up significantly after deploying to group2 wikis, however, it may be isolated to commons in which case I don't expect the error rate to change.

mmodell triaged this task as High priority.Aug 12 2021, 8:19 PM
mmodell moved this task from Untriaged to Aug 2021 on the Wikimedia-production-error board.Aug 12 2021, 8:21 PM
matmarex added a subscriber: matmarex.Aug 12 2021, 8:31 PM

The impact of this is that Echo notifications will not be sent for some edits where they should have been. No other visible breakage.

matmarex added a comment.Aug 12 2021, 8:35 PM

I'll prepare a quick patch to fail more neatly in this case, and thus limit the impact further to only DiscussionTools notifications (which are pre-beta right now), which will unblock the train if backported. I'll investigate more closely tomorrow.

mmodell added a parent task: T281159: 1.37.0-wmf.18 deployment blockers.Aug 12 2021, 8:37 PM

Thanks @matmarex

gerritbot added a comment.Aug 12 2021, 8:38 PM

Change 712576 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/712576

gerritbot added a project: Patch-For-Review.Aug 12 2021, 8:38 PM
mmodell updated the task description. (Show Details)Aug 12 2021, 8:38 PM
gerritbot added a comment.Aug 12 2021, 8:42 PM

Change 711720 had a related patch set uploaded (by 20after4; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.18] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/711720

matmarex added a project: Editing-team (Kanban Board).Aug 12 2021, 8:43 PM
matmarex moved this task from Incoming to Doing on the Editing-team (Kanban Board) board.
gerritbot added a comment.Aug 12 2021, 8:48 PM

Change 712576 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/712576

gerritbot added a comment.Aug 12 2021, 8:49 PM

Change 711720 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.18] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/711720

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.18; 2021-08-09).Aug 12 2021, 9:00 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 12 2021, 9:10 PM
Stashbot mentioned this in T281159: 1.37.0-wmf.18 deployment blockers.Aug 12 2021, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-12T21:13:19Z] <twentyafterfour@deploy1002> Synchronized php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: sync Ic27418a0ec976347be5fa586bbd32cc4a0d8d511 to unblock the train refs T288775 and T281159 (duration: 01m 07s)

mmodell added a comment.Aug 12 2021, 9:20 PM

Deployed and no longer spamming logs. leaving the task open to track any further work but this is no longer a blocker.

mmodell removed a parent task: T281159: 1.37.0-wmf.18 deployment blockers.Aug 12 2021, 9:20 PM
gerritbot added a comment.Aug 14 2021, 4:11 PM

Change 713054 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Fix ignoring level 3+ headings

https://gerrit.wikimedia.org/r/713054

gerritbot added a project: Patch-For-Review.Aug 14 2021, 4:11 PM
matmarex moved this task from Doing to Code Review on the Editing-team (Kanban Board) board.Aug 14 2021, 4:11 PM
gerritbot added a comment.Aug 16 2021, 10:08 PM

Change 713054 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Fix ignoring level 3+ headings

https://gerrit.wikimedia.org/r/713054

Maintenance_bot removed a project: Patch-For-Review.Aug 16 2021, 10:10 PM
• mewoph moved this task from Inbox to Needs Discussion/Analysis on the Growth-Team board.Aug 16 2021, 10:15 PM
ReleaseTaggerBot edited projects, added MW-1.37-notes (1.37.0-wmf.19; 2021-08-16); removed MW-1.37-notes (1.37.0-wmf.18; 2021-08-09).Aug 16 2021, 11:00 PM
matmarex removed projects: Growth-Team, Notifications.Aug 17 2021, 3:09 PM
matmarex moved this task from Code Review to Ready for Sign Off on the Editing-team (Kanban Board) board.Aug 17 2021, 3:22 PM
matmarex added a project: Skipped QA.

Logstash search showing the errors, for reference: https://logstash.wikimedia.org/goto/bf4338fa3af353cd46d05eb959c76dd8

image.png (511×2 px, 67 KB)

The errors occurred on pages that started with a level 3 heading, without any level 2 headings or comments preceding it. DiscussionTools topic subscription feature only allows subscribing to level 2 headings, and so it tried to find the preceding level 2 heading to generate notifications for those subscribed to it, and failed when there wasn't one.

ppelberg closed this task as Resolved.Aug 18 2021, 3:43 PM
ppelberg claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL