Page MenuHomePhabricator
Create Task
Maniphest T288775

InvariantException: Invariant failed: Comments are always preceded by headings
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Assert\InvariantException: Invariant failed: Comments are always preceded by headings
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.18/vendor/wikimedia/assert/src/Assert.php(224)
#0 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(119): Wikimedia\Assert\Assert::invariant(boolean, string)
#1 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(144): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::groupCommentsByThreadAndName(array)
#2 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php(102): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::generateEventsFromParsers(array, MediaWiki\Extension\DiscussionTools\CommentParser, MediaWiki\Extension\DiscussionTools\CommentParser, MediaWiki\Revision\RevisionStoreRecord, Title, User)
#3 /srv/mediawiki/php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Hooks/EchoHooks.php(91): MediaWiki\Extension\DiscussionTools\Notifications\EventDispatcher::generateEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord)
#4 /srv/mediawiki/php-1.37.0-wmf.18/includes/HookContainer/HookContainer.php(338): MediaWiki\Extension\DiscussionTools\Hooks\EchoHooks::onEchoGetEventsForRevision(array, MediaWiki\Revision\RevisionStoreRecord, boolean)
#5 /srv/mediawiki/php-1.37.0-wmf.18/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#6 /srv/mediawiki/php-1.37.0-wmf.18/includes/Hooks.php(137): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#7 /srv/mediawiki/php-1.37.0-wmf.18/extensions/Echo/includes/DiscussionParser.php(165): Hooks::run(string, array)
#8 /srv/mediawiki/php-1.37.0-wmf.18/extensions/Echo/includes/EchoHooks.php(494): EchoDiscussionParser::generateEventsForRevision(MediaWiki\Revision\RevisionStoreRecord, boolean)
#9 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/MWCallableUpdate.php(38): EchoHooks::{closure}()
#10 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(515): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(391): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#12 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(221): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#13 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(MWCallableUpdate, integer)
#14 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#15 /srv/mediawiki/php-1.37.0-wmf.18/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#16 /srv/mediawiki/php-1.37.0-wmf.18/includes/MediaWiki.php(1136): DeferredUpdates::doUpdates(string)
#17 /srv/mediawiki/php-1.37.0-wmf.18/includes/MediaWiki.php(846): MediaWiki->restInPeace()
#18 /srv/mediawiki/php-1.37.0-wmf.18/includes/MediaWiki.php(584): MediaWiki->doPostOutputShutdown()
#19 /srv/mediawiki/php-1.37.0-wmf.18/index.php(53): MediaWiki->run()
#20 /srv/mediawiki/php-1.37.0-wmf.18/index.php(46): wfIndexMain()
#21 /srv/mediawiki/w/index.php(3): require(string)
#22 {main}
Impact
  1. Echo notifications will not be sent for some edits where they should have been
  2. logspam.
Notes

Not sure if there is visible breakage but there is quite a lot of logspam from this.

Details

Request URL
https://commons.wikimedia.org/w/index.php?title=*&action=submit
SubjectRepoBranchLines +/-
EventDispatcher: Fix ignoring level 3+ headingsmediawiki/extensions/DiscussionToolsmaster+86 -5
EventDispatcher: Remove failing invariant checkmediawiki/extensions/DiscussionToolsmaster+2 -2
EventDispatcher: Remove failing invariant checkmediawiki/extensions/DiscussionToolswmf/1.37.0-wmf.18+2 -2
Customize query in gerrit

Related Objects

Event Timeline

• mmodell created this task.Aug 12 2021, 8:17 PM
Restricted Application added a project: Growth-Team. · View Herald TranscriptAug 12 2021, 8:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• mmodell added a comment.Aug 12 2021, 8:18 PM

This may be a train blocker (T281159) if the error rate goes up significantly after deploying to group2 wikis, however, it may be isolated to commons in which case I don't expect the error rate to change.

• mmodell triaged this task as High priority.Aug 12 2021, 8:19 PM
• mmodell moved this task from Untriaged to Aug 2021 on the Wikimedia-production-error board.Aug 12 2021, 8:21 PM
matmarex subscribed.Aug 12 2021, 8:31 PM

The impact of this is that Echo notifications will not be sent for some edits where they should have been. No other visible breakage.

matmarex added a comment.Aug 12 2021, 8:35 PM

I'll prepare a quick patch to fail more neatly in this case, and thus limit the impact further to only DiscussionTools notifications (which are pre-beta right now), which will unblock the train if backported. I'll investigate more closely tomorrow.

• mmodell added a parent task: T281159: 1.37.0-wmf.18 deployment blockers.Aug 12 2021, 8:37 PM

Thanks @matmarex

gerritbot added a comment.Aug 12 2021, 8:38 PM

Change 712576 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/712576

gerritbot added a project: Patch-For-Review.Aug 12 2021, 8:38 PM
• mmodell updated the task description. (Show Details)Aug 12 2021, 8:38 PM
gerritbot added a comment.Aug 12 2021, 8:42 PM

Change 711720 had a related patch set uploaded (by 20after4; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.18] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/711720

matmarex added a project: Editing-team (Kanban Board).Aug 12 2021, 8:43 PM
matmarex moved this task from Incoming to Doing on the Editing-team (Kanban Board) board.
gerritbot added a comment.Aug 12 2021, 8:48 PM

Change 712576 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/712576

gerritbot added a comment.Aug 12 2021, 8:49 PM

Change 711720 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@wmf/1.37.0-wmf.18] EventDispatcher: Remove failing invariant check

https://gerrit.wikimedia.org/r/711720

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.18; 2021-08-09).Aug 12 2021, 9:00 PM
Maintenance_bot removed a project: Patch-For-Review.Aug 12 2021, 9:10 PM
Stashbot mentioned this in T281159: 1.37.0-wmf.18 deployment blockers.Aug 12 2021, 9:13 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-12T21:13:19Z] <twentyafterfour@deploy1002> Synchronized php-1.37.0-wmf.18/extensions/DiscussionTools/includes/Notifications/EventDispatcher.php: sync Ic27418a0ec976347be5fa586bbd32cc4a0d8d511 to unblock the train refs T288775 and T281159 (duration: 01m 07s)

• mmodell added a comment.Aug 12 2021, 9:20 PM

Deployed and no longer spamming logs. leaving the task open to track any further work but this is no longer a blocker.

• mmodell removed a parent task: T281159: 1.37.0-wmf.18 deployment blockers.Aug 12 2021, 9:20 PM
gerritbot added a comment.Aug 14 2021, 4:11 PM

Change 713054 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Fix ignoring level 3+ headings

https://gerrit.wikimedia.org/r/713054

gerritbot added a project: Patch-For-Review.Aug 14 2021, 4:11 PM
matmarex moved this task from Doing to Code Review on the Editing-team (Kanban Board) board.Aug 14 2021, 4:11 PM
gerritbot added a comment.Aug 16 2021, 10:08 PM

Change 713054 merged by jenkins-bot:

[mediawiki/extensions/DiscussionTools@master] EventDispatcher: Fix ignoring level 3+ headings

https://gerrit.wikimedia.org/r/713054

Maintenance_bot removed a project: Patch-For-Review.Aug 16 2021, 10:10 PM
• mewoph moved this task from Inbox to Needs Discussion on the Growth-Team board.Aug 16 2021, 10:15 PM
ReleaseTaggerBot edited projects, added MW-1.37-notes (1.37.0-wmf.19; 2021-08-16); removed MW-1.37-notes (1.37.0-wmf.18; 2021-08-09).Aug 16 2021, 11:00 PM
matmarex removed projects: Growth-Team, Notifications.Aug 17 2021, 3:09 PM
matmarex moved this task from Code Review to Ready for Sign Off on the Editing-team (Kanban Board) board.Aug 17 2021, 3:22 PM
matmarex added a project: Skipped QA.

Logstash search showing the errors, for reference: https://logstash.wikimedia.org/goto/bf4338fa3af353cd46d05eb959c76dd8

image.png (511×2 px, 67 KB)

The errors occurred on pages that started with a level 3 heading, without any level 2 headings or comments preceding it. DiscussionTools topic subscription feature only allows subscribing to level 2 headings, and so it tried to find the preceding level 2 heading to generate notifications for those subscribed to it, and failed when there wasn't one.

• ppelberg closed this task as Resolved.Aug 18 2021, 3:43 PM
• ppelberg claimed this task.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL