The main two offenders are:
```
Expectation (writes <= 0) by MediaWiki::restInPeace not met (actual: 5):
query-m: DELETE FROM `echo_unread_wikis` WHERE euw_user = 'X'
#0 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/TransactionProfiler.php(245): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/database/Database.php(1318): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion()
#2 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/database/Database.php(1198): Wikimedia\Rdbms\Database->attemptQuery()
#3 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/database/Database.php(3004): Wikimedia\Rdbms\Database->query()
#4 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/UnreadWikis.php(141): Wikimedia\Rdbms\Database->delete()
#5 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/NotifUser.php(440): EchoUnreadWikis->updateCount()
#6 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/NotifUser.php(249): MWEchoNotifUser->resetNotificationCount()
#7 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/EchoHooks.php(940): MWEchoNotifUser->markRead()
#8 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/MWCallableUpdate.php(34): Closure$EchoHooks::processMarkAsRead#2()
#9 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/DeferredUpdates.php(273): MWCallableUpdate->doUpdate()
#10 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/DeferredUpdates.php(219): DeferredUpdates::runUpdate()
#11 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/DeferredUpdates.php(143): DeferredUpdates::execute()
#12 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(909): DeferredUpdates::doUpdates()
#13 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(733): MediaWiki->restInPeace()
#14 (): Closure$MediaWiki::doPostOutputShutdown()
#15 {main}
```
```
Expectation (writes <= 0) by MediaWiki::restInPeace not met (actual: 13):
query-m: INSERT INTO `echo_event` (event_type,event_variant,event_deleted,event_extra,event_agent_id,event_page_id) VALUES ('X')
#0 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/TransactionProfiler.php(245): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/database/Database.php(1318): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion()
#2 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/database/Database.php(1198): Wikimedia\Rdbms\Database->attemptQuery()
#3 /srv/mediawiki/php-1.33.0-wmf.23/includes/libs/rdbms/database/Database.php(2123): Wikimedia\Rdbms\Database->query()
#4 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/mapper/EventMapper.php(22): Wikimedia\Rdbms\Database->insert()
#5 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/model/Event.php(232): EchoEventMapper->insert()
#6 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/model/Event.php(167): EchoEvent->insert()
#7 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/DiscussionParser.php(95): EchoEvent::create()
#8 /srv/mediawiki/php-1.33.0-wmf.23/extensions/Echo/includes/EchoHooks.php(561): EchoDiscussionParser::generateEventsForRevision()
#9 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/MWCallableUpdate.php(34): Closure$EchoHooks::onPageContentSaveComplete()
#10 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/DeferredUpdates.php(273): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/DeferredUpdates.php(219): DeferredUpdates::runUpdate()
#12 /srv/mediawiki/php-1.33.0-wmf.23/includes/deferred/DeferredUpdates.php(143): DeferredUpdates::execute()
#13 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(909): DeferredUpdates::doUpdates()
#14 /srv/mediawiki/php-1.33.0-wmf.23/includes/MediaWiki.php(733): MediaWiki->restInPeace()
#15 (): Closure$MediaWiki::doPostOutputShutdown()
#16 {main}
```
## Locations where the problem occurs
See https://logstash.wikimedia.org/goto/4fb8c9be478c3795bd2e59b90ad75b5b
### UserClearNewTalkNotification hook
``` lang=php
/**
* Handler for UserClearNewTalkNotification hook.
* @see https://www.mediawiki.org/wiki/Manual:Hooks/UserClearNewTalkNotification
* @param UserIdentity $user User whose talk page notification should be marked as read
*/
public static function onUserClearNewTalkNotification( UserIdentity $user ) {
if ( $user->isRegistered() ) {
$userObj = User::newFromIdentity( $user );
DeferredUpdates::addCallableUpdate( static function () use ( $userObj ) {
MWEchoNotifUser::newFromUser( $userObj )->clearUserTalkNotifications();
} );
}
}
```
That in turn is invoked in `WatchlistManager::clearTitleUserNotifications()` and that is called on GET from `WikiPage::doViewUpdates()` and Flow's `ViewAction::showForAction()`.
### onSkinTemplateNavigationUniversal hook
This hook calls `EchoHooks::processMarkAsRead()` which ends up doing:
```lang=php
DeferredUpdates::addCallableUpdate( static function () use ( $user, $eventIds ) {
$notifUser = MWEchoNotifUser::newFromUser( $user );
$notifUser->markRead( $eventIds );
} );
```
### onUserSaveSettings hook
```lang=php
public static function onUserSaveSettings( $user ) {
// Extensions like AbuseFilter might create an account, but
// the tables we need might not exist. Bug 57335
if ( !defined( 'MW_UPDATER' ) ) {
// Reset the notification count since it may have changed due to user
// option changes. This covers both explicit changes in the preferences
// and changes made through the options API (since both call this hook).
DeferredUpdates::addCallableUpdate( static function () use ( $user ) {
MWEchoNotifUser::newFromUser( $user )->resetNotificationCount();
} );
}
```
I'm not sure how this ends up in a GET context, this is invoked in `User::saveSettings()` which is doing database writes.
### LoginNotify extension
LoginNotify.php has a function `recordLoginFailureFromKnownSystem()` which creates an Echo notification:
```
from /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(444)
#0 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(280): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, integer)
#1 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(1471): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer)
#2 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(1353): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(1278): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(2405): Wikimedia\Rdbms\Database->query(string, string, integer)
#5 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(2385): Wikimedia\Rdbms\Database->doInsert(string, array, string)
#6 /srv/mediawiki/php-1.37.0-wmf.7/extensions/Echo/includes/mapper/EventMapper.php(22): Wikimedia\Rdbms\Database->insert(string, array, string)
#7 /srv/mediawiki/php-1.37.0-wmf.7/extensions/Echo/includes/model/Event.php(259): EchoEventMapper->insert(EchoEvent)
#8 /srv/mediawiki/php-1.37.0-wmf.7/extensions/Echo/includes/model/Event.php(189): EchoEvent->insert()
#9 /srv/mediawiki/php-1.37.0-wmf.7/extensions/LoginNotify/includes/LoginNotify.php(717): EchoEvent::create(array)
#10 /srv/mediawiki/php-1.37.0-wmf.7/extensions/LoginNotify/includes/LoginNotify.php(698): LoginNotify\LoginNotify->sendNotice(User, string, integer)
#11 /srv/mediawiki/php-1.37.0-wmf.7/extensions/LoginNotify/includes/LoginNotify.php(783): LoginNotify\LoginNotify->recordLoginFailureFromKnownSystem(User)
#12 /srv/mediawiki/php-1.37.0-wmf.7/extensions/LoginNotify/includes/Hooks.php(154): LoginNotify\LoginNotify->recordFailure(User)
#13 /srv/mediawiki/php-1.37.0-wmf.7/extensions/LoginNotify/includes/Hooks.php(128): LoginNotify\Hooks::doFailedLogin(User)
#14 /srv/mediawiki/php-1.37.0-wmf.7/includes/HookContainer/HookContainer.php(330): LoginNotify\Hooks::onAuthManagerLoginAuthenticateAudit(MediaWiki\Auth\AuthenticationResponse, NULL, string, array)
#15 /srv/mediawiki/php-1.37.0-wmf.7/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#16 /srv/mediawiki/php-1.37.0-wmf.7/includes/HookContainer/HookRunner.php(895): MediaWiki\HookContainer\HookContainer->run(string, array)
#17 /srv/mediawiki/php-1.37.0-wmf.7/includes/auth/AuthManager.php(400): MediaWiki\HookContainer\HookRunner->onAuthManagerLoginAuthenticateAudit(MediaWiki\Auth\AuthenticationResponse, NULL, string, array)
#18 /srv/mediawiki/php-1.37.0-wmf.7/includes/api/ApiLogin.php(162): MediaWiki\Auth\AuthManager->beginAuthentication(array, string)
#19 /srv/mediawiki/php-1.37.0-wmf.7/includes/api/ApiMain.php(1669): ApiLogin->execute()
#20 /srv/mediawiki/php-1.37.0-wmf.7/includes/api/ApiMain.php(639): ApiMain->executeAction()
#21 /srv/mediawiki/php-1.37.0-wmf.7/includes/api/ApiMain.php(610): ApiMain->executeActionWithErrorHandling()
#22 /srv/mediawiki/php-1.37.0-wmf.7/api.php(90): ApiMain->execute()
#23 /srv/mediawiki/php-1.37.0-wmf.7/api.php(45): wfApiMain()
#24 /srv/mediawiki/w/api.php(3): require(string)
#25 {main}
```
```
Expectation (writes <=) 0 by ApiMain::setRequestExpectations not met (actual: 1):
query-m: INSERT INTO `echo_event` (event_type,event_variant,event_deleted,event_extra,event_agent_id) VALUES ('X',N)
```
I suppose either ApiLogin.php needs `isWriteMode()` set to true, or this could be a good case for enqueuing a job.
### EchoModerationController
```
from /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(444)
#0 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/TransactionProfiler.php(280): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, integer)
#1 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(1471): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer)
#2 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(1353): Wikimedia\Rdbms\Database->executeQueryAttempt(string, string, boolean, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(1278): Wikimedia\Rdbms\Database->executeQuery(string, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/DatabaseMysqlBase.php(1379): Wikimedia\Rdbms\Database->query(string, string, integer)
#5 /srv/mediawiki/php-1.37.0-wmf.7/includes/libs/rdbms/database/Database.php(3434): Wikimedia\Rdbms\DatabaseMysqlBase->doUpsert(string, array, array, array, string)
#6 /srv/mediawiki/php-1.37.0-wmf.7/extensions/Echo/includes/UnreadWikis.php(131): Wikimedia\Rdbms\Database->upsert(string, array, array, array, string)
#7 /srv/mediawiki/php-1.37.0-wmf.7/extensions/Echo/includes/NotifUser.php(463): EchoUnreadWikis->updateCount(string, integer, boolean, integer, MWTimestamp)
#8 /srv/mediawiki/php-1.37.0-wmf.7/extensions/Echo/includes/controller/ModerationController.php(40): MWEchoNotifUser->resetNotificationCount()
#9 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/MWCallableUpdate.php(38): EchoModerationController::{closure}()
#10 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdates.php(513): MWCallableUpdate->doUpdate()
#11 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdates.php(390): DeferredUpdates::attemptUpdate(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti)
#12 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdates.php(234): DeferredUpdates::run(MWCallableUpdate, Wikimedia\Rdbms\LBFactoryMulti, Monolog\Logger, BufferingStatsdDataFactory, string)
#13 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(MWCallableUpdate, integer)
#14 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#15 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdates.php(237): DeferredUpdatesScope->processUpdates(integer, Closure)
#16 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdatesScope.php(267): DeferredUpdates::{closure}(EchoDeferredMarkAsDeletedUpdate, integer)
#17 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdatesScope.php(196): DeferredUpdatesScope->processStageQueue(integer, integer, Closure)
#18 /srv/mediawiki/php-1.37.0-wmf.7/includes/deferred/DeferredUpdates.php(242): DeferredUpdatesScope->processUpdates(integer, Closure)
#19 /srv/mediawiki/php-1.37.0-wmf.7/includes/MediaWiki.php(1128): DeferredUpdates::doUpdates(string)
#20 /srv/mediawiki/php-1.37.0-wmf.7/includes/MediaWiki.php(838): MediaWiki->restInPeace()
#21 /srv/mediawiki/php-1.37.0-wmf.7/api.php(125): MediaWiki->doPostOutputShutdown()
#22 /srv/mediawiki/php-1.37.0-wmf.7/api.php(45): wfApiMain()
#23 /srv/mediawiki/w/api.php(3): require(string)
#24 {main}
```
Maybe this is originating from `DeferredMarkAsDeletedUpdate.php`? Not sure. That might seem like a good candidate for a job.