Page MenuHomePhabricator

Jobrunners generate mediawiki exceptions upon calling Closure$RecentChange::save
Closed, DeclinedPublicPRODUCTION ERROR

Description

A lot of errors today due to errors in Closure$RecentChange::save causing alarms on #wikimedia-operations:

exception.message	Cannot access the database: Unknown error (10.64.48.150) --> db1084, commonswiki

#0 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(995): Wikimedia\Rdbms\Database->reportConnectionError(string)
#1 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(666): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#2 /srv/mediawiki/php-1.30.0-wmf.7/includes/GlobalFunctions.php(3100): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, boolean)
#3 /srv/mediawiki/php-1.30.0-wmf.7/includes/user/User.php(783): wfGetDB(integer)
#4 /srv/mediawiki/php-1.30.0-wmf.7/includes/user/User.php(402): User::idFromName(string, integer)
#5 /srv/mediawiki/php-1.30.0-wmf.7/includes/user/User.php(2232): User->load()
#6 /srv/mediawiki/php-1.30.0-wmf.7/includes/WatchedItemStore.php(736): User->getId()
#7 /srv/mediawiki/php-1.30.0-wmf.7/includes/mail/EmailNotification.php(135): WatchedItemStore->updateNotificationTimestamp(User, Title, string)
#8 /srv/mediawiki/php-1.30.0-wmf.7/includes/changes/RecentChange.php(369): EmailNotification->notifyOnPageChange(User, Title, string, string, integer, integer, string)
#9 [internal function]: Closure$RecentChange::save(integer)
#10 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/database/Database.php(2657): call_user_func_array(Closure$RecentChange::save;1387, array)
#11 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/database/Database.php(2891): Wikimedia\Rdbms\Database->runOnTransactionIdleCallbacks(integer)
#12 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/database/Database.php(756): Wikimedia\Rdbms\Database->commit(string, string)
#13 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1084): Wikimedia\Rdbms\Database->close()
#14 [internal function]: Closure$Wikimedia\Rdbms\LoadBalancer::closeAll(Wikimedia\Rdbms\DatabaseMysqli)
#15 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1485): call_user_func_array(Closure$Wikimedia\Rdbms\LoadBalancer::closeAll;1052, array)
#16 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1085): Wikimedia\Rdbms\LoadBalancer->forEachOpenConnection(Closure$Wikimedia\Rdbms\LoadBalancer::closeAll;1052)
#17 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1076): Wikimedia\Rdbms\LoadBalancer->closeAll()
#18 /srv/mediawiki/php-1.30.0-wmf.7/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1665): Wikimedia\Rdbms\LoadBalancer->disable()
#19 [internal function]: Wikimedia\Rdbms\LoadBalancer->__destruct()
#20 {main}

https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2017.07.06/mediawiki?id=AV0YE_14vQld8_eNxipI&_g=()

Event Timeline

elukey updated the task description. (Show Details)
Krinkle subscribed.

Not seen in Logstash for at least 7 days (searching mediawiki-errors for RecentChange and Cannot access the database).

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:10 PM