Page MenuHomePhabricator

Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'arbcom_dewiki.echo_push_subscription' doesn't exist (db1175)Function: EchoPush\SubscriptionManager::getSubscriptionsForUserQuery: SELECT * FROM `echo_push_subscription` INNER JOIN `echo_push_provider` ON ((eps_provider = epp_id)) LEFT JOIN `echo_push_topic` ON ((eps_topic = ept_id)) WHERE eps_user = 91
Closed, ResolvedPublicPRODUCTION ERROR

Description

Error
normalized_message
[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'arbcom_dewiki.echo_push_subscription' doesn't exist (db1175)
Function: EchoPush\SubscriptionManager::getSubscriptionsForUser
Query: SELECT  *  FROM `echo_push_subscription` INNER
exception.trace
from /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1794)
#0 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1778): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1753): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1312): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1997): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select(array, string, array, string, array, array)
#5 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(317): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.37.0-wmf.21/extensions/Echo/includes/Push/SubscriptionManager.php(102): Wikimedia\Rdbms\DBConnRef->select(array, string, array, string, array, array)
#7 /srv/mediawiki/php-1.37.0-wmf.21/extensions/Echo/includes/Push/NotificationRequestJob.php(17): EchoPush\SubscriptionManager->getSubscriptionsForUser(integer)
#8 /srv/mediawiki/php-1.37.0-wmf.21/extensions/EventBus/includes/JobExecutor.php(79): EchoPush\NotificationRequestJob->run()
#9 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#10 {main}
Impact
Notes

There are variant of the messages for different db:

  • db1175
  • db1179
  • db1123
  • db1112

Possibly others.

Details

Request URL
https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

Event Timeline

hashar triaged this task as Unbreak Now! priority.Sep 15 2021, 7:42 PM
hashar created this task.
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 721396 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Set wmgEchoEnablePush to false explicitly on arbcom_* wikis

https://gerrit.wikimedia.org/r/721396

Change 721396 merged by jenkins-bot:

[operations/mediawiki-config@master] Set wmgEchoEnablePush to false explicitly on arbcom_* wikis

https://gerrit.wikimedia.org/r/721396

Mentioned in SAL (#wikimedia-operations) [2021-09-15T21:00:35Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 60e7e515d7034a9f839d78851f1dcc2be3df7f3b: Set wmgEchoEnablePush to false explicitly on arbcom_* wikis (T291128) (duration: 01m 06s)

Urbanecm claimed this task.
Urbanecm removed a project: MW-1.37-release.
Urbanecm added a subscriber: Urbanecm.

Should be fixed now :). Removing the release tag, as it is caused by WMF's config being incorrect, not by code.

To memorialize more fully what happened here:

App push notification support is currently only enabled on the wikipedia group of wikis. The intention behind this was to enable the feature only on public Wikipedias. DB tables supporting the push feature were only created in x1/wikishared on the assumption that the feature was only to be enabled on wikis for which Echo uses a shared cluster/DB (i.e., only on SUL wikis).

The error in this task was triggered because the wikipedia group in fact includes several private wikis, including the arbcom_* wikis, sysop_itwiki, and wg_enwiki, which do not use the shared cluster/DB, and a recent change (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/713708, see T288770) updated app push to default to enabled for several categories of notifications. This resulted in MediaWiki searching for a nonexistent local subscriptions table when an event triggering a push notification occurred on one of those wikis.

The immediate solution was to disable push on those wikis individually. I'll be filing a follow-up task to document the issue and propose next steps.

Thank you for the quick fix up @Urbanecm and thank you @Mholloway for the root cause investigation and the summary.