Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'arbcom_dewiki.echo_push_subscription' doesn't exist (db1175)Function: EchoPush\SubscriptionManager::getSubscriptionsForUserQuery: SELECT * FROM `echo_push_subscription` INNER JOIN `echo_push_provider` ON ((eps_provider = epp_id)) LEFT JOIN `echo_push_topic` ON ((eps_topic = ept_id)) WHERE eps_user = 91
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Assigned To

Authored By

	hashar
	Sep 15 2021, 7:42 PM

Description

Error

mwversion: 1.37.0-wmf.21
reqId: a2cda76d-fa69-488c-bf42-af57e16d5b79
Find reqId in Logstash
Find normalized_message in Logstash

normalized_message

[{reqId}] {exception_url}   Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'arbcom_dewiki.echo_push_subscription' doesn't exist (db1175)
Function: EchoPush\SubscriptionManager::getSubscriptionsForUser
Query: SELECT  *  FROM `echo_push_subscription` INNER

exception.trace

from /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1794)
#0 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1778): Wikimedia\Rdbms\Database->getQueryException(string, integer, string, string)
#1 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1753): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#2 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1312): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#3 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/Database.php(1997): Wikimedia\Rdbms\Database->query(string, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(68): Wikimedia\Rdbms\Database->select(array, string, array, string, array, array)
#5 /srv/mediawiki/php-1.37.0-wmf.21/includes/libs/rdbms/database/DBConnRef.php(317): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#6 /srv/mediawiki/php-1.37.0-wmf.21/extensions/Echo/includes/Push/SubscriptionManager.php(102): Wikimedia\Rdbms\DBConnRef->select(array, string, array, string, array, array)
#7 /srv/mediawiki/php-1.37.0-wmf.21/extensions/Echo/includes/Push/NotificationRequestJob.php(17): EchoPush\SubscriptionManager->getSubscriptionsForUser(integer)
#8 /srv/mediawiki/php-1.37.0-wmf.21/extensions/EventBus/includes/JobExecutor.php(79): EchoPush\NotificationRequestJob->run()
#9 /srv/mediawiki/rpc/RunSingleJob.php(76): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#10 {main}

Impact

Notes

There are variant of the messages for different db:

db1175
db1179
db1123
db1112

Possibly others.

Details

Request URL: https://jobrunner.discovery.wmnet/rpc/RunSingleJob.php

	Subject	Repo	Branch	Lines +/-
	Set wmgEchoEnablePush to false explicitly on arbcom_* wikis	operations/mediawiki-config	master	+10 -0

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	BUG REPORT	Jdforrester-WMF	T290640 Undefined variable: wgFileBlacklist
Resolved	Release	• mmodell	T281164 1.37.0-wmf.23 deployment blockers
Resolved	PRODUCTION ERROR	Urbanecm	T291128 Wikimedia\Rdbms\DBQueryError: Error 1146: Table 'arbcom_dewiki.echo_push_subscription' doesn't exist (db1175)Function: EchoPush\SubscriptionManager::getSubscriptionsForUserQuery: SELECT * FROM `echo_push_subscription` INNER JOIN `echo_push_provider` ON ((eps_provider = epp_id)) LEFT JOIN `echo_push_topic` ON ((eps_topic = ept_id)) WHERE eps_user = 91

Event Timeline

hashar triaged this task as Unbreak Now! priority.Sep 15 2021, 7:42 PM

hashar created this task.

Restricted Application added a project: Growth-Team. · View Herald TranscriptSep 15 2021, 7:42 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

hashar added a parent task: T281164: 1.37.0-wmf.23 deployment blockers.Sep 15 2021, 7:42 PM

kostajh added a project: Product-Infrastructure-Team-Backlog-Deprecated.Sep 15 2021, 7:43 PM

RhinosF1 added a project: MW-1.37-release.Sep 15 2021, 7:55 PM

Restricted Application added a subscriber: RhinosF1. · View Herald TranscriptSep 15 2021, 7:55 PM

Zabe subscribed.Sep 15 2021, 8:07 PM

Change 721396 had a related patch set uploaded (by Urbanecm; author: Urbanecm):

[operations/mediawiki-config@master] Set wmgEchoEnablePush to false explicitly on arbcom_* wikis

https://gerrit.wikimedia.org/r/721396

gerritbot added a project: Patch-For-Review.Sep 15 2021, 8:55 PM

Change 721396 merged by jenkins-bot:

[operations/mediawiki-config@master] Set wmgEchoEnablePush to false explicitly on arbcom_* wikis

https://gerrit.wikimedia.org/r/721396

Mentioned in SAL (#wikimedia-operations) [2021-09-15T21:00:35Z] <urbanecm@deploy1002> Synchronized wmf-config/InitialiseSettings.php: 60e7e515d7034a9f839d78851f1dcc2be3df7f3b: Set wmgEchoEnablePush to false explicitly on arbcom_* wikis (T291128) (duration: 01m 06s)

Should be fixed now :). Removing the release tag, as it is caused by WMF's config being incorrect, not by code.

Restricted Application added a project: User-Urbanecm. · View Herald TranscriptSep 15 2021, 9:05 PM

Maintenance_bot removed a project: Patch-For-Review.Sep 15 2021, 9:10 PM

To memorialize more fully what happened here:

App push notification support is currently only enabled on the wikipedia group of wikis. The intention behind this was to enable the feature only on public Wikipedias. DB tables supporting the push feature were only created in x1/wikishared on the assumption that the feature was only to be enabled on wikis for which Echo uses a shared cluster/DB (i.e., only on SUL wikis).

The error in this task was triggered because the wikipedia group in fact includes several private wikis, including the arbcom_* wikis, sysop_itwiki, and wg_enwiki, which do not use the shared cluster/DB, and a recent change (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/713708, see T288770) updated app push to default to enabled for several categories of notifications. This resulted in MediaWiki searching for a nonexistent local subscriptions table when an event triggering a push notification occurred on one of those wikis.

The immediate solution was to disable push on those wikis individually. I'll be filing a follow-up task to document the issue and propose next steps.

Thank you for the quick fix up @Urbanecm and thank you @Mholloway for the root cause investigation and the summary.

• Mholloway mentioned this in T291200: wgEchoEnablePush config is brittle.Sep 16 2021, 3:50 PM

Krinkle moved this task from Untriaged to Sep 2021 on the Wikimedia-production-error board.Mar 14 2022, 4:23 PM