Page MenuHomePhabricator
Create Task
Maniphest T216287

BannerMessageGroup::registerGroupHook of CentralNotice must not query master on GET request (page views)
Open, HighPublicPRODUCTION ERROR
Actions

Description

The master DB of the deployment-prep MediaWiki cluster is down but the replicas are up:

tgr@deployment-deploy01:~$ sql metawiki --write
ERROR 2003 (HY000): Can't connect to MySQL server on '172.16.5.23' (113 "No route to host")

tgr@deployment-deploy01:~$ sql metawiki
Welcome to the MariaDB monitor.

Visiting https://meta.wikimedia.beta.wmflabs.org/wiki/Main_Page in an incognito window gives this exception:

Sorry! This site is experiencing technical difficulties.
Try waiting a few minutes and reloading.

(Cannot access the database: Cannot access the database: Unknown error (172.16.5.23:3306))

Backtrace:

#0 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(751): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-master/includes/GlobalFunctions.php(2654): Wikimedia\Rdbms\LoadBalancer->getConnection()
#2 /srv/mediawiki/php-master/extensions/CentralNotice/includes/CNDatabase.php(62): wfGetDB()
#3 /srv/mediawiki/php-master/extensions/CentralNotice/includes/BannerMessageGroup.php(230): CNDatabase::getDb()
#4 /srv/mediawiki/php-master/includes/Hooks.php(174): BannerMessageGroup::registerGroupHook()
#5 /srv/mediawiki/php-master/includes/Hooks.php(202): Hooks::callHook()
#6 /srv/mediawiki/php-master/extensions/Translate/MessageGroups.php(62): Hooks::run()
#7 /srv/mediawiki/php-master/includes/libs/objectcache/WANObjectCache.php(1326): Closure$MessageGroups::getCachedGroupDefinitions()
#8 /srv/mediawiki/php-master/includes/libs/objectcache/WANObjectCache.php(1190): WANObjectCache->doGetWithSetCallback()
#9 /srv/mediawiki/php-master/extensions/Translate/MessageGroups.php(92): WANObjectCache->getWithSetCallback()
#10 /srv/mediawiki/php-master/extensions/Translate/MessageGroups.php(42): MessageGroups->getCachedGroupDefinitions()
#11 /srv/mediawiki/php-master/extensions/Translate/MessageGroups.php(586): MessageGroups->init()
#12 /srv/mediawiki/php-master/extensions/Translate/MessageGroups.php(316): MessageGroups->getGroups()
#13 /srv/mediawiki/php-master/extensions/Translate/tag/TranslatablePage.php(209): MessageGroups::getGroup()
#14 /srv/mediawiki/php-master/extensions/Translate/tag/TranslatablePage.php(769): TranslatablePage->getMessageGroup()
#15 /srv/mediawiki/php-master/extensions/Translate/tag/PageTranslationHooks.php(411): TranslatablePage->getTranslationPercentages()
#16 /srv/mediawiki/php-master/extensions/Translate/tag/PageTranslationHooks.php(256): PageTranslationHooks::getTranslatablePageStatus()
#17 /srv/mediawiki/php-master/includes/parser/Parser.php(3987): PageTranslationHooks::languages()
#18 /srv/mediawiki/php-master/includes/parser/Preprocessor_Hash.php(1188): Parser->extensionSubstitution()
#19 /srv/mediawiki/php-master/includes/parser/Parser.php(3036): PPFrame_Hash->expand()
#20 /srv/mediawiki/php-master/includes/parser/Parser.php(1354): Parser->replaceVariables()
#21 /srv/mediawiki/php-master/includes/parser/Parser.php(482): Parser->internalParse()
#22 /srv/mediawiki/php-master/includes/content/WikitextContent.php(369): Parser->parse()
#23 /srv/mediawiki/php-master/includes/content/AbstractContent.php(555): WikitextContent->fillParserOutput()
#24 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(266): AbstractContent->getParserOutput()
#25 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(234): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached()
#26 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(193): MediaWiki\Revision\RenderedRevision->getSlotParserOutput()
#27 /srv/mediawiki/php-master/includes/Revision/RevisionRenderer.php(142): MediaWiki\Revision\RevisionRenderer->combineSlotOutput()
#28 (): Closure$MediaWiki\Revision\RevisionRenderer::getRenderedRevision#2()
#29 /srv/mediawiki/php-master/includes/Revision/RenderedRevision.php(197): call_user_func()
#30 /srv/mediawiki/php-master/includes/poolcounter/PoolWorkArticleView.php(194): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#31 /srv/mediawiki/php-master/includes/poolcounter/PoolCounterWork.php(123): PoolWorkArticleView->doWork()
#32 /srv/mediawiki/php-master/includes/page/Article.php(774): PoolCounterWork->execute()
#33 /srv/mediawiki/php-master/includes/actions/ViewAction.php(68): Article->view()
#34 /srv/mediawiki/php-master/includes/MediaWiki.php(501): ViewAction->show()
#35 /srv/mediawiki/php-master/includes/MediaWiki.php(294): MediaWiki->performAction()
#36 /srv/mediawiki/php-master/includes/MediaWiki.php(867): MediaWiki->performRequest()
#37 /srv/mediawiki/php-master/includes/MediaWiki.php(517): MediaWiki->main()
#38 /srv/mediawiki/php-master/index.php(42): MediaWiki->run()
#39 /srv/mediawiki/w/index.php(3): include()
#40 {main}

That's not great error recovery. MediaWiki should not need master reads just to show the main page, but if it does, it should at least resume in readonly mode on error. (Also, what's up with the error message?)

Related Objects
Search...

Event Timeline

Tgr created this task.Feb 15 2019, 10:09 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 15 2019, 10:09 PM
DatGuy subscribed.Feb 15 2019, 10:23 PM
Krinkle edited projects, added MediaWiki-extensions-CentralNotice, Wikimedia-production-error, Sustainability (MediaWiki-MultiDC); removed MediaWiki-libs-Rdbms.Feb 19 2019, 12:31 AM
Krinkle subscribed.

MediaWiki should not need master reads just to show the main page, but if it does, it should at least resume in readonly mode on error.

It doesn't. The error comes from CentralNotice via Translate, and are from rare code paths (various levels of cache miss). If they happen in prod (not seen recently), they'd emit DBPerformance warnings and prioritised as a Wikimedia-production-error.

Krinkle renamed this task from MediaWiki should work in readonly mode when the master is down to CentralNotice must not query master on GET request (page views).Feb 19 2019, 12:31 AM
Krinkle added a parent task: T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc.
Krinkle updated the task description. (Show Details)
Krinkle added a comment.Feb 19 2019, 12:34 AM
CentralNotice/includes/BannerMessageGroup.php
	/**
	 * TranslatePostInitGroups hook handler
	 *
	 * Add banner message groups to the list of message groups that should be
	 * translated through the Translate extension.
	 *
	 * @param array &$list
	 * @return bool
	 */
	public static function registerGroupHook( &$list ) {
		$dbr = CNDatabase::getDb( DB_MASTER );
		                        # ^^^^^^^^^

Source

Krinkle moved this task from Untriaged to Mar 2021 on the Wikimedia-production-error board.Feb 19 2019, 12:35 AM

(Tagging as Meta for now, as only seen in beta, but no reason to believe it is limited to that.)

Krinkle added a project: Performance-Team (Radar).Feb 19 2019, 12:35 AM
Krinkle moved this task from Limbo to Perf recommendation on the Performance-Team (Radar) board.
AndyRussG added a project: Fundraising-Backlog.Feb 19 2019, 2:36 AM
AndyRussG subscribed.
• Quennlatifa0303 triaged this task as Medium priority.Feb 19 2019, 4:01 AM
• Quennlatifa0303 set Due Date to Feb 19 2019, 12:00 AM.
Restricted Application changed the subtype of this task from "Task" to "Deadline". · View Herald TranscriptFeb 19 2019, 4:01 AM
Krinkle raised the priority of this task from Medium to Needs Triage.Feb 19 2019, 4:11 AM
Krinkle removed Due Date.
Restricted Application changed the subtype of this task from "Deadline" to "Task". · View Herald TranscriptFeb 19 2019, 4:11 AM
• DStrine moved this task from Triage to Q3 2021-2022 on the Fundraising-Backlog board.Feb 19 2019, 9:30 PM
aaron subscribed.Feb 20 2019, 12:57 AM

This looks like it should use DB_REPLICA.

Umherirrender subscribed.Apr 16 2019, 4:29 PM
In T216287#4967345, @aaron wrote:

This looks like it should use DB_REPLICA.

	public static function registerGroupHook( &$list ) {
		// Must be explicitly master for runs under a jobqueue
		$dbr = CNDatabase::getDb( DB_MASTER );
aaron added a comment.Apr 16 2019, 10:00 PM

If it's not actually part of a job, it should be DB_REPLICA. Also, even when it should use DB_MASTER it should not be using getCachedGroupDefinitions() since it should skip the cache.

Krinkle removed a project: Beta-Cluster-reproducible.May 22 2019, 6:38 PM
Krinkle moved this task from Mar 2021 to Older on the Wikimedia-production-error board.
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM
AndyRussG added projects: FR-Q2-FY2019-20-cleanup-list, Fr-CentralNotice-Translation-Bugs.Sep 6 2019, 4:11 AM
• DStrine moved this task from Q3 2021-2022 to Triage on the Fundraising-Backlog board.Dec 11 2019, 3:51 PM
• DStrine moved this task from Triage to Sprint +3 on the Fundraising-Backlog board.Dec 11 2019, 6:13 PM
• DStrine moved this task from Sprint +3 to Q3 2021-2022 on the Fundraising-Backlog board.Mar 3 2020, 4:01 PM
Krinkle triaged this task as High priority.Jul 2 2020, 4:03 PM
Krinkle moved this task from Later to Current: Other on the Sustainability (MediaWiki-MultiDC) board.
aaron added a comment.EditedJul 16 2020, 8:18 PM

BannerMessageGroup::registerGroupHook is triggering master queries (a) on HTTP GET requests and also (b) inside of a getWithSetCallback() call, which should use DB_REPLICA data.

aaron renamed this task from CentralNotice must not query master on GET request (page views) to BannerMessageGroup::registerGroupHook of CentralNotice must not query master on GET request (page views).Jul 16 2020, 8:19 PM
• DStrine moved this task from Q3 2021-2022 to Triage on the Fundraising-Backlog board.Jul 16 2020, 8:23 PM
• DStrine moved this task from Triage to Sprint +2 on the Fundraising-Backlog board.Jul 20 2020, 8:09 PM
• DStrine moved this task from Sprint +2 to Q3 2021-2022 on the Fundraising-Backlog board.Jan 3 2021, 8:36 PM
RhinosF1 subscribed.Jan 25 2021, 8:38 PM
Umherirrender unsubscribed.Feb 27 2021, 9:43 PM
Krinkle mentioned this in T164897: Translate DB master queries on page views.Mar 9 2021, 7:12 PM
Krinkle added a comment.Mar 9 2021, 7:14 PM

Still seen:

  • Request: GET meta.wikimedia.org /wiki/<some page name>/cs
  • reqId: YEfCorGzrOfMwJS5Kr-RUQAAAIs
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1):
trace
from /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/TransactionProfiler.php(378)
#0 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/TransactionProfiler.php(189): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, integer)
#1 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1006): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, boolean)
#2 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(960): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#3 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1126): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#4 /srv/mediawiki/php-1.36.0-wmf.33/includes/GlobalFunctions.php(2468): Wikimedia\Rdbms\LoadBalancer->getMaintenanceConnectionRef(integer, array, string)
#5 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CentralNotice/includes/CNDatabase.php(63): wfGetDB(integer)
#6 /srv/mediawiki/php-1.36.0-wmf.33/extensions/CentralNotice/includes/BannerMessageGroup.php(235): CNDatabase::getDb(integer)
#7 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookContainer.php(333): BannerMessageGroup::registerGroupHook(array, array, array)
#8 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookContainer.php(140): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#9 /srv/mediawiki/php-1.36.0-wmf.33/includes/Hooks.php(137): MediaWiki\HookContainer\HookContainer->run(string, array, array)
#10 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/MessageGroups.php(68): Hooks::run(string, array)
#11 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/objectcache/wancache/WANObjectCache.php(1604): MessageGroups->{closure}(DependencyWrapper, integer, array, double, array)
#12 /srv/mediawiki/php-1.36.0-wmf.33/includes/libs/objectcache/wancache/WANObjectCache.php(1432): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#13 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/MessageGroups.php(97): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#14 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/MessageGroups.php(44): MessageGroups->getCachedGroupDefinitions()
#15 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/MessageGroups.php(559): MessageGroups->init()
#16 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/MessageGroups.php(293): MessageGroups->getGroups()
#17 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/tag/TranslatablePage.php(187): MessageGroups::getGroup(string)
#18 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/tag/TranslatablePage.php(586): TranslatablePage->getMessageGroup()
#19 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/tag/PageTranslationHooks.php(1167): TranslatablePage->getTranslationPercentages()
#20 /srv/mediawiki/php-1.36.0-wmf.33/extensions/Translate/tag/PageTranslationHooks.php(1075): PageTranslationHooks::translationPageHeader(RequestContext, TranslatablePage)
#21 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookContainer.php(333): PageTranslationHooks::translatablePageHeader(Article, NULL, boolean)
#22 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookContainer.php(140): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#23 /srv/mediawiki/php-1.36.0-wmf.33/includes/HookContainer/HookRunner.php(902): MediaWiki\HookContainer\HookContainer->run(string, array)
#24 /srv/mediawiki/php-1.36.0-wmf.33/includes/page/Article.php(633): MediaWiki\HookContainer\HookRunner->onArticleViewHeader(Article, NULL, boolean)
#25 /srv/mediawiki/php-1.36.0-wmf.33/includes/page/Article.php(571): Article->generateContentOutput(User, ParserOptions, integer, OutputPage, array)
#26 /srv/mediawiki/php-1.36.0-wmf.33/includes/actions/ViewAction.php(74): Article->view()
#27 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(531): ViewAction->show()
#28 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(315): MediaWiki->performAction(Article, Title)
#29 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(925): MediaWiki->performRequest()
#30 /srv/mediawiki/php-1.36.0-wmf.33/includes/MediaWiki.php(547): MediaWiki->main()
#31 /srv/mediawiki/php-1.36.0-wmf.33/index.php(53): MediaWiki->run()
#32 /srv/mediawiki/php-1.36.0-wmf.33/index.php(46): wfIndexMain()
#33 /srv/mediawiki/w/index.php(3): require(string)
#34 {main}
XenoRyet moved this task from Q3 2021-2022 to Unscheduled on the Fundraising-Backlog board.Mar 13 2023, 5:48 PM
Krinkle edited projects, added Wikimedia-Performance-recommendation; removed Performance-Team (Radar).Aug 18 2023, 8:42 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL