Page MenuHomePhabricator
Create Task
Maniphest T151159

The Beta-Cluster is down
Closed, ResolvedPublic
Actions

Description

https://en.wikipedia.beta.wmflabs.org/wiki/

MediaWiki internal error.

Exception caught inside exception handler.

Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.

Related Objects

Event Timeline

Luke081515 created this task.Nov 20 2016, 5:40 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2016, 5:40 PM
Luke081515 triaged this task as Unbreak Now! priority.Nov 20 2016, 5:40 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptNov 20 2016, 5:40 PM
Paladox added a comment.Nov 20 2016, 5:41 PM

shinken-wm reported problem first happening at 12:59 pm utc + 0 time.

<shinken-wm> PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1164 bytes in 0.072 second response time
<shinken-wm> PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1164 bytes in 0.072 second response time
<shinken-wm> PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 728 bytes in 0.075 second response time

Luke081515 added a comment.Nov 20 2016, 5:41 PM

deployment & de show a different message, zh shows the same as at en.

Jogo.obb awarded a token.Nov 20 2016, 5:41 PM
Jogo.obb subscribed.
Reedy mentioned this in T148957: $wgShowExceptionDetails = true apparently is broken.Nov 20 2016, 9:09 PM
Reedy subscribed.Nov 20 2016, 9:45 PM

Various hacking to get this done

MediaWiki internal error.

Original exception: [WDIZMQpEE4AAAEoeUxYAAAAG] /wiki/?foobar DBQueryError from line 1054 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SHOW SLAVE STATUS
Function: DatabaseMysqlBase::getLagFromSlaveStatus
Error: 1227 Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation (10.68.18.35)

Backtrace:
#0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(912): Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(660): Database->query(string, string)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(645): DatabaseMysqlBase->getLagFromSlaveStatus()
#3 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(167): DatabaseMysqlBase->getLag()
#4 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(65): LoadMonitor->getServerStates(array, boolean)
#5 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(302): LoadMonitor->scaleLoads(array, boolean)
#6 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(544): LoadBalancer->getReaderIndex(boolean, boolean)
#7 /srv/mediawiki/php-master/includes/GlobalFunctions.php(3077): LoadBalancer->getConnection(integer, array, boolean)
#8 /srv/mediawiki/php-master/includes/cache/MessageCache.php(454): wfGetDB(integer)
#9 /srv/mediawiki/php-master/includes/cache/MessageCache.php(413): MessageCache->loadFromDB(string, NULL)
#10 /srv/mediawiki/php-master/includes/cache/MessageCache.php(335): MessageCache->loadFromDBWithLock(string, array, NULL)
#11 /srv/mediawiki/php-master/includes/cache/MessageCache.php(950): MessageCache->load(string)
#12 /srv/mediawiki/php-master/includes/cache/MessageCache.php(880): MessageCache->getMsgFromNamespace(string, string)
#13 /srv/mediawiki/php-master/includes/cache/MessageCache.php(847): MessageCache->getMessageForLang(Language, string, boolean, array)
#14 /srv/mediawiki/php-master/includes/cache/MessageCache.php(788): MessageCache->getMessageFromFallbackChain(Language, string, boolean)
#15 /srv/mediawiki/php-master/includes/Message.php(1196): MessageCache->get(string, boolean, Language)
#16 /srv/mediawiki/php-master/includes/Message.php(816): Message->fetchMessage()
#17 /srv/mediawiki/php-master/includes/Message.php(908): Message->toString(string)
#18 /srv/mediawiki/php-master/includes/Title.php(557): Message->text()
#19 /srv/mediawiki/php-master/includes/MediaWiki.php(116): Title::newMainPage()
#20 /srv/mediawiki/php-master/includes/MediaWiki.php(136): MediaWiki->parseTitle()
#21 /srv/mediawiki/php-master/includes/MediaWiki.php(766): MediaWiki->getTitle()
#22 /srv/mediawiki/php-master/includes/MediaWiki.php(512): MediaWiki->main()
#23 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#24 /srv/mediawiki/w/index.php(3): include(string)
#25 {main}

Exception caught inside exception handler: [WDIZMQpEE4AAAEoeUxYAAAAG] /wiki/?foobar DBQueryError from line 1054 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SHOW SLAVE STATUS
Function: DatabaseMysqlBase::getLagFromSlaveStatus
Error: 1227 Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation (10.68.18.35)

Backtrace:
#0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(912): Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(660): Database->query(string, string)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(645): DatabaseMysqlBase->getLagFromSlaveStatus()
#3 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(167): DatabaseMysqlBase->getLag()
#4 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(65): LoadMonitor->getServerStates(array, boolean)
#5 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(302): LoadMonitor->scaleLoads(array, boolean)
#6 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(544): LoadBalancer->getReaderIndex(boolean, boolean)
#7 /srv/mediawiki/php-master/includes/GlobalFunctions.php(3077): LoadBalancer->getConnection(integer, array, boolean)
#8 /srv/mediawiki/php-master/includes/cache/MessageCache.php(454): wfGetDB(integer)
#9 /srv/mediawiki/php-master/includes/cache/MessageCache.php(413): MessageCache->loadFromDB(string, NULL)
#10 /srv/mediawiki/php-master/includes/cache/MessageCache.php(335): MessageCache->loadFromDBWithLock(string, array, NULL)
#11 /srv/mediawiki/php-master/includes/cache/MessageCache.php(950): MessageCache->load(string)
#12 /srv/mediawiki/php-master/includes/cache/MessageCache.php(880): MessageCache->getMsgFromNamespace(string, string)
#13 /srv/mediawiki/php-master/includes/cache/MessageCache.php(847): MessageCache->getMessageForLang(Language, string, boolean, array)
#14 /srv/mediawiki/php-master/includes/cache/MessageCache.php(788): MessageCache->getMessageFromFallbackChain(Language, string, boolean)
#15 /srv/mediawiki/php-master/includes/Message.php(1196): MessageCache->get(string, boolean, Language)
#16 /srv/mediawiki/php-master/includes/Message.php(816): Message->fetchMessage()
#17 /srv/mediawiki/php-master/includes/Message.php(908): Message->toString(string)
#18 /srv/mediawiki/php-master/includes/OutputPage.php(906): Message->text()
#19 /srv/mediawiki/php-master/includes/OutputPage.php(954): OutputPage->setHTMLTitle(Message)
#20 /srv/mediawiki/php-master/includes/OutputPage.php(2372): OutputPage->setPageTitle(string)
#21 /srv/mediawiki/php-master/includes/exception/MWExceptionRenderer.php(162): OutputPage->prepareErrorPage(string)
#22 /srv/mediawiki/php-master/includes/exception/MWExceptionRenderer.php(49): MWExceptionRenderer::reportHTML(DBQueryError)
#23 /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php(71): MWExceptionRenderer::output(DBQueryError, integer)
#24 /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php(137): MWExceptionHandler::report(DBQueryError)
#25 /srv/mediawiki/php-master/includes/MediaWiki.php(540): MWExceptionHandler::handleException(DBQueryError)
#26 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#27 /srv/mediawiki/w/index.php(3): include(string)
#28 {main}
Krenair closed this task as Resolved.Nov 20 2016, 10:01 PM
Krenair claimed this task.
Krenair subscribed.

This was probably due to my messing with the grants etc. - beta used to use wikiadmin for *everything*, after fixing up the grants I made it use wikiadmin for CLI stuff and wikiuser for web requests, but I missed the grant needed for wikiuser to run 'show slave status'. Fixed by running GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'wikiuser'@'10.%'; as mysql root on deployment-db03

Reedy mentioned this in T151171: hhvm error logs on beta showing no errors when cluster down.Nov 20 2016, 10:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL