Page MenuHomePhabricator

The Beta-Cluster is down
Closed, ResolvedPublic

Description

https://en.wikipedia.beta.wmflabs.org/wiki/

MediaWiki internal error.

Exception caught inside exception handler.

Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 20 2016, 5:40 PM
Luke081515 triaged this task as Unbreak Now! priority.Nov 20 2016, 5:40 PM
Restricted Application added subscribers: Jay8g, TerraCodes. · View Herald TranscriptNov 20 2016, 5:40 PM

shinken-wm reported problem first happening at 12:59 pm utc + 0 time.

<shinken-wm> PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1164 bytes in 0.072 second response time
<shinken-wm> PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1164 bytes in 0.072 second response time
<shinken-wm> PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 728 bytes in 0.075 second response time

deployment & de show a different message, zh shows the same as at en.

Jogo.obb added a subscriber: Jogo.obb.
Reedy added a subscriber: Reedy.Nov 20 2016, 9:45 PM

Various hacking to get this done

MediaWiki internal error.

Original exception: [WDIZMQpEE4AAAEoeUxYAAAAG] /wiki/?foobar DBQueryError from line 1054 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SHOW SLAVE STATUS
Function: DatabaseMysqlBase::getLagFromSlaveStatus
Error: 1227 Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation (10.68.18.35)

Backtrace:
#0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(912): Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(660): Database->query(string, string)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(645): DatabaseMysqlBase->getLagFromSlaveStatus()
#3 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(167): DatabaseMysqlBase->getLag()
#4 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(65): LoadMonitor->getServerStates(array, boolean)
#5 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(302): LoadMonitor->scaleLoads(array, boolean)
#6 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(544): LoadBalancer->getReaderIndex(boolean, boolean)
#7 /srv/mediawiki/php-master/includes/GlobalFunctions.php(3077): LoadBalancer->getConnection(integer, array, boolean)
#8 /srv/mediawiki/php-master/includes/cache/MessageCache.php(454): wfGetDB(integer)
#9 /srv/mediawiki/php-master/includes/cache/MessageCache.php(413): MessageCache->loadFromDB(string, NULL)
#10 /srv/mediawiki/php-master/includes/cache/MessageCache.php(335): MessageCache->loadFromDBWithLock(string, array, NULL)
#11 /srv/mediawiki/php-master/includes/cache/MessageCache.php(950): MessageCache->load(string)
#12 /srv/mediawiki/php-master/includes/cache/MessageCache.php(880): MessageCache->getMsgFromNamespace(string, string)
#13 /srv/mediawiki/php-master/includes/cache/MessageCache.php(847): MessageCache->getMessageForLang(Language, string, boolean, array)
#14 /srv/mediawiki/php-master/includes/cache/MessageCache.php(788): MessageCache->getMessageFromFallbackChain(Language, string, boolean)
#15 /srv/mediawiki/php-master/includes/Message.php(1196): MessageCache->get(string, boolean, Language)
#16 /srv/mediawiki/php-master/includes/Message.php(816): Message->fetchMessage()
#17 /srv/mediawiki/php-master/includes/Message.php(908): Message->toString(string)
#18 /srv/mediawiki/php-master/includes/Title.php(557): Message->text()
#19 /srv/mediawiki/php-master/includes/MediaWiki.php(116): Title::newMainPage()
#20 /srv/mediawiki/php-master/includes/MediaWiki.php(136): MediaWiki->parseTitle()
#21 /srv/mediawiki/php-master/includes/MediaWiki.php(766): MediaWiki->getTitle()
#22 /srv/mediawiki/php-master/includes/MediaWiki.php(512): MediaWiki->main()
#23 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#24 /srv/mediawiki/w/index.php(3): include(string)
#25 {main}

Exception caught inside exception handler: [WDIZMQpEE4AAAEoeUxYAAAAG] /wiki/?foobar DBQueryError from line 1054 of /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema updater after upgrading? 
Query: SHOW SLAVE STATUS
Function: DatabaseMysqlBase::getLagFromSlaveStatus
Error: 1227 Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation (10.68.18.35)

Backtrace:
#0 /srv/mediawiki/php-master/includes/libs/rdbms/database/Database.php(912): Database->reportQueryError(string, integer, string, string, boolean)
#1 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(660): Database->query(string, string)
#2 /srv/mediawiki/php-master/includes/libs/rdbms/database/DatabaseMysqlBase.php(645): DatabaseMysqlBase->getLagFromSlaveStatus()
#3 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(167): DatabaseMysqlBase->getLag()
#4 /srv/mediawiki/php-master/includes/libs/rdbms/loadmonitor/LoadMonitor.php(65): LoadMonitor->getServerStates(array, boolean)
#5 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(302): LoadMonitor->scaleLoads(array, boolean)
#6 /srv/mediawiki/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php(544): LoadBalancer->getReaderIndex(boolean, boolean)
#7 /srv/mediawiki/php-master/includes/GlobalFunctions.php(3077): LoadBalancer->getConnection(integer, array, boolean)
#8 /srv/mediawiki/php-master/includes/cache/MessageCache.php(454): wfGetDB(integer)
#9 /srv/mediawiki/php-master/includes/cache/MessageCache.php(413): MessageCache->loadFromDB(string, NULL)
#10 /srv/mediawiki/php-master/includes/cache/MessageCache.php(335): MessageCache->loadFromDBWithLock(string, array, NULL)
#11 /srv/mediawiki/php-master/includes/cache/MessageCache.php(950): MessageCache->load(string)
#12 /srv/mediawiki/php-master/includes/cache/MessageCache.php(880): MessageCache->getMsgFromNamespace(string, string)
#13 /srv/mediawiki/php-master/includes/cache/MessageCache.php(847): MessageCache->getMessageForLang(Language, string, boolean, array)
#14 /srv/mediawiki/php-master/includes/cache/MessageCache.php(788): MessageCache->getMessageFromFallbackChain(Language, string, boolean)
#15 /srv/mediawiki/php-master/includes/Message.php(1196): MessageCache->get(string, boolean, Language)
#16 /srv/mediawiki/php-master/includes/Message.php(816): Message->fetchMessage()
#17 /srv/mediawiki/php-master/includes/Message.php(908): Message->toString(string)
#18 /srv/mediawiki/php-master/includes/OutputPage.php(906): Message->text()
#19 /srv/mediawiki/php-master/includes/OutputPage.php(954): OutputPage->setHTMLTitle(Message)
#20 /srv/mediawiki/php-master/includes/OutputPage.php(2372): OutputPage->setPageTitle(string)
#21 /srv/mediawiki/php-master/includes/exception/MWExceptionRenderer.php(162): OutputPage->prepareErrorPage(string)
#22 /srv/mediawiki/php-master/includes/exception/MWExceptionRenderer.php(49): MWExceptionRenderer::reportHTML(DBQueryError)
#23 /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php(71): MWExceptionRenderer::output(DBQueryError, integer)
#24 /srv/mediawiki/php-master/includes/exception/MWExceptionHandler.php(137): MWExceptionHandler::report(DBQueryError)
#25 /srv/mediawiki/php-master/includes/MediaWiki.php(540): MWExceptionHandler::handleException(DBQueryError)
#26 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#27 /srv/mediawiki/w/index.php(3): include(string)
#28 {main}
Krenair closed this task as Resolved.Nov 20 2016, 10:01 PM
Krenair claimed this task.
Krenair added a subscriber: Krenair.

This was probably due to my messing with the grants etc. - beta used to use wikiadmin for *everything*, after fixing up the grants I made it use wikiadmin for CLI stuff and wikiuser for web requests, but I missed the grant needed for wikiuser to run 'show slave status'. Fixed by running GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'wikiuser'@'10.%'; as mysql root on deployment-db03