Page MenuHomePhabricator
Create Task
Maniphest T124388

Can not create account at beta cluster: "Unable to connect to redis server deployment-redis01.eqiad.wmflabs."
Closed, DuplicatePublic
Actions

Description

Internal error:
[06b26103] /w/index.php?title=Special:UserLogin&action=submitlogin&type=signup&returnto=Main+Page JobQueueConnectionError from line 753 of /srv/mediawiki/php-master/includes/jobqueue/JobQueueRedis.php: Unable to connect to redis server deployment-redis01.eqiad.wmflabs.

Backtrace:

#0 /srv/mediawiki/php-master/includes/jobqueue/JobQueueRedis.php(210): JobQueueRedis->getConnection()
#1 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(324): JobQueueRedis->doBatchPush(array, integer)
#2 /srv/mediawiki/php-master/includes/jobqueue/JobQueue.php(296): JobQueue->batchPush(array, integer)
#3 /srv/mediawiki/php-master/includes/jobqueue/JobQueueGroup.php(129): JobQueue->push(array)
#4 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthPlugin.php(390): JobQueueGroup->push(array)
#5 /srv/mediawiki/php-master/extensions/CentralAuth/includes/CentralAuthPlugin.php(309): CentralAuthPlugin->autoCreateAccounts(CentralAuthUser)
#6 /srv/mediawiki/php-master/includes/specials/SpecialUserlogin.php(665): CentralAuthPlugin->addUser(User, string, string, string)
#7 /srv/mediawiki/php-master/includes/specials/SpecialUserlogin.php(421): LoginForm->addNewAccountInternal()
#8 /srv/mediawiki/php-master/includes/specials/SpecialUserlogin.php(353): LoginForm->addNewAccount()
#9 /srv/mediawiki/php-master/includes/specialpage/SpecialPage.php(384): LoginForm->execute(NULL)
#10 /srv/mediawiki/php-master/includes/specialpage/SpecialPageFactory.php(564): SpecialPage->run(NULL)
#11 /srv/mediawiki/php-master/includes/MediaWiki.php(280): SpecialPageFactory::executePath(Title, RequestContext)
#12 /srv/mediawiki/php-master/includes/MediaWiki.php(745): MediaWiki->performRequest()
#13 /srv/mediawiki/php-master/includes/MediaWiki.php(517): MediaWiki->main()
#14 /srv/mediawiki/php-master/index.php(43): MediaWiki->run()
#15 /srv/mediawiki/w/index.php(3): include(string)
#16 {main}

Related Objects

Event Timeline

Bugreporter created this task.Jan 22 2016, 7:06 AM
Bugreporter raised the priority of this task from to Unbreak Now!.
Bugreporter updated the task description. (Show Details)
Bugreporter added a project: Beta-Cluster-Infrastructure.
Bugreporter subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 22 2016, 7:06 AM
Aklapper renamed this task from Can not create account at beta cluster to Can not create account at beta cluster: "Unable to connect to redis server deployment-redis01.eqiad.wmflabs.".Jan 22 2016, 8:12 AM
Aklapper set Security to None.
Luke081515 awarded a token.Jan 22 2016, 2:09 PM
hashar subscribed.Jan 22 2016, 9:54 PM

Thu Jan 21 19:24:12 2016] init: redis-instance-tcp_6380 main process (2135) terminated with status 1
[Thu Jan 21 19:24:12 2016] init: redis-instance-tcp_6380 main process ended, respawning
[Thu Jan 21 19:24:13 2016] init: redis-instance-tcp_6381 main process (2162) terminated with status 1
[Thu Jan 21 19:24:13 2016] init: redis-instance-tcp_6381 main process ended, respawning
[Thu Jan 21 19:24:14 2016] init: redis-instance-tcp_6379 main process (2174) terminated with status 1
[Thu Jan 21 19:24:14 2016] init: redis-instance-tcp_6379 main process ended, respawning
[Thu Jan 21 19:24:20 2016] init: redis-instance-tcp_6378 main process (2410) terminated with status 1
[Thu Jan 21 19:24:20 2016] init: redis-instance-tcp_6378 main process ended, respawning

hashar added a comment.Jan 22 2016, 9:55 PM

And it is dead:

root@deployment-redis01:/var/log# /etc/init.d/redis-server status
redis-server is not running
root@deployment-redis01:/var/log# ps -A|grep redis
root@deployment-redis01:/var/log#
hashar added a comment.Jan 22 2016, 9:57 PM

From /var/log/upstart/redis-instance* files which are all at Jan 21 19:24:

*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 549
>>> 'latency-monitor-threshold 100'
Bad directive or wrong number of arguments
hashar added a comment.Jan 22 2016, 9:59 PM

Well puppet patch from Dec 29th introduced the 'latency-monitor-threshold' config https://gerrit.wikimedia.org/r/#/c/261303/ ... That is quite old.

hashar closed this task as Resolved.Jan 22 2016, 10:08 PM
hashar claimed this task.

So the redis-server errors above are unrelated maybe.

The services are running:

# initctl list|grep redis
redis-instance-tcp_6379 start/running
redis-instance-tcp_6378 start/running
redis-instance-tcp_6380 start/running
redis-instance-tcp_6381 start/running

They got killed at Jan 21 19:24:11 for some reason.

MGChecker awarded a token.Jan 22 2016, 10:40 PM
MGChecker reopened this task as Open.Jan 22 2016, 10:45 PM
MGChecker subscribed.

If I try to delete something right now (and earlier today too) I get a similar error wit the same backtrace, only the first line is different:

[88cf8895] /w/index.php?title=EN-Tests&action=delete JobQueueConnectionError from line 753 of /srv/mediawiki/php-master/includes/jobqueue/JobQueueRedis.php: Unable to connect to redis server deployment-redis01.eqiad.wmflabs.
Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJan 22 2016, 10:45 PM
greg subscribed.Jan 25 2016, 7:23 PM

@MGChecker: please try again, we just restarted Redis in Beta Cluster again, see: T124677#1962666

greg added a subscriber: • mmodell.Jan 25 2016, 7:29 PM
In T124388#1957339, @hashar wrote:

Well puppet patch from Dec 29th introduced the 'latency-monitor-threshold' config https://gerrit.wikimedia.org/r/#/c/261303/ ... That is quite old.

That was actually it, see rOPUP1c16154c6ee76433722d0f01462db57727d37b64 (that same commit, but with @mmodell's commentary).

I'm merging this task into the other one where Ori is assigned to fix the puppet config issue.

greg closed this task as a duplicate of T124677: Redis latency monitor config change broke Redis in Beta Cluster.Jan 25 2016, 7:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL