Page MenuHomePhabricator

tools-redis is down
Closed, ResolvedPublic

Description

Traceback (most recent call last):

File "../toredis.py", line 41, in <module>
  method(channel, data)
File "/usr/lib/python2.7/dist-packages/redis/client.py", line 1063, in publish
  return self.execute_command('PUBLISH', channel, message)
File "/usr/lib/python2.7/dist-packages/redis/client.py", line 256, in execute_command
  connection.send_command(*args)
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 197, in send_command
  self.send_packed_command(self.pack_command(*args))
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 180, in send_packed_command
  self.connect()
File "/usr/lib/python2.7/dist-packages/redis/connection.py", line 127, in connect
  raise ConnectionError(self._error_message(e))

redis.exceptions.ConnectionError: Error 111 connecting tools-redis:6379. Connection refused.

Also reported by legoktm in the SAL @ https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL

08:59 legoktm: grrrit-wm: 2014-04-20T08:28:15.889Z - error: Caught error in redisClient.brpop: Redis connection to tools-redis:6379 failed - connect ECONNREFUSED


Version: unspecified
Severity: normal

Details

Reference
bz64150

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:20 AM
bzimport added a project: Toolforge.
bzimport set Reference to bz64150.

The /var partition was full due to Redis using 1.6 GByte in /var/lib/redis.

After verifying that using role::labs::lvm::mnt wouldn't cause any existing partition to be overwritten, I tried to use it with "$lvm_mount_point = /var/lib/redis", but in modules/labs_lvm/manifests/volume.pp this defines a file resource with "ensure => directory" that collides with a file resource with more detailed parameters in modules/redis/manifests/init.pp. I /think/ that volume.pp's file resource should be replaced with a "resource collector" ("User <| title == 'luke' |>").

Once that is done, we should consider defining the partitions in tools::redis so they don't get lost and can be smaller than the whole 68 GBytes that are allocated at the moment.

I should have added what /worked/ :-): Setting $lvm_mount_point to "/var/lib", running Puppet (leaves an empty /var/lib), "umount /var/lib && mv -i /var/lib /var/lib2 && puppetd -tv && mv -i /var/lib2/* /var/lib/ && rmdir /var/lib2" and a reboot for good measure.