Page MenuHomePhabricator

Access to deployment-redis3-changeprop broken
Closed, ResolvedPublic


This appears to be a new deployment-prep instance which has showed up on my tool for finding instances with broken puppet. However:

alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (production)$ ssh deployment-redis3-changeprop
krenair@deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs: Permission denied (publickey).
alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (production)$ ssh root@deployment-redis3-changeprop
root@deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs: Permission denied (publickey).



Created by

Effie Mouzeli

Event Timeline

krenair@deployment-cumin:~$ sudo cumin 'name:deployment-redis3-changeprop' id
1 hosts will be targeted:
Confirm to continue [y/n]? y
----- OUTPUT of 'id' -----                                                                                                                                                                                  
Permission denied (publickey).                                                                                                                                                                              
PASS:  |                                                                                                                                                               |   0% (0/1) [00:00<?, ?hosts/s]     
FAIL:  |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  8.14hosts/s]     
100.0% (1/1) of nodes failed to execute command 'id': deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'id'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

Yeah looks like this instance was created with a puppet role before it had successfully run puppet for the first time, which explains why my root key didn't work:
Evaluation Error: Error while evaluating a Function Call, undefined local variable or method `port' for Scope(Class[Profile::Redis::Multidc]):Puppet::Parser::Scope at /etc/puppet/modules/profile/manifests/redis/multidc.pp:13:18 on node deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs#033[0m

It actually doesn't have anything individual, it gets that role from the prefix deployment-redis*

Mentioned in SAL (#wikimedia-releng) [2018-10-24T06:59:43Z] <Krenair> T207825 moved role::jobqueue_redis::master role from deployment-redis prefix to deployment-redis0[56]

Mentioned in SAL (#wikimedia-releng) [2018-10-24T07:00:56Z] <Krenair> T207825 replacing deployment-redis3-changeprop with deployment-redis3-changeprop02 (jessie m1.small)

Created a new instance called deployment-redis3-changeprop02, sudo -i, cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt, paste in file from another deployment-prep host, update-ca-certificates --fresh; puppet agent -tv, sign cert on puppetmaster, puppet agent -tv
Shut down broken instance - @jijiki I assume that's safe to delete as it never would've become accessible?

Mentioned in SAL (#wikimedia-releng) [2018-10-24T07:27:35Z] <Krenair> T207825 reapplied role::jobqueue_redis::master to deployment-redis prefix

This is WIP, please refrain from using it.