Maniphest T207825

Access to deployment-redis3-changeprop broken
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krenair
	Oct 24 2018, 6:51 AM

Description

This appears to be a new deployment-prep instance which has showed up on my tool for finding instances with broken puppet. However:

alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (production)$ ssh deployment-redis3-changeprop
krenair@deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs: Permission denied (publickey).
alex@alex-laptop:~/Development/Wikimedia/Operations-Puppet (production)$ ssh root@deployment-redis3-changeprop
root@deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs: Permission denied (publickey).

Created

2018-10-22T16:05:49Z

Created by

Effie Mouzeli

Event Timeline

Krenair created this task.Oct 24 2018, 6:51 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 24 2018, 6:51 AM

krenair@deployment-cumin:~$ sudo cumin 'name:deployment-redis3-changeprop' id
1 hosts will be targeted:
deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs
Confirm to continue [y/n]? y
----- OUTPUT of 'id' -----                                                                                                                                                                                  
Permission denied (publickey).                                                                                                                                                                              
================                                                                                                                                                                                            
PASS:  |                                                                                                                                                               |   0% (0/1) [00:00<?, ?hosts/s]     
FAIL:  |███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (1/1) [00:00<00:00,  8.14hosts/s]     
100.0% (1/1) of nodes failed to execute command 'id': deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs
0.0% (0/1) success ratio (< 100.0% threshold) for command: 'id'. Aborting.
0.0% (0/1) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

Yeah looks like this instance was created with a puppet role before it had successfully run puppet for the first time, which explains why my root key didn't work: https://horizon.wikimedia.org/project/instances/5f402f6d-3f79-4a15-9250-de0f1e6814ce/console
Evaluation Error: Error while evaluating a Function Call, undefined local variable or method `port' for Scope(Class[Profile::Redis::Multidc]):Puppet::Parser::Scope at /etc/puppet/modules/profile/manifests/redis/multidc.pp:13:18 on node deployment-redis3-changeprop.deployment-prep.eqiad.wmflabs#033[0m

It actually doesn't have anything individual, it gets that role from the prefix deployment-redis*

Mentioned in SAL (#wikimedia-releng) [2018-10-24T06:59:43Z] <Krenair> T207825 moved role::jobqueue_redis::master role from deployment-redis prefix to deployment-redis0[56]

Mentioned in SAL (#wikimedia-releng) [2018-10-24T07:00:56Z] <Krenair> T207825 replacing deployment-redis3-changeprop with deployment-redis3-changeprop02 (jessie m1.small)

Krenair claimed this task.Oct 24 2018, 7:04 AM

Created a new instance called deployment-redis3-changeprop02, sudo -i, cd /var/lib/puppet; mv ssl ssl_old; rm /usr/local/share/ca-certificates/Puppet_Internal_CA.crt; nano /usr/local/share/ca-certificates/Puppet_Internal_CA.crt, paste in file from another deployment-prep host, update-ca-certificates --fresh; puppet agent -tv, sign cert on puppetmaster, puppet agent -tv
Shut down broken instance - @jijiki I assume that's safe to delete as it never would've become accessible?

Mentioned in SAL (#wikimedia-releng) [2018-10-24T07:27:35Z] <Krenair> T207825 reapplied role::jobqueue_redis::master to deployment-redis prefix

This is WIP, please refrain from using it.

snip

jijiki closed this task as Resolved.Oct 25 2018, 4:34 PM

Access to deployment-redis3-changeprop brokenClosed, ResolvedPublicActions

Description

Event Timeline

Access to deployment-redis3-changeprop broken
Closed, ResolvedPublic
Actions