integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth]
Closed, ResolvedPublic

Description

From integration-cumin.integration.eqiad.wmflabs , cumin now fails to ssh to target instances :(

hashar@integration-cumin:~$ sudo cumin --force 'name:docker' 'hostname'
19 hosts will be targeted:
integration-slave-docker-[1001-1017,1020-1021].integration.eqiad.wmflabs
FORCE mode enabled, continuing without confirmation
===== NODE GROUP =====                                                                                                                                                                           
(19) integration-slave-docker-[1001-1017,1020-1021].integration.eqiad.wmflabs                                                                                                                    
----- OUTPUT of 'hostname' -----                                                                                                                                                                 
Permission denied (publickey).                                                                                                                                                                   
================                                                                                                                                                                                 
PASS:  |                                             |   0% (0/19) [00:00<?, ?hosts/s]     
FAIL:  |█████████████████████████████████████████████| 100% (19/19) [00:00<00:00, 31.94hosts/s]     
100.0% (19/19) of nodes failed to execute command 'hostname': integration-slave-docker-[1001-1017,1020-1021].integration.eqiad.wmflabs
0.0% (0/19) success ratio (< 100.0% threshold) for command: 'hostname'. Aborting.
0.0% (0/19) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

On one of the instances:

Jun 14 07:05:35 integration-slave-docker-1004 sshd[30650]: Connection from 10.68.18.238 port 56586 on 10.68.16.233 port 22
Jun 14 07:05:35 integration-slave-docker-1004 sshd[30650]: Connection closed by 10.68.18.238 [preauth]

I guess the key is not recognized somehow.

hashar created this task.Jun 14 2018, 7:08 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 14 2018, 7:08 AM
$ keyholder status
keyholder-agent: active
- The agent has no identities.
keyholder-proxy: active
- The agent has no identities.
hashar closed this task as Resolved.Jun 14 2018, 7:12 AM

Found the passphrase from integration-puppetmaster01 in the labs/private repo

# keyholder arm
Enter passphrase for /etc/keyholder.d/cumin_openstack_integration_master: 
Identity added: /etc/keyholder.d/cumin_openstack_integration_master (/etc/keyholder.d/cumin_openstack_integration_master)
root@integration-cumin:~# keyholder status
keyholder-agent: active
- 256 06:36:d8:17:14:ac:73:73:3b:71:ea:bf:1f:59:e1:23 /etc/keyholder.d/cumin_openstack_integration_master (ED25519)
keyholder-proxy: active
- 256 06:36:d8:17:14:ac:73:73:3b:71:ea:bf:1f:59:e1:23 /etc/keyholder.d/cumin_openstack_integration_master (ED25519)

Mentioned in SAL (#wikimedia-releng) [2018-06-14T07:40:16Z] <hashar> Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01| T197207

Vvjjkkii renamed this task from integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth] to c1aaaaaaaa.Jul 1 2018, 1:04 AM
Vvjjkkii reopened this task as Open.
Vvjjkkii triaged this task as High priority.
Vvjjkkii removed hashar as the assignee of this task.
Vvjjkkii updated the task description. (Show Details)
Vvjjkkii removed a subscriber: Aklapper.
CommunityTechBot raised the priority of this task from High to Needs Triage.
CommunityTechBot closed this task as Resolved.
CommunityTechBot renamed this task from c1aaaaaaaa to integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth].
CommunityTechBot added a subscriber: Aklapper.