Maniphest T197207

integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth]
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Jun 14 2018, 7:08 AM

Description

From integration-cumin.integration.eqiad.wmflabs , cumin now fails to ssh to target instances :(

hashar@integration-cumin:~$ sudo cumin --force 'name:docker' 'hostname'
19 hosts will be targeted:
integration-slave-docker-[1001-1017,1020-1021].integration.eqiad.wmflabs
FORCE mode enabled, continuing without confirmation
===== NODE GROUP =====                                                                                                                                                                           
(19) integration-slave-docker-[1001-1017,1020-1021].integration.eqiad.wmflabs                                                                                                                    
----- OUTPUT of 'hostname' -----                                                                                                                                                                 
Permission denied (publickey).                                                                                                                                                                   
================                                                                                                                                                                                 
PASS:  |                                             |   0% (0/19) [00:00<?, ?hosts/s]     
FAIL:  |█████████████████████████████████████████████| 100% (19/19) [00:00<00:00, 31.94hosts/s]     
100.0% (19/19) of nodes failed to execute command 'hostname': integration-slave-docker-[1001-1017,1020-1021].integration.eqiad.wmflabs
0.0% (0/19) success ratio (< 100.0% threshold) for command: 'hostname'. Aborting.
0.0% (0/19) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.

On one of the instances:

Jun 14 07:05:35 integration-slave-docker-1004 sshd[30650]: Connection from 10.68.18.238 port 56586 on 10.68.16.233 port 22
Jun 14 07:05:35 integration-slave-docker-1004 sshd[30650]: Connection closed by 10.68.18.238 [preauth]

I guess the key is not recognized somehow.

Event Timeline

hashar created this task.Jun 14 2018, 7:08 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 14 2018, 7:08 AM

$ keyholder status
keyholder-agent: active
- The agent has no identities.
keyholder-proxy: active
- The agent has no identities.

hashar claimed this task.Jun 14 2018, 7:09 AM

hashar added a project: Release-Engineering-Team (Kanban).

Found the passphrase from integration-puppetmaster01 in the labs/private repo

# keyholder arm
Enter passphrase for /etc/keyholder.d/cumin_openstack_integration_master: 
Identity added: /etc/keyholder.d/cumin_openstack_integration_master (/etc/keyholder.d/cumin_openstack_integration_master)
root@integration-cumin:~# keyholder status
keyholder-agent: active
- 256 06:36:d8:17:14:ac:73:73:3b:71:ea:bf:1f:59:e1:23 /etc/keyholder.d/cumin_openstack_integration_master (ED25519)
keyholder-proxy: active
- 256 06:36:d8:17:14:ac:73:73:3b:71:ea:bf:1f:59:e1:23 /etc/keyholder.d/cumin_openstack_integration_master (ED25519)

Mentioned in SAL (#wikimedia-releng) [2018-06-14T07:40:16Z] <hashar> Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01| T197207

• Vvjjkkii renamed this task from integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth] to c1aaaaaaaa.Jul 1 2018, 1:04 AM

• Vvjjkkii reopened this task as Open.

• Vvjjkkii removed hashar as the assignee of this task.

• Vvjjkkii triaged this task as High priority.

• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).

• Vvjjkkii updated the task description. (Show Details)

• Vvjjkkii removed a subscriber: Aklapper.

CommunityTechBot updated the task description. (Show Details)Jul 2 2018, 7:00 AM

CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.

CommunityTechBot renamed this task from c1aaaaaaaa to integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth].Jul 3 2018, 12:15 AM

CommunityTechBot closed this task as Resolved.

CommunityTechBot assigned this task to hashar.

CommunityTechBot raised the priority of this task from High to Needs Triage.

CommunityTechBot added a subscriber: Aklapper.

Aklapper edited projects, added Cumin; removed SRE-tools.May 10 2023, 11:15 AM

integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth]Closed, ResolvedPublicActions

Description

Event Timeline

integration cumin can't ssh to instances: Connection closed by integration-cumin [preauth]
Closed, ResolvedPublic
Actions