cumin broken on integration project
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	hashar
	Sep 30 2019, 11:28 AM

Description

cumin fails from integration-cumin:

===== NODE GROUP =====                                                                                                                                                                    
(15) integration-agent-docker-[1001-1003,1005-1007,1009-1016].integration.eqiad.wmflabs,integration-agent-puppet-docker-1001.integration.eqiad.wmflabs                                    
----- OUTPUT of 'hostname' -----                                                                                                                                                          
Permission denied (publickey).

On the affected hosts, the ssh root key that is allowed comes from WMCS with some restrictions:

/etc/ssh/userkeys/root.d/cumin

# Cumin Masters. TODO: use 'restrict' once available across the fleet (> jessie)
from="172.16.4.46,172.16.6.133",no-agent-forwarding,no-port-forwarding,no-x11-forwarding,no-user-rc ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINeHalEAaoGG0AAO13yk+ttCyoAGUunnS20K6Wa6yz3r cumin-openstack-master

File dated roughly 2019 Sep 26 10:51

On a host that had puppet broken for unrelated reasons:

# Cumin Masters. TODO: use 'restrict' once available across the fleet (> jessie)
from="172.16.4.46,172.16.6.133",no-agent-forwarding,no-port-forwarding,no-x11-forwarding,no-user-rc ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINeHalEAaoGG0AAO13yk+ttCyoAGUunnS20K6Wa6yz3r cumin-openstack-master

from="172.16.1.103",no-agent-forwarding,no-port-forwarding,no-x11-forwarding,no-user-rc ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPPfdabE1Fej0X86QgjY72LXvA3Wawrg0ZcDL0PF56/A root@integration-cumin

The last Puppet run was at Thu Sep 26 10:30:01 UTC 2019 (5823 minutes ago).

File dated Sep 23 18:55

So somehow the cumin project configuration is not applied.

Event Timeline

hashar created this task.Sep 30 2019, 11:28 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 30 2019, 11:28 AM

hashar updated the task description. (Show Details)Sep 30 2019, 11:35 AM

On https://horizon.wikimedia.org/project/puppet/ :

profile::openstack::eqiad1::cumin::project_masters:
- 172.16.1.103
profile::openstack::eqiad1::cumin::project_pub_key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPPfdabE1Fej0X86QgjY72LXvA3Wawrg0ZcDL0PF56/A
  root@integration-cumin

Apparently the puppet project configuration went missing at some point :-\

Mentioned in SAL (#wikimedia-releng) [2019-09-30T12:49:49Z] <hashar> Fixed cumin on integration project # T234203

cumin broken on integration projectClosed, ResolvedPublicActions

Description

Event Timeline

cumin broken on integration project
Closed, ResolvedPublic
Actions