Page MenuHomePhabricator
Create Task
Maniphest T94866

/etc/ssh/userkeys/ubuntu or /etc/ssh/userkeys/admin notices for every puppet run on labs instances
Closed, ResolvedPublic
Actions

Description

Puppet is throwing a ton of notices these days:

Notice: /Stage[main]/Role::Labs::Instance/Notify[instanceproject: puppet]/message: defined 'message' as 'instanceproject: puppet'
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys ]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys ]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public/keys]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public/keys]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public/keys/ubuntu]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public/keys/ubuntu]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public/keys/ubuntu/.ssh]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/ubuntu/.ssh/authorized_keys /public/keys/ubuntu/.ssh]/ensure: removed

Fixing that is trivial, but I'm dumping this on Yuvi because he might have a fix in the works already.

Related / dupes:
T59752: fetch ssh authorized_keys via LDAP
{T81099}
T85814: stray files created in /etc/ssh/userkeys

Details

SubjectRepoBranchLines +/-
Override default cloud.cfg with a custom file.operations/puppetproduction+115 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Apr 2 2015, 5:45 PM
Andrew assigned this task to yuvipanda.
Andrew raised the priority of this task from to Medium.
Andrew updated the task description. (Show Details)
Andrew added a project: Cloud-Services.
Andrew subscribed.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 2 2015, 5:45 PM
Krinkle renamed this task from /etc/ssh/userkeys/ubuntu puppet notices on labs instances to /etc/ssh/userkeys/ubuntu notices for every puppet run on labs instances.Apr 2 2015, 9:22 PM
Krinkle set Security to None.
Krinkle subscribed.
scfc merged a task: T94996: Jessie labs instance puppet run attempt to remove non empty /etc/ssh/userkeys/admin/.ssh.Apr 3 2015, 7:56 PM
scfc added a subscriber: hashar.
scfc added subscribers: scfc, faidon, yuvipanda.Apr 3 2015, 7:59 PM

These seem to be related to https://gerrit.wikimedia.org/r/#/c/183814/ & Co. where I thought that the issue was just transient for the migration from the old system to the new.

Note that then some of the affected directories had three links.

Krinkle added a parent task: T94916: Re-create ci slaves (April 2015).Apr 5 2015, 5:35 PM
faidon added a comment.Apr 6 2015, 5:47 AM

I'd rather not use force => true in puppet for this, if possible. This can be done massively with salt, no?

I still haven't figured out where these files came from, though; maybe they are included in the base image? If so, we'd need to update that as well.

scfc added a comment.Apr 7 2015, 10:27 AM

I only noticed T85814 now; trying to draw a line that task's scope is about finding and removing the cause of the files' existence in freshly provisioned instances, this task's scope is about removing the files (once) in already existing instances (for which using salt probably makes sense).

yuvipanda added a comment.Apr 16 2015, 8:24 PM

I think these keys are in the base image and should be removed from there.

yuvipanda removed yuvipanda as the assignee of this task.Apr 16 2015, 8:26 PM

(and I don't have a fix atm)

akosiaris subscribed.Apr 16 2015, 8:29 PM

I noticed the same thing in a freshly created jessie image today.

Notice: /Stage[main]/Puppetmaster::Ssl/File[/var/lib/puppet/server/ssl]/group: group changed 'puppet' to 'root'
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys ]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys ]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public/keys]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public/keys]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public/keys/admin]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public/keys/admin]/ensure: removed
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public/keys/admin/.ssh]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/userkeys/admin/.ssh/authorized_keys /public/keys/admin/.ssh]/ensure: removed

A quick glance download, qemu convert and mount of the image showed no traces of that file in the image itself. A grep in /var/log/ shows

cloud-init.log:Apr 15 19:34:09 testakis [CLOUDINIT] util.py[DEBUG]: Writing to /etc/ssh/userkeys/admin/.ssh/authorized_keys /public/keys/admin/.ssh/authorized_keys - wb: [384] 0 bytes

could cloud init be the culrpit? I am not familiar with the software

hashar updated the task description. (Show Details)Apr 26 2015, 7:14 PM

Related / dupes:
T59752: fetch ssh authorized_keys via LDAP
{T81099}
T85814: stray files created in /etc/ssh/userkeys

akosiaris renamed this task from /etc/ssh/userkeys/ubuntu notices for every puppet run on labs instances to /etc/ssh/userkeys/ubuntu or /etc/ssh/userkeys/admin notices for every puppet run on labs instances.Apr 27 2015, 7:57 AM
faidon merged a task: T85814: stray files created in /etc/ssh/userkeys.May 28 2015, 5:09 PM
faidon added a subscriber: fgiunchedi.
faidon added a comment.Jun 10 2015, 1:18 PM

Ping?

yuvipanda added a comment.Jun 10 2015, 1:19 PM

Basically - live with it until we make new images, which will happen today (for T101916)

faidon added a comment.Jun 10 2015, 3:05 PM

Have we found why these were included in the base image in the old image and in the first place?

Andrew added a parent task: T102001: Build new labs base images.Jun 10 2015, 3:10 PM
Andrew added a comment.Jun 11 2015, 4:43 AM

I've verified that those files are not present in the base image before first boot. So I suspect that cloud-init is creating them -- I'll see about changing the config.

gerritbot subscribed.Jun 11 2015, 4:43 AM

Change 217455 had a related patch set uploaded (by Andrew Bogott):
Override default cloud.cfg with a custom file.

https://gerrit.wikimedia.org/r/217455

gerritbot added a project: Patch-For-Review.Jun 11 2015, 4:43 AM
gerritbot added a comment.Jun 11 2015, 3:34 PM

Change 217455 merged by Andrew Bogott:
Override default cloud.cfg with a custom file.

https://gerrit.wikimedia.org/r/217455

Andrew mentioned this in rOPUP038df98bf1c6: Override default cloud.cfg with a custom file..Jun 11 2015, 3:35 PM
Andrew closed this task as Resolved.Jun 13 2015, 4:35 PM
Andrew claimed this task.
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 6:52 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL