Set up puppet exported resources to collect ssh host keys for beta
Closed, ResolvedPublic

Description

If deployment-bastion had an up to date list of ssh host keys we would be able to remove the scap cherry-pick of I6a56b5ec7887737d1662a3cc9e0cd1a5c13ec95d that scap is using in beta to ignore the remote host key.

Mukunda took a stab at this in Iac547efa83cf059a1276b6e279c3ebd4c7224b2c but we pulled the patch when it had merge conflicts with an upstream change.

Details

Reference
bz70792
bzimport raised the priority of this task from to Needs Triage.
bzimport set Reference to bz70792.
bzimport added a subscriber: Unknown Object (MLST).
bd808 created this task.Sep 12 2014, 10:10 PM

gerritadmin wrote:

Change 161248 had a related patch set uploaded by Prtksxna:
button.less: Set min-width to 80px

https://gerrit.wikimedia.org/r/161248

Sorry! Typed the wrong bug number.

greg triaged this task as Normal priority.
Prtksxna removed a subscriber: Prtksxna.Nov 26 2014, 4:25 AM

I no longer see this cherry-picked on beta, and jenkins seems to be just as happy?

bd808 added a comment.Feb 2 2015, 5:02 PM

I no longer see this cherry-picked on beta, and jenkins seems to be just as happy?

This is "fixed" for beta using a local hack to scap (https://gerrit.wikimedia.org/r/#/c/148112/) that ignores host keys.

Right. Perhaps we could manually set the keys in hiera or someplace?

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 20 2015, 2:20 PM
hashar updated the task description. (Show Details)Nov 9 2015, 1:05 PM
hashar set Security to None.

https://gerrit.wikimedia.org/r/#/c/148112/ is still cherry picked on deployment-bastion in /srv/deployment/scap/scap

Why does the approach production uses not work for us? I'm vaguely aware of exported resources only being available in production - where can I find more information about that?

We chatted about this on IRC and my impression is that the issue is mostly (?) that production puppetmasters (which have exported resources setup) are an entirely separate module to the labs per-project puppetmasters (which don't): T120159: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module

That's the underlying cause, yes - if they used the same setup, they'd have the same config which would include support for exported resources.

scfc added a comment.Apr 12 2016, 4:06 PM

The general issue with exported resources is that they require only friendlies on the same puppetmaster. In Labs (almost) anyone can be root on an instance and thus fake exported resources with the potential to disrupt or gain control of other instances (or other hosts under the control of the same puppetmaster).

With Beta-Cluster-Infrastructure, this situation might be different as it uses its own puppetmaster, but I don't know if by default self-hosted puppetmasters have exported resources enabled and if the setup is secure.

AFAIK everyone with access to deployment-prep has root on deployment-puppetmaster, which is the puppetmaster for all instances in the project
The problem is the labs self-hosted puppetmasters don't have exported resources enabled, they're an entirely separate module to the production puppetmasters

The self puppetmaster also tend to auto sign the client certificates. So in theory any labs instance could point to the beta puppet master and collect its resources..

So if you use resource collection and care about security (you could block access to puppetmaster from outside the project with security groups), you should turn off auto-signing. Tools runs a limited puppetmaster setup with autosigning turned off, for example.

Generating the keys on the puppetmaster and distributing them as regular files would be easy enough. No need for exported resources.

E.g. this: http://emeraldreverie.org/blog/2013/03/11/managing-ssh-host-keys-in-reliable-way/

Which is exactly the same technique that I've been experimenting with for keyholder ssh keys (works well now that I've got it all debugged)

Aklapper renamed this task from Setup puppet exported resources to collect ssh host keys for beta to Set up puppet exported resources to collect ssh host keys for beta.Apr 30 2016, 2:35 PM
Krenair claimed this task.Jan 21 2017, 11:21 PM
Krenair added a comment.EditedJan 21 2017, 11:28 PM

I have a commit on -puppetmaster02 that does this, and it seems to mostly work. It seems to only include ECDSA keys (preferred over RSA, ED25519 not possible because prod still has some precise hosts, and trusty's puppet version doesn't allow for them, DSA unused), but it excludes hosts that no longer exist, and it has the FQDN, instance name, IPv4 address, and 'undef' (IPv6 I think, which of course we don't use in labs) on each line.

scfc added a comment.Jan 22 2017, 12:01 AM

See also T153163, subtasks and associated commits (plus two or three that I have yet to publish, polishing code isn't fun :-)).

Change 333471 had a related patch set uploaded (by Alex Monk):
Allow use of PuppetDB in labs for sshknowngen

https://gerrit.wikimedia.org/r/333471

Change 333472 had a related patch set uploaded (by Alex Monk):
ssh: Don't add IPv6 address as an alias in exported resource if it's undefined

https://gerrit.wikimedia.org/r/333472

Change 333473 had a related patch set uploaded (by Alex Monk):
puppetdb: Allow tuning.conf to have a different shared_buffers value

https://gerrit.wikimedia.org/r/333473

Change 333473 merged by Alexandros Kosiaris:
puppetdb: Allow tuning.conf to have a different shared_buffers value

https://gerrit.wikimedia.org/r/333473

Change 333472 merged by Filippo Giunchedi:
ssh: Don't add IPv6 address as an alias in exported resource if it's undefined

https://gerrit.wikimedia.org/r/333472

Change 333471 merged by Alexandros Kosiaris:
[operations/puppet@production] Allow use of PuppetDB in labs for ssh_known_hosts

https://gerrit.wikimedia.org/r/333471

It's working but it's very loud - I've made https://gerrit.wikimedia.org/r/#/c/436624/ to deal with that
Also probably isn't handling deployment-snapshot01 as that uses deployment-dumps-puppetmaster instead - @ArielGlenn?

Krenair added a comment.EditedJun 3 2018, 4:47 PM

Also probably isn't handling deployment-snapshot01 as that uses deployment-dumps-puppetmaster instead - @ArielGlenn?

I've fixed this up a bit. It has a new (stretch) puppetmaster of its own that talks to Puppet DB and uses the normal deployment-puppetmaster03 CA (had to generate the cert for the client on -puppetmaster03 and copy the files to the correct place on -snapshot01)
So -snapshot01 shares ssh known hosts with other hosts and vice versa now
Still need to tidy up the old puppetmaster and deal with the extra class I added to -dumps-puppetmaster02

Mentioned in SAL (#wikimedia-releng) [2018-07-10T18:31:31Z] <Krenair> deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in T72792)

cleaned that up today per
<apergos> Krenair: I finally got a chance to note the config info for the old dumps puppetmaster in deployment-prep, you can kill it. I should have everything in my notes now
<apergos> thanks again

Just waiting on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/435631/ to merge now, should probably mention this task in it

Change 435631 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Allow PuppetDB use on standalone puppetmasters

https://gerrit.wikimedia.org/r/435631

hashar removed a subscriber: hashar.

Change 435631 merged by Bstorm:
[operations/puppet@production] Allow PuppetDB use on standalone puppetmasters

https://gerrit.wikimedia.org/r/435631