If deployment-bastion had an up to date list of ssh host keys we would be able to remove the scap cherry-pick of I6a56b5ec7887737d1662a3cc9e0cd1a5c13ec95d that scap is using in beta to ignore the remote host key.
Mukunda took a stab at this in Iac547efa83cf059a1276b6e279c3ebd4c7224b2c but we pulled the patch when it had merge conflicts with an upstream change.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Krenair | T72792 Set up puppet exported resources to collect ssh host keys for beta | |||
| Resolved | Andrew | T120159 Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module | |||
| Resolved | Andrew | T182810 Remove role::puppet::self and related support code | |||
| Declined | None | T128740 role::puppet::self requires a puppetmaster restart during apply | |||
| Resolved | None | T187622 role::puppet::self referenced in puppet_ssldir.rb | |||
| Resolved | Krenair | T153577 Make standalone puppetmasters optionally use PuppetDB | |||
| Resolved | scfc | T154104 role::puppetmaster::puppetdb depends on Ganglia and cannot be used in Labs |
gerritadmin wrote:
Change 161248 had a related patch set uploaded by Prtksxna:
button.less: Set min-width to 80px
This is "fixed" for beta using a local hack to scap (https://gerrit.wikimedia.org/r/#/c/148112/) that ignores host keys.
https://gerrit.wikimedia.org/r/#/c/148112/ is still cherry picked on deployment-bastion in /srv/deployment/scap/scap
Why does the approach production uses not work for us? I'm vaguely aware of exported resources only being available in production - where can I find more information about that?
We chatted about this on IRC and my impression is that the issue is mostly (?) that production puppetmasters (which have exported resources setup) are an entirely separate module to the labs per-project puppetmasters (which don't): T120159: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module
That's the underlying cause, yes - if they used the same setup, they'd have the same config which would include support for exported resources.
The general issue with exported resources is that they require only friendlies on the same puppetmaster. In Labs (almost) anyone can be root on an instance and thus fake exported resources with the potential to disrupt or gain control of other instances (or other hosts under the control of the same puppetmaster).
With Beta-Cluster-Infrastructure, this situation might be different as it uses its own puppetmaster, but I don't know if by default self-hosted puppetmasters have exported resources enabled and if the setup is secure.
AFAIK everyone with access to deployment-prep has root on deployment-puppetmaster, which is the puppetmaster for all instances in the project
The problem is the labs self-hosted puppetmasters don't have exported resources enabled, they're an entirely separate module to the production puppetmasters
The self puppetmaster also tend to auto sign the client certificates. So in theory any labs instance could point to the beta puppet master and collect its resources..
So if you use resource collection and care about security (you could block access to puppetmaster from outside the project with security groups), you should turn off auto-signing. Tools runs a limited puppetmaster setup with autosigning turned off, for example.
Generating the keys on the puppetmaster and distributing them as regular files would be easy enough. No need for exported resources.
E.g. this: http://emeraldreverie.org/blog/2013/03/11/managing-ssh-host-keys-in-reliable-way/
Which is exactly the same technique that I've been experimenting with for keyholder ssh keys (works well now that I've got it all debugged)
I have a commit on -puppetmaster02 that does this, and it seems to mostly work. It seems to only include ECDSA keys (preferred over RSA, ED25519 not possible because prod still has some precise hosts, and trusty's puppet version doesn't allow for them, DSA unused), but it excludes hosts that no longer exist, and it has the FQDN, instance name, IPv4 address, and 'undef' (IPv6 I think, which of course we don't use in labs) on each line.
See also T153163, subtasks and associated commits (plus two or three that I have yet to publish, polishing code isn't fun :-)).
Change 333471 had a related patch set uploaded (by Alex Monk):
Allow use of PuppetDB in labs for sshknowngen
Change 333472 had a related patch set uploaded (by Alex Monk):
ssh: Don't add IPv6 address as an alias in exported resource if it's undefined
Change 333473 had a related patch set uploaded (by Alex Monk):
puppetdb: Allow tuning.conf to have a different shared_buffers value
Change 333473 merged by Alexandros Kosiaris:
puppetdb: Allow tuning.conf to have a different shared_buffers value
Change 333472 merged by Filippo Giunchedi:
ssh: Don't add IPv6 address as an alias in exported resource if it's undefined
Change 333471 merged by Alexandros Kosiaris:
[operations/puppet@production] Allow use of PuppetDB in labs for ssh_known_hosts
It's working but it's very loud - I've made https://gerrit.wikimedia.org/r/#/c/436624/ to deal with that
Also probably isn't handling deployment-snapshot01 as that uses deployment-dumps-puppetmaster instead - @ArielGlenn?
I've fixed this up a bit. It has a new (stretch) puppetmaster of its own that talks to Puppet DB and uses the normal deployment-puppetmaster03 CA (had to generate the cert for the client on -puppetmaster03 and copy the files to the correct place on -snapshot01)
So -snapshot01 shares ssh known hosts with other hosts and vice versa now
Still need to tidy up the old puppetmaster and deal with the extra class I added to -dumps-puppetmaster02
Mentioned in SAL (#wikimedia-releng) [2018-07-10T18:31:31Z] <Krenair> deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in T72792)
cleaned that up today per
<apergos> Krenair: I finally got a chance to note the config info for the old dumps puppetmaster in deployment-prep, you can kill it. I should have everything in my notes now
<apergos> thanks again
Just waiting on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/435631/ to merge now, should probably mention this task in it
Change 435631 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] Allow PuppetDB use on standalone puppetmasters
Change 435631 merged by Bstorm:
[operations/puppet@production] Allow PuppetDB use on standalone puppetmasters