Right now we secure the enc by limiting API access. Since this is primarily access by Horizon it should check Keystone tokens like any other openstack service.
Description
Description
Details
Details
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T305453 Horizon should use a proxy to access cloud vps hosted apis | |||
| Resolved | • taavi | T274666 Add keystone auth middleware to the puppet enc api | |||
| Resolved | • taavi | T295247 Cloud VPS Puppet ENC API should have their own servers | |||
| Resolved | • taavi | T308486 openstack-browser stopped showing puppet classes in use |
Event Timeline
Comment Actions
This is likely easier if we use https://github.com/Rackspace-DOT/flask_keystone. It doesn't appear to be packaged for Debian, at least yet.
Comment Actions
Change 778616 had a related patch set uploaded (by Majavah; author: Majavah):
[openstack/horizon/wmf-puppet-dashboard@main] cleanup and keystone auth support
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2022-04-09T19:37:42Z] <taavi> add 'puppet-enc' service & endpoint to keystone T274666
Comment Actions
Change 779899 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] openstack: make enc-cli authenticate via keystone
Comment Actions
Change 781977 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] openstack: make wmf_sink authenticate to enc api via keystone
Comment Actions
Change 781977 merged by Andrew Bogott:
[operations/puppet@production] openstack: make wmf_sink authenticate to enc api via keystone
Comment Actions
Change 778616 merged by jenkins-bot:
[openstack/horizon/wmf-puppet-dashboard@main] cleanup and keystone auth support
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2022-04-20T14:55:13Z] <taavi@deploy1002> Started deploy [horizon/deploy@9d02cd6]: updating wmf-puppet-dashboard for keystone authentication support T274666 (eqiad1)
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2022-04-20T15:00:16Z] <taavi@deploy1002> Finished deploy [horizon/deploy@9d02cd6]: updating wmf-puppet-dashboard for keystone authentication support T274666 (eqiad1) (duration: 05m 03s)
Comment Actions
Change 785110 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:openstack::encapi: add tls for write endpoint
Comment Actions
Change 785128 had a related patch set uploaded (by Majavah; author: Majavah):
[labs/private@master] add dummy password for cloudinfra token validator
Comment Actions
Change 785134 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] P:openstack::encapi: add keystone token verification
Comment Actions
Change 785128 merged by Andrew Bogott:
[labs/private@master] add dummy password for cloudinfra token validator
Comment Actions
Change 785110 merged by Andrew Bogott:
[operations/puppet@production] P:openstack::encapi: add tls for write endpoint
Comment Actions
Change 779899 merged by David Caro:
[operations/puppet@production] openstack: make enc-cli authenticate via keystone
Comment Actions
Change 785134 merged by David Caro:
[operations/puppet@production] P:openstack::encapi: add keystone token verification
Comment Actions
Change 792228 had a related patch set uploaded (by David Caro; author: David Caro):
[operations/puppet@production] Revert "P:openstack::encapi: add keystone token verification"
Comment Actions
Change 792228 abandoned by David Caro:
[operations/puppet@production] Revert "P:openstack::encapi: add keystone token verification"
Reason:
The issue was fixed :)
Comment Actions
[17:11] < mutante> openstack-browser stopped showing puppet classes and clicking other links seems slow / working sometimes. should I make a ticket or ongoing work? [17:12] < mutante> f.e. https://openstack-browser.toolforge.org/puppetclass/ seems empty [17:12] < mutante> I see backlog about outage on horizon. so maybe it's that then ack [17:13] < mutante> ok, i'll just check in later again
[17:50] < bd808> mutante: I think we need a bug report for the openstack-browser things. I don't have time to dig in deeper at the moment, but it looks to me like the puppet-enc backend is now requiring auth even for browsing and that is making all the anon queries done by the tool fail. This is possibly unintended fallout of work taavi has been doing to add better auth to that puppet-enc service. [17:51] < bd808> the app is doing things that are functionally `curl https://puppet-enc.cloudinfra.wmcloud.org:8143/v1/roles` -- https://phabricator.wikimedia.org/source/tool-keystone-browser/browse/master/keystone_browser/puppetclasses.py [17:57] < mutante> bd808: ACK! will do that in a bit. thanks [17:59] < taavi> that sounds like the most likely explanation, I thought I already migrated openstack-browser to use keystone authentication with puppet-enc :/ [18:00] < taavi> patches welcome, the proxies.py might have useful code you can almost copy-paste. otherwise will try to find some time to fix that soon-ish, but no promises due to irl priorities [18:05] < bd808> *nod* looks like the `proxy_client` magic from https://phabricator.wikimedia.org/source/tool-keystone-browser/browse/master/keystone_browser/proxies.py is what puppetclasses.py also needs.
Comment Actions
Change 792619 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] openstack: Make enc api enforce keystone policy
Comment Actions
Change 792619 merged by Andrew Bogott:
[operations/puppet@production] openstack: Make enc api enforce keystone policy
Comment Actions
Change 793524 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] openstack: encapi: add a custom error class
Comment Actions
Change 793524 merged by Andrew Bogott:
[operations/puppet@production] openstack: encapi: add a custom error class
Comment Actions
Change 809721 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):
[operations/puppet@production] wmcs-enc-cli.py: fix args passed to requests.post
Comment Actions
Change 809721 merged by Andrew Bogott:
[operations/puppet@production] wmcs-enc-cli.py: fix args passed to requests.post
Comment Actions
I don't know if this is related to this exact task, but right now the enc is complaining a lot about auth failures from the the cloud puppetmaster:
Of course, it can't actually be as broken as it claims since puppet is still mostly working.
2022-10-25 23:13:09.036 592 INFO flask_keystone [req-6eb6e434-16ab-4a79-af9b-d69951136095 novaadmin admin-monitoring None default default] Couldn't authenticate user 'None' with X-Identity-Status 'Invalid' 2022-10-25 23:13:09.044 592 INFO flask_oslolog.middleware [req-6eb6e434-16ab-4a79-af9b-d69951136095 novaadmin admin-monitoring None default default] ::ffff:172.16.0.49 - Invalid "GET /v1/cloudvirt-canary/node/canary1040-01.cloudvirt-canary.eqiad1.wikimedia.cloud" status: 200 len: 41 2022-10-25 23:13:09.256 592 INFO flask_keystone [req-6eb6e434-16ab-4a79-af9b-d69951136095 novaadmin admin-monitoring None default default] Couldn't authenticate user 'None' with X-Identity-Status 'Invalid' 2022-10-25 23:13:09.313 592 INFO flask_oslolog.middleware [req-6eb6e434-16ab-4a79-af9b-d69951136095 novaadmin admin-monitoring None default default] ::ffff:172.16.0.49 - Invalid "GET /v1/analytics/node/kafka-test-cloud-5.analytics.eqiad1.wikimedia.cloud" status: 200 len: 8284
Comment Actions
Change 849494 had a related patch set uploaded (by Majavah; author: Majavah):
[operations/puppet@production] openstack: modernize puppetleaks script
Comment Actions
This is mostly expected and can be ignored - the endpoint called by puppetmasters currently doens't require authentication. I guess the library still tries to authenticate them, which fails as expected since there is no keystone token provided.
Comment Actions
Change 849494 merged by David Caro:
[operations/puppet@production] openstack: modernize puppetleaks script