With scoped membership we should also be able to reform our custom policy.yaml files. The whole Train-era model is described here: https://docs.openstack.org/keystone/latest/admin/service-api-protection.html

I'm pleased to note that the upstream has at last converged on a setup much like the one we use already.

Ideally our end state will look something like this:

• system-scope admins -

novaadmin
wmcs SRE staff

• system-scope members -

<probably none>

• system-scope readers -

novaobserver

• domain-scope roles -

<we don't currently use domains that aren't == to projects>

• project-scope admin -

any user that currently has the 'projectadmin' role in a project

• project-scope member -

any user that currently has the 'user' role in a project

• project-scope reader -

<no need to use this thanks to the system scope for novaobserver>

The existing glanceadmin and designateadmin roles should probably remain as they are.

The changes for novaadmin and novaobserver can be made immediately and safely.

Mapping 'user' and 'projectadmin' must be done with care; it's quite likely that the upstream project-scoped roles are similar to our custom policies but it's not guaranteed. In theory oslo provides scripts for auditing the diffs between a policy.yaml and the default policies, so we will need to

1. create a duplicate of existing policy files which does a blind renaming of the roles
2. determine the diff between those new policies and the in-code defaults
3. create yet new policies that implement those diffs for the admin/member roles along with our custom policies for projectadmin/user
4. *big hand wave* update or find a replacement for our 'Project Members' UI in Horizon
5. run a script that maps projectadmin/user roles onto project-scoped admin/member roles
6. remove projectadmin/member policies
7. run another script that removes all projectadmin/user roles

This would be awesome for multi-region as well, possibly allowing the keystone link between them to strictly be LDAP.

So #4...that whole UI doesn't work with this setup?
On #5, you mean "projectadmin/user roles onto admin/member roles", right?

In T274385#6849846, @Bstorm wrote:

This would be awesome for multi-region as well, possibly allowing the keystone link between them to strictly be LDAP.

So #4...that whole UI doesn't work with this setup?

The current UI for this is made by me for our exact use-case. If we're lucky, the regular Horizon membership panel will work for this once we use scoped roles; if not then our custom panel will have to change (probably just to use different role IDs)

On #5, you mean "projectadmin/user roles onto admin/member roles", right?

Yes! Fixed, I think.

A few notes:

• Setting a role on all projects works via the --inheritence arg; that seems to be unrelated to other issues of role scope. So, e.g. to give a user admin in all projects in the 'default' domain (our only domain):

openstack role add --user testadmin --inherited --domain default admin

• There's a setting in each of our .conf files that turns on the new scoped role checking:

[oslo_policy]
enforce_scope = True

Making that work with system=all scope requires a bunch of special ENV settings for the user using it:

export OS_SYSTEM_SCOPE=all
export OS_PROJECT_ID=
export OS_PROJECT_NAME=

• I'm not sure we care about scope at all, it seems unrelated to my immediate goal of getting novaadmin and novaobserver to work everywhere.

• Default policies assume admin/member/reader roles but I'm not clear on where those roles are supposed to come from or how the default policies can recognize them. Hopefully it's just by name but that needs more testing.

This task has become a bit of a catch-all for role/policy changes so I've created a new parent task for the larger issue, T276018. This task will specifically be about novaadmin and novaobserver.

I think handling novaobserver and novaadmin is simple.

• Add inherited domain-wide roles:

openstack role add --user novaadmin --domain default --inherited user
openstack role add --user novaadmin --domain default --inherited projectadmin
openstack role add --user novaobserver --domain default --inherited observer

• Remove the icinga monitors that assert universal project membership for novaadmin and novaobserver

https://gerrit.wikimedia.org/r/c/operations/puppet/+/667422

• Remove the hooks that insert novaobserver and novaadmin into new projects

https://gerrit.wikimedia.org/r/c/operations/puppet/+/667423

• Remove novaadmin and novaobserver roles from every project and leave them in just the default domain

can do this with a one-off script.

• Leave the observer and admin projects where they are

I'm not totally sure about this one but it seems moderately useful to have a persistent project for the observer user to auth against.

...

I think that will get us what we want. A side-effect of this will be that novaobserver and novaadmin will be removed from ldap for all projects, which will prevent them from having login access to VMs. I'm pretty sure that's an improvement unless I'm forgetting some reason why they need to be able to ssh or sudo or whatever on actual VMs.

The bad news is this solution won't work perfectly for adding universal rights to actual human users; they would get all the openstack permissions but would still need to be explicitly added to a project for ssh access. (Come to think of it this might be an improvement for me as well since I can always use root keys for ssh but would appreciate universal Horizon access).

Change 667422 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: stop monitoring for novaadmin/novaobserver project membership

https://gerrit.wikimedia.org/r/667422

Change 667423 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfkeystonehooks: stop adding service users to all projects

https://gerrit.wikimedia.org/r/667423

Change 667422 merged by Andrew Bogott:
[operations/puppet@production] Keystone: stop monitoring for novaadmin/novaobserver project membership

https://gerrit.wikimedia.org/r/667422

Change 667423 merged by Andrew Bogott:
[operations/puppet@production] wmfkeystonehooks: stop adding service users to all projects

https://gerrit.wikimedia.org/r/667423

Mentioned in SAL (#wikimedia-cloud) [2021-03-01T19:50:54Z] <andrewbogott> adding inherited domain-wide roles to novaadmin and novaobserver as per T274385

Mentioned in SAL (#wikimedia-cloud) [2021-03-01T19:51:11Z] <andrewbogott> removing novaobserver from all projects save 'observer' for T274385

Mentioned in SAL (#wikimedia-cloud) [2021-03-01T20:12:11Z] <andrewbogott> removing novaadmin from all projects save 'admin' for T274385

Change 668774 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove check_keystone_projects.py

https://gerrit.wikimedia.org/r/668774

Change 668775 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Keystone: remove some wkfkeystonehooks config flags

https://gerrit.wikimedia.org/r/668775

Change 668774 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove check_keystone_projects.py

https://gerrit.wikimedia.org/r/668774

Change 668775 merged by Andrew Bogott:
[operations/puppet@production] Keystone: remove some wkfkeystonehooks config flags

https://gerrit.wikimedia.org/r/668775