linking T277778: Toolforge: consider decoupling user & accounts from CloudVPS accounts for reference, in case is relevant

I'm trying to understand pros & cons of the different protocols, especially OIDC (OpenID Connect) vs CAS.

I played a bit with OIDC in a previous job, and I remember it as fairly complex but also quite well supported by a number of libraries that can be used to implement a client. I'm not sure how that compares with CAS.

A quick Google search led me to this presentation (although not very recent) with some useful comparisons: https://ldapcon.org/2017/wp-content/uploads/2017/08/16_Cl%C3%A9ment-Oudot_PRE_LDAPCon2017_SSO-1.pdf

In T363983#9777461, @fnegri wrote:

I'm trying to understand pros & cons of the different protocols, especially OIDC (OpenID Connect) vs CAS.

I played a bit with OIDC in a previous job, and I remember it as fairly complex but also quite well supported by a number of libraries that can be used to implement a client. I'm not sure how that compares with CAS.

A quick Google search led me to this presentation (although not very recent) with some useful comparisons: https://ldapcon.org/2017/wp-content/uploads/2017/08/16_Cl%C3%A9ment-Oudot_PRE_LDAPCon2017_SSO-1.pdf

Nice!

Just checked the openid python libraries mentioned in the openid page and all have been archived :/

We might be a bit restricted here on what can we actually use underneath though, as in we have our users in LDAP, no matter what, and we have one single-sign-on implementation in prod that we can use (idp.w.o).

My current ideas (still exploring) are:

• directly LDAP: we want to avoid this, it's what keystone, toolsadmin and wikitech currently do, ldap is kinda flaky and load sensitive
• keystone (as LDAP proxy): the main downside I currently see without having played much, is that you need many things to authenticate, around 9 settings between domain, user, region, ..., not many libraries (that I have found), and no sso. Big advantage, if we have storage/trove integration, the auth is going to be proobably very similar, so users would be used to it xd
• idp.w.o (as LDAP proxy): we get sso, the protocol is CAS, kinda easy, probably don't need any extra library, only user-pass, no two-factor auth yet though)
• keycloak with ldap as federated backend: we have an extra layer, but no sso, we can add local users, have different deployments, etc., requires having the extra service
• keycloak, with idp.w.o as federated backend: this allows us to add an extra layer on top of idp, we still have sso, but we can for example add local users and such (easier local deployment for example), but requires having an extra service

We also might want to change keystone to authenticate using idp.w.o at some point, to allow sso, if that's the case, we will have to move to idp auth eventually, so we might want to do so already.

I have not tried to setup a local keystone (will try with https://quay.io/repository/openstack.kolla/keystone?tab=tags&tag=latest from today's check-in), or idp instance, if it's easy, we might not gain anything by adding keycloak (as we would be able to run keystone in local deployments for example).

In any case, it seems quite possible that the storage access (s3 buckets) will have to be done through keystone with app credentials, so the users will have to authenticate that way to access non-public buckets. I was thinking on exposing that as part of the 'storage-api' service thingie, like:

> toolforge storage s3 list

+------------------------------------------------+
| name    |   credential | url                   |
+---------+--------------+-----------------------+
| bucket1 | somelonghash | https://url-to-bucket |
...

Depending on how it's implemented, the credentials might not be per-bucket though, but for the whole tool there (app credential associated to the tool user, if I understand correctly... @Andrew please correct me if I'm wrong xd).

In any case, it seems that unless we use some front (ex. keycloak) and adding extra info there, what we end up having is the LDAP data, that is, when you log in (as your user), we get the list of groups you belong to (the tools).

So there's no per-tool authentication token, making T363808: [builds-api, builds-cli] Prefix all endpoints with `/tool/<toolname>` needed (or similar, like passing always a tool parameter, though I'd prefer the path for namespacing) right?

I think using idp.w.o is my favourite solution, as it can potentially be a true "single" sign-on that all applications can rely on. Using CAS without any extra library is probably a good first step, and we can evaluate migrating to OIDC iff we find it provides any advantage.

Just checked the openid python libraries mentioned in the openid page and all have been archived :/

I found a discussion about that here: https://www.reddit.com/r/Python/comments/16pin4l/a_maintained_library_for_oidc_in_python/

Looks like https://github.com/IdentityPython/idpy-oidc is maintained and certified, it includes both a server implementation (which we don't need) and a client.

So there's no per-tool authentication token, making T363808: [builds-api] Prefix all endpoints with /tool/<toolname> needed (or similar, like passing always a tool parameter, though I'd prefer the path for namespacing) right?

I agree, after thinking about it I'm in favour of adding the /tool prefix. We can discuss the implementation in T363808.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/22

Draft: investigate authentication

If you would like to experiment with tool-specific keystone auth, I've set up an example account. The creds for the tool user are in cloudcontrol1006:/home/dcaro/.config/openstack/clouds.yaml.

project: toolsbeta.test (id: a40d6252508b4fdaa057279aa306d151)
domain: toolsbeta (id: 31a151b68a61450a9a75a187a3ab4eb9)
service user: toolsbeta.test (id: toolsbeta.test)
group: toolsbeta.test (id: 51595)

You can also add/remove human users to/from that tool with striker and they will also be added to/removed from the toolsbeta.test project automatically due to the group assignment.

This was set up using 'keystoneify' at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1040243 -- we can adapt that code to run within the tools domain as a daemon if we want projects like that for every tool.

In T363983#9875492, @Andrew wrote:

If you would like to experiment with tool-specific keystone auth, I've set up an example account. The creds for the tool user are in cloudcontrol1006:/home/dcaro/.config/openstack/clouds.yaml.

project: toolsbeta.test (id: a40d6252508b4fdaa057279aa306d151)
domain: toolsbeta (id: 31a151b68a61450a9a75a187a3ab4eb9)
service user: toolsbeta.test (id: toolsbeta.test)
group: toolsbeta.test (id: 51595)

You can also add/remove human users to/from that tool with striker and they will also be added to/removed from the toolsbeta.test project automatically due to the group assignment.

This was set up using 'keystoneify' at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1040243 -- we can adapt that code to run within the tools domain as a daemon if we want projects like that for every tool.

<3 thanks!

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/23

DONOTMERGE_api: proxy requests to the backend APIs

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/23

api: auth and proxy requests to the backend APIs

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/387

api-gateway: bump to 0.0.29-20240704101259-2cfd83b3

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/387

api-gateway: bump to 0.0.29-20240704101259-2cfd83b3

We will use custom deploy tokens for deployments, and now horizon has moved to using idp, so that reduces the options.

I'll move this back to the queue for now, but will retake probably q3/q4 2024-25, before we start working on the UI.

Quick update, I've been looking into enabling CAS or OIDC (openid connect) into nginx, and it seems it's not supported unless you have the paid "plus" version :/, there's a small lua plugin that seems a bit unmaintained to do some of the CAS process.

On the other hand, apache has a certified module for oidc (https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-oidc/), so we might want to move from nginx to apache, and take advantage of that, might need some investigation. Another alternative is implementing the auth on the api-gateway fastapi side, though might be faster/simpler on the apache/nginx, we can use the fastapi as a fallback if it gets complicated.

Some notes on CAS, and client authentication.

By default CAS does not enable proxy tokens (https://apereo.github.io/cas/7.2.x/services/Configuring-Service-Proxy-Policy.html), so the only action you can do with the token is validate it and get the gorups/info of the user as far as I understand it (to be tested).

I think this is not a security issue, specially because all that info is already public, so in order to enable clis to authenticate users, I think it would be ok to enable a service http://127.0.0.1.* (callback url) similar to how oauth 2 enables native apps, but well, given that all we need and want is just info about the user groups it's way simpler and safer (as the app can't act on the name of the user anywhere else).

This is a draft MR trying to create a POC for it https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/69

There's the question also about how to expire/renew these tokens too, but I'll investigate that later.

For the cli flow, it would be something like:

• Listen on 127.0.0.1:<randomport> for the callback
• Ask the user to log in in the browser giving the url of the idp with said callback in the command line
• Retrieve the token from the callback, then call the toolforge api callback with that token
• Retrieve the session cookie from that call and store it for the session

We might want at some point to be able to have a bit longer-lived tokens for the cli and other automated applications, maybe something similar to the deployment-tokens for the components-api (or even something that will include those).

This looks great! I also checked the POC code and it looks good.

Do you plan on testing the OIDC protocol as well, in addition to the CAS protocol? OIDC protocol is supported by Apereo CAS, but it might require tweaking some settings in the Apereo config. I did some more research today after seeing your POC, and it looks like OIDC might give us some advantages, in particular it could entirely replace the need for generating session tokens in the Toolforge API.

Note: I don't think that testing OIDC should necessarily be a blocker, we could start with CAS if it's easier, and we should be able to migrate to OIDC at a later moment in a way that is transparent to users.

OIDC flow (RFC 8252) as I understand it:

• CLI listens on 127.0.0.1
• CLI points user to log in with the browser, the local server on 127.0.0.1 receives an Authorization code
• CLI sends that Authorization code to IDP (either directly or proxied by our API gateway) and gets an Access Token
• Once the CLI has an Access Token, it can attach it to each Toolforge API request
• The Toolforge API can validate and decode that token on each request, extracting the user/group information
• This means the Toolforge API would become fully OAuth-based, no need for API session tokens/cookies at all
• Access Tokens are short lived and can be refreshed automatically by the CLI

I've been looking into enabling CAS or OIDC (openid connect) into nginx, and it seems it's not supported unless you have the paid "plus" version :/

That sucks. :( It looks like Envoy supports it though, maybe we could consider replacing Nginx with Envoy? But it also looks easy enough to implement the token validation in Python in the Toolforge API gateway.

I found this Python library that might be helpful to implement both the token validation (in the API gateway) and the Access Token generation and refresh (in the CLI client): https://docs.authlib.org/. The docs are not as clear as I would like, but it looks quite powerful and in active development.

We might want at some point to be able to have a bit longer-lived tokens for the cli and other automated applications

I agree this will be necessary for example for toolforge cron jobs that need to call the toolforge API. We should provide a way for users to generate long-lived tokens. But I would only use them for non-interactive jobs and applications, not for the CLI. As a data point, the AWS CLI recently added an OAuth-based login flow.