Change 916522 merged by Jelto:

[operations/puppet@production] gitlab: sync all configured providers

https://gerrit.wikimedia.org/r/916522

Change 935707 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: remove unused sso hiera config

https://gerrit.wikimedia.org/r/935707

Change 935708 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: use openid_connect as default sso method

https://gerrit.wikimedia.org/r/935708

Change 935707 merged by Jelto:

[operations/puppet@production] gitlab: remove unused sso hiera config

https://gerrit.wikimedia.org/r/935707

Change 935708 merged by Jelto:

[operations/puppet@production] gitlab: use openid_connect as default sso method

https://gerrit.wikimedia.org/r/935708

GitLab replicas (e.g. https://gitlab-replica.wikimedia.org) use oidc as the default login method now. Login (normal user login and admin login) works fine for me, no more issues with nda/ops/wmf ldap groups.

I'll do some more testing (also with new/other users). Then we can replace idp-test with production idp again. We used idp-test for troubleshooting.

After that we can set profile::gitlab::auto_sign_in_with: openid_connect globally, so also for production gitlab1004.

I ran into a 422 error trying to login to gitlab-replica.

Thanks for reporting the issue @Arnoldokoth !

I grepped a bit in /var/log/cas/cas-2023-07-05.log on idp-test1002 and found something about a potential attribute mismatch of mail and email:

2023-07-05 15:06:03,161 DEBUG [org.apereo.cas.oidc.claims.BaseOidcScopeAttributeReleasePolicy] - <Found mapped attribute [mail] with value [[...@wikimedia.org]] for claim [email]>

So I'm wondering if we have to change the OIDC scope from

"scope" => ["openid", "profile", "email"]

to

"scope" => ["openid", "profile", "mail"]

Otherwise I don't see anything obvious in the log. Arnolds account has a mail in ldap and GitLab.

@jbond do you have an idea what could be wrong?

In T320390#8993613, @Jelto wrote:

Thanks for reporting the issue @Arnoldokoth !

I grepped a bit in /var/log/cas/cas-2023-07-05.log on idp-test1002 and found something about a potential attribute mismatch of mail and email:

2023-07-05 15:06:03,161 DEBUG [org.apereo.cas.oidc.claims.BaseOidcScopeAttributeReleasePolicy] - <Found mapped attribute [mail] with value [[...@wikimedia.org]] for claim [email]>

So I'm wondering if we have to change the OIDC scope from

No This should be all correct. What this is saying is that we have received a mail attribute from ldap and we will map it to the OIDC email scope for release. You can see whats released by cas by looking at the following audit entry

WHAT: {service=https://gitlab-replica.wikimedia.org/users/auth/openid_connect/callback, attributes={name=[AOkoth], preferred_username=[aokoth], email=[removed]}, id=AOkoth, scopes=[email, openid, profile], client_id=gitlab_replica_oidc}
ACTION: OAUTH2_USER_PROFILE_CREATED
APPLICATION: CAS
WHEN: Wed Jul 05 15:06:14 UTC 2023
CLIENT IP ADDRESS: 2620:0:860:1:208:80:153:7
SERVER IP ADDRESS: 127.0.0.1

Otherwise I don't see anything obvious in the log. Arnolds account has a mail in ldap and GitLab.

Every thing looks good on the cas side of things

Depending on how the Gitlab OIDC pulls information we might have to change: userinfo_endpoint in client config options. If Gitlab users to profile to pull the email, then we need to set it to: /oidc/oidcProfile default appears to be /user_info

Change 937945 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: set userinfo_endpoint in client_options:

https://gerrit.wikimedia.org/r/937945

Change 937945 merged by Jelto:

[operations/puppet@production] gitlab: set userinfo_endpoint in client_options:

https://gerrit.wikimedia.org/r/937945

We didn't find a solution yet, but I'll spend some time looking into the CAS side of things tomorrow.

@Jelto @SLyngshede-WMF I believe my login works now.

I did sign in to the replica and I can see Signed in with openid_connect authentication on the authentication logs.

The only thing that potentially changed was I went to the account section and clicked "Connect Wikimedia Dev Account (OIDC)".

Yeah, this might have been the issue for me. I just signed into the admin URL as well on gitlab-replica.

@Arnoldokoth thanks for testing this. You are talking about the your individual profile settings in https://gitlab-replica.wikimedia.org/-/profile/account right? And your log in https://gitlab-replica.wikimedia.org/-/profile/audit_log?

@SLyngshede-WMF did you change anything else on the cas/oidc config? Are you able to login as well on gitlab-replica?

I'm not sure what happens if you click on the OIDC button in your account. Maybe that links your account different from a normal login? But I don't think that's a reasonable approach to let all users do that manually before migrating to OIDC only.

@Jelto No not yet, and I just tried to login still the same error. I can see how it might work if the accounts are simply linked, but that will still be an issue for new users.

Also you'd need to be able to login somewhere so you can link the accounts.

For OIDC via CAS in our Python applications we're relying on a special OIDC backend https://github.com/python-social-auth/social-core/blob/master/social_core/backends/cas.py this deals with the fact that CAS will put attributes fetched from the userinfo URL into a JSON structure that looks something like this:

"attributes": {
    "preferred_username": "<username>",
    "email": "user@example.org",
    "given_name": "John",
    "family_name": "Doe"
}

OmniAuth in Ruby and GitLab doesn't know this and expects the attributes to be at the root, not in a directory called "attributes".

One thing we did attempt which sort of works is to force CAS to always return claims in the response, and force the response to be an id_token. That's not really ideal nor is it recommended. It basically relies on a hack that ensures backwards compatibility with older CAS installation.

What CAS is doing it's exact wrong, it's not clearly specified how attributes are to be returned from an OIDC server. It's just also not what anyone expects, hence the reason to add the new CAS backend to Python Social Auth.

A few ideas:

• Patch OmniAuth
• Patch Apereo CAS
• Attempt to use OAuth2 rather than OIDC.

The fixes that worked a little, but failed to provide correct user information was to set the following in CAS:

cas.authn.oidc.id-token.include-id-token-claims=true

Last Friday we've done some troubleshooting and tested a lot of different configurations, thanks @SLyngshede-WMF again! In one configuration login to the replica worked and new accounts were created using a temporary email (like temp-email-for-oauth-<username>@gitlab.localhost). Giving a user the message ""Your account is pending approval from your GitLab administrator" which is expected.

This happened today as well even though we re-enabled puppet on gitlab2002 and idp-test1002. I'm not entirely sure which setting on cas/idp caused this, I thought the cas.authn.oidc.id-token.include-id-token-claims=true. But I check this config on cas and it's set to false again. Does cas needs to be restarted one more time? @SLyngshede-WMF @jbond do you remember what setting caused the usage of temporary emails in the profile?

I'd like to continue testing different settings on GitLab side but the "e-mail can't be blank" may be a bit closer to what we want instead of using temporary mails.

There are two settings which we may test, one is send_scope_to_token_endpoint which is set to false currently (here). And another one is gitlab_rails['omniauth_auto_link_user'] (docs).

As requested, @jbond your two users on the replica and production dumped via API (curl "https://gitlab-replica.wikimedia.org/api/v4/users?username=jbond):

replica.

[
   {
      "id":1102,
      "username":"jbond",
      "name":"Jbond",
      "state":"active",
      "avatar_url":null,
      "web_url":"https://gitlab-replica.wikimedia.org/jbond",
      "created_at":"2023-07-17T12:35:43.749Z",
      "bio":"",
      "location":"",
      "public_email":null,
      "skype":"",
      "linkedin":"",
      "twitter":"",
      "discord":"",
      "website_url":"",
      "organization":"",
      "job_title":"",
      "pronouns":null,
      "bot":false,
      "work_information":null,
      "followers":0,
      "following":0,
      "is_followed":false,
      "local_time":null,
      "last_sign_in_at":null,
      "confirmed_at":"2023-07-17T12:35:43.600Z",
      "last_activity_on":null,
      "email":"jbond@wikimedia.org",
      "theme_id":2,
      "color_scheme_id":1,
      "projects_limit":250,
      "current_sign_in_at":null,
      "identities":[
         {
            "provider":"openid_connect",
            "extern_uid":"jbond"
         }
      ],
      "can_create_group":false,
      "can_create_project":true,
      "two_factor_enabled":false,
      "external":false,
      "private_profile":false,
      "commit_email":"jbond@wikimedia.org",
      "is_admin":false,
      "note":null,
      "namespace_id":2704,
      "created_by":null
   }
]

production:

[
   {
      "id":520,
      "username":"jbond",
      "name":"Jbond",
      "state":"active",
      "avatar_url":null,
      "web_url":"https://gitlab.wikimedia.org/jbond",
      "created_at":"2022-10-10T09:43:07.756Z",
      "bio":"",
      "location":"",
      "public_email":null,
      "skype":"",
      "linkedin":"",
      "twitter":"",
      "discord":"",
      "website_url":"",
      "organization":"",
      "job_title":"",
      "pronouns":null,
      "bot":false,
      "work_information":null,
      "followers":0,
      "following":0,
      "is_followed":false,
      "local_time":null,
      "last_sign_in_at":"2023-06-27T09:07:30.662Z",
      "confirmed_at":"2022-10-10T09:43:07.630Z",
      "last_activity_on":"2023-07-13",
      "email":"jbond@wikimedia.org",
      "theme_id":2,
      "color_scheme_id":1,
      "projects_limit":250,
      "current_sign_in_at":"2023-07-12T22:03:16.202Z",
      "identities":[
         {
            "provider":"cas3",
            "extern_uid":"jbond"
         }
      ],
      "can_create_group":false,
      "can_create_project":true,
      "two_factor_enabled":true,
      "external":false,
      "private_profile":false,
      "commit_email":"jbond@wikimedia.org",
      "is_admin":false,
      "note":null,
      "namespace_id":1507,
      "created_by":null
   }
]

I did some more testing today and can confirm that the required config is cas.authn.oidc.id-token.include-id-token-claims=true i suspect that tomcat wasn't restarted on Friday and puppet doesn't do it automagically. Logins are now also working registering the correct email address. This was not working with the config tested on frioday as we had the fgollowig entry

cas.authn.oidc.core.user-defined-scopes.email=mail

moving this to a user defined scope seemed to make it so that gitlab was unable to find the email address and thus used something temporary. I also confirmed that gitlab does not care about the email_verified attribute.

One other thing i noticed is that when i first attempted to login this morning i was received a 422 error with the message email address already used. This happens because my account, synced from production, has identities['provider'] = 'cas3' however the new account uses identities['provider'] = 'openid_connect'` . This seems to be the only difference between the records as such we will need to account for this when we migrate production to OIDC, if there is no gitlab way to do this we will probably need to do a manual DB update.

from our side we will need to check if cas.authn.oidc.id-token.include-id-token-claims=true is ok to enable globally or if its possible to enable it per service

from our side we will need to check if cas.authn.oidc.id-token.include-id-token-claims=true is ok to enable globally or if its possible to enable it per service

Something else to try is setting this back to false and in gitlab, setting send_scope_to_token_endpoint=true in the gitlab_rails['omniauth_providers']. @Jelto let me know when we can test this

Thanks for troubleshooting this more! I can confirm existing users have cas3 in the identities section. This leads to a email address already used error.

I've done a test (with @LSobanski ) and removed the cas3 identity in the GitLab account using API:

DELETE --header 'PRIVATE-TOKEN: ...' "https://gitlab-replica.wikimedia.org/api/v4/users/62/identities/cas3"

Lukasz was not able to login, same error message email address already used.

Something else to try is setting this back to false and in gitlab, setting send_scope_to_token_endpoint=true in the gitlab_rails['omniauth_providers']. @Jelto let me know when we can test this

We can do that any time. We can pause puppet on gitlab2002 and edit the config manually or setup a puppet patch for that.

In T320390#9018611, @Jelto wrote:

...
There are two settings which we may test, one is send_scope_to_token_endpoint which is set to false currently (here). And another one is gitlab_rails['omniauth_auto_link_user'] (docs).

A login with gitlab_rails['omniauth_auto_link_user'] = ["cas3", "openid_connect"] and a existing cas3 account gives the same error message "email address already used".

Also the usage of send_scope_to_token_endpoint = true made little difference according to @jbond in IRC

I looked at the GitLab gitlabhq_production database and identities table.

I connected to the psql database using: sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql -d gitlabhq_production and done some exploration:

\dt
\d identities 
 select * from identities limit 10;
 select * from identities where extern_uid='eoghan';

Then I tried to update the provider field for the user eoghan to openid_connect:

update identities SET provider='openid_connect' where extern_uid='eoghan';

That worked and api returns for eoghan:

"identities": [{"provider":"openid_connect","extern_uid":"eoghan"}]

However @eoghan reported the same error "email address already used".

In T320390#9022500, @Jelto wrote:

In T320390#9018611, @Jelto wrote:

...
There are two settings which we may test, one is send_scope_to_token_endpoint which is set to false currently (here). And another one is gitlab_rails['omniauth_auto_link_user'] (docs).

A login with gitlab_rails['omniauth_auto_link_user'] = ["cas3", "openid_connect"] and a existing cas3 account gives the same error message "email address already used".

The docs Suggest this is the way to go. i wonder if its worth raising a bug?

In T320390#9022808, @jbond wrote:

The docs Suggest this is the way to go. i wonder if its worth raising a bug?

https://gitlab.com/gitlab-org/gitlab/-/issues/393840 should cover our issue mostly?

While searching for open issues I also found https://gitlab.com/gitlab-org/gitlab/-/issues/338293., which mentions omniauth_auto_link_saml_user. I just tried the following additional settings one more time (with the saml stuff)

gitlab_rails['omniauth_auto_link_user'] = ["openid_connect", "cas3"]
gitlab_rails['omniauth_auto_link_saml_user'] = true

and @eoghan confirmed a successful login! In tests before I applied a manual db migration to eoghan (in the identities table). I'll put the changes into puppet and re-enable the restore from production again. So we can try that again in a "fresh" state without manual migrations. I'm not sure if they are needed.

But it seems either the saml flag, a different order of ["openid_connect", "cas3"] (instead of ["cas3", "openid_connect"]) and or the db migration are the missing piece.

Change 939307 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: auto link existing users with OIDC

https://gerrit.wikimedia.org/r/939307

Change 939307 merged by Jelto:

[operations/puppet@production] gitlab: auto link existing users with OIDC

https://gerrit.wikimedia.org/r/939307

I deployed the change above which adds this two lines to the gitlab.rb file:

gitlab_rails['omniauth_auto_link_user'] = ["cas3", "openid_connect"]
gitlab_rails['omniauth_auto_link_saml_user'] = true

@eoghan and @LSobanski confirmed a successful login with production data on the replica (so no manual db migration).

It seems we have a solution for the login problems :)

The next steps are to use production idp instead of idp-test and enable oidc login on production host.

btw raising priority to high because this is a blocker for T338460.

I switched GitLab oidc login back to produciton idp (https://gerrit.wikimedia.org/r/c/operations/puppet/+/939345). I get the same error message mentioned in T320390#8930839 when I try to login to gitlab-replica.

Application Not Authorized to Use CAS

CAS log shows:

2023-07-24 06:58:52,629 DEBUG [org.apereo.cas.services.util.RegisteredServiceAccessStrategyEvaluator] - <Access is denied. The principal does not have the required attributes [{memberOf=[cn=nda,ou=groups,dc=wikimedia,dc=org, cn=ops,ou=groups,dc=wikimedia,dc=org, cn=wmf,ou=groups,dc=wikimedia,dc=org]}]>
2023-07-24 06:58:52,629 WARN [org.apereo.cas.services.RegisteredServiceAccessStrategyUtils] - <Cannot grant access to service [gitlab_replica_oidc]; it is not authorized for use by [Jelto].>

@jbond @SLyngshede-WMF did you change any IDP config last time when I posted the error for idp-test?

idp-test and production idp should allow gitlab_replica_oidc and are configured similar:
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/idp_test.yaml#40
https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/role/common/idp.yaml#206

@Jelto I think you have the wrong client id. Should be: "gitlab_replica_oidc".

CAS will check the serviceId / URL and for gitlab_oidc that only gitlab.wikimedia.org.

No, just checked, client id is correct.

We are using gitlab_replica_oidc on the replicas ("identifier" => "gitlab_replica_oidc" in /etc/gitlab/gitlab.rb.).
There is some puppet code to use gitlab_replica_oidc if the host is not the active one: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/gitlab.pp#61

My links in gitiles were a bit off and pointed to the production config.

When I attempt a login I get the following:

<Difference of checking required attributes: [[]]>
2023-07-24 12:14:03,692 DEBUG [org.apereo.cas.services.util.RegisteredServiceAccessStrategyEvaluator] - <Access is denied. The principal does not have the required attributes [{memberOf=[cn=nda,ou=groups,dc=wikimedia,dc=org, cn=ops,ou=groups,dc=wikimedia,dc=org, cn=wmf,ou=groups,dc=wikimedia,dc=org]}]>

I've seen it before, but I thought we fixed it. CAS doesn't think I'm in the correct groups.

Yes we had the same error before, I reported it in T320390#8930839. After my vacations the error was gone, so I assumed someone else fixed it. I did not upload a patch for that afaik.

When switching back to idp-test, I don't see the error. With production idp I see the error again. So I think there is some different configuration running on idp-test compared to production idp.

Change 941319 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] idp: remove nda from required_groups for gitlab_replica_oidc

https://gerrit.wikimedia.org/r/941319

I did a diff of the configurations for idp and idp-test, and they are basically the same, none of the settings that could affect OIDC differs.

We restarted idp1002 and idp-test1002. It seems the running configuration was not the one configured, because tomcat is not restarted automatically.

With both idp hosts we are back to e-mail can't be blank. So we have to find the correct idp/gitlab config and make sure it's persistent and survives a restart of idp.

Change 941319 abandoned by Jelto:

[operations/puppet@production] idp: remove nda from required_groups for gitlab_replica_oidc

Reason:

not needed at the moment

https://gerrit.wikimedia.org/r/941319

We previously suspected that the issue was that CAS nested the attributes it returns via the profile, but gave up on pursuing that as I believed it would require patching OmniAuth in Gitlab.

As a backup solution we started investigating OAuth2 as an option. That lead to https://apereo.github.io/cas/6.5.x/authentication/OAuth-Authentication-UserProfiles.html which explains that OAuth2 can return attributes either nested or flat. Nested being default.

Given that OIDC is build on top of OAuth2 we reasoned that it it might be possible to configure OIDC profiles to be returned in the FLAT format as well. In the documentation for OAuth2 we find "userProfileViewType" https://apereo.github.io/cas/6.6.x/authentication/OAuth-Authentication-Clients.html which does exactly what we want while not clear from the documentation this is valid for OIDC clients as well.

Testing on idp-test and gitlab-replica indicates that this does work an allows OmniAuth to see the email address.

I'll create a patch for CAS/Puppet to allow the profile view type to be set.

Change 941391 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] D:apereo_cas::service support FLAT profiles.

https://gerrit.wikimedia.org/r/941391

GitLab test instance has OIDC enabled as well now, thanks @jbond for that!

We expect a GitLab security update somewhere around next week, which means we have to upgrade to version 16 (T338460) with no more cas3 support.

I'd like to switch to OIDC before migrate to version 16, to reduce the number of moving parts during the version 16 migration.

Change 941391 merged by Slyngshede:

[operations/puppet@production] D:apereo_cas::service support FLAT profiles.

https://gerrit.wikimedia.org/r/941391

Change 942540 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] R:idp Enable flat OIDC profiles for gitlab

https://gerrit.wikimedia.org/r/942540

Change 942540 merged by Slyngshede:

[operations/puppet@production] R:idp Enable flat OIDC profiles for gitlab

https://gerrit.wikimedia.org/r/942540

Change 942600 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: auto_sign_in_with openid_connect on test instance

https://gerrit.wikimedia.org/r/942600

Change 942601 had a related patch set uploaded (by Slyngshede; author: Slyngshede):

[operations/puppet@production] Cloud IDP: Set profile format to flat for GitLab.

https://gerrit.wikimedia.org/r/942601

Change 942601 merged by Slyngshede:

[operations/puppet@production] Cloud IDP: Set profile format to flat for GitLab.

https://gerrit.wikimedia.org/r/942601

Change 942600 merged by Jelto:

[operations/puppet@production] gitlab: auto_sign_in_with openid_connect on test instance

https://gerrit.wikimedia.org/r/942600

The fix of using a FLAT profile with GitLab oidc was deployed to all idp servers (including wmcs/cloud). Thanks for @SLyngshede-WMF for finding this and all the help!

Replicas and test instance use OIDC by default now. Login works and was confirmed by other as well.

I'd like to switch production GitLab to OIDC on Monday. I'll send a short announcement mail.

Change 942646 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: auto_sign_in_with openid_connect on all instances

https://gerrit.wikimedia.org/r/942646

In https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/group-management/helpers.py#L223 the ldap group sync bot (T319211) creates new gitlab users with "provider": "cas3". Will that need to be updated during this transition?

Change 942646 merged by Jelto:

[operations/puppet@production] gitlab: auto_sign_in_with openid_connect on all instances

https://gerrit.wikimedia.org/r/942646

OIDC is enabled instance-wide now.

In T320390#9052117, @dancy wrote:

In https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/blob/main/group-management/helpers.py#L223 the ldap group sync bot (T319211) creates new gitlab users with "provider": "cas3". Will that need to be updated during this transition?

Thanks for the hint! We most probably want to update this to "provider": "openid_connect". I'll upload a patch for that.

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/38

use provider openid_connect for new synced ldap users

Change 943563 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: remove cas support

https://gerrit.wikimedia.org/r/943563

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/38

use provider openid_connect for new synced ldap users

Change 943583 had a related patch set uploaded (by Ahmon Dancy; author: Ahmon Dancy):

[operations/puppet@production] gitlab: Use gitlab-settings v1.2.0

https://gerrit.wikimedia.org/r/943583

Change 943583 merged by Jelto:

[operations/puppet@production] gitlab: Use gitlab-settings v1.2.0

https://gerrit.wikimedia.org/r/943583

Change 944170 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: remove cas omniauth_provider

https://gerrit.wikimedia.org/r/944170

Change 944170 merged by Jelto:

[operations/puppet@production] gitlab: remove cas omniauth_provider

https://gerrit.wikimedia.org/r/944170

Noticed today that display names changed to using cn instead of uid (discussed back in T288392: GitLab uses 'real name' as username (rather than 'shell name' or an user-specified name)):

In T320390#9067911, @thcipriani wrote:

Noticed today that display names changed to using cn instead of uid (discussed back in T288392: GitLab uses 'real name' as username (rather than 'shell name' or an user-specified name)):

Further investigation seems to show that the change is only in the foreign key for the external provider. CAS used the Developer account's uid attribute (shellname) and OIDC is using cn (username). The accounts created in GitLab as a result of the authn appear to continue to use uid as the GitLab identity and cn as the display name. https://gitlab.wikimedia.org/pepepiton is an example of a GitLab account that has only ever used OIDC for authn.

To summarize discussion from Slack and libera.chat #wikimedia-gitlab:

• Apart from T343485, we don't believe this has affected end-user experience.
• <bd808> My current belief is that either cn and uid should be fine as the foreign identifier, but in the long term I believe that cn is a more unstable primary key than uid.
• uid is probably stable long term, because it's associated with unix users.
• cn is currently stable, because it's enmeshed in Gerrit, but we want to be able to change it eventually.

Given all that, we're pretty sure OIDC should be switched to rely on uid instead of cn.

Thanks for the troubleshooting @brennen and @bd808 .

I've done some tests changing oidc settings on the test instance, mostly the uid_field in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/gitlab/manifests/init.pp#79. I tried cn, preferred_username and name. But this doesn't seem to have an impact on the name I get in GitLab when logging in with OIDC. My testing capabilities are a bit limited, because I use jelto everywhere and my OIDC identity is just Jelto.
I created a test account named "Jelto Test". And I was not able to find the correct settings to use my username jelto-test instead of Jelto Test.

@jbond @SLyngshede-WMF do you have a idea how to change the name GitLab uses with OIDC back to the one used in cas? As far as I understand we want to use the shell name/uid(?) instead of the full name?

I think cn and uid are equally stable in practice:

• Our current account handling doesn't allow to change either after creation, e.g. Neil Shah-Quinn recently changed his account by creating a new one and moving over his permissions: https://phabricator.wikimedia.org/T337591
• The forthcoming IDM will also not allow it; moving to a new CN is essentially a new account and a change of the UID would necessitate changes to existing files in user's homes

And given that the login on CAS/SSO happens with the CN, using it actually appears less confusing to me.

In T320390#9068533, @MoritzMuehlenhoff wrote:

I think cn and uid are equally stable in practice:

• Our current account handling doesn't allow to change either after creation, e.g. Neil Shah-Quinn recently changed his account by creating a new one and moving over his permissions: https://phabricator.wikimedia.org/T337591

We stopped renaming Developer accounts because Gerrit made an internal account handling change that basically made it impossible for us to continue to allow them. Prior to that Gerrit complication it was actually relatively normal to perform an account rename on wikitech which manifested as a cn change in LDAP.

• The forthcoming IDM will also not allow it; moving to a new CN is essentially a new account and a change of the UID would necessitate changes to existing files in user's homes

Was there a pragmatic design reason for treating cn as an immutable value/primary key in your new system?

Account renaming is a common action on Wikimedia wikis. A lot of work has actually been put into supporting this feature in MediaWiki. It is confusing for our community when they find that Wikimedia's technical spaces do not have parity in this feature.

Maybe it makes sense to create a dedicated task to discuss the general usage and policies for developer account naming? The discussion is a bit far away from the actual GitLab OIDC migration.

As far as I understand login and registration of new accounts works fine and the correct username (uid?) is used (for example in https://gitlab.wikimedia.org/admin/users/randojr). But users may not want to have their full name (cn?) in GitLab displayed (for example Rando McRandomface Jr). So the uid should be used for username and display name?

In T320390#9076268, @Jelto wrote:

As far as I understand login and registration of new accounts works fine and the correct username (uid?) is used (for example in https://gitlab.wikimedia.org/admin/users/randojr). But users may not want to have their full name (cn?) in GitLab displayed (for example Rando McRandomface Jr). So the uid should be used for username and display name?

I guess this might be a thing, but it is not what the active discussion of the usage of cn in the system is. Instead the issue at hand is that the current configuration/implementation of the OIDC connector for GitLab uses the IDP provided Developer account's cn attribute as the "Identifier" for the provider. This value is a foreign key for finding the correct GitLab account to activate when an OIDC authentication happens. My contention is that cn should not be treated as a stable identifier as that blocks legitimate account renaming requests. These requests are actively blocked today because Gerrit's authn system also treats cn as a stable identifier, but they were possible prior to Gerrit changing its internal account storage to make cn a key value.

In T320390#9068521, @Jelto wrote:

@jbond @SLyngshede-WMF do you have a idea how to change the name GitLab uses with OIDC back to the one used in cas? As far as I understand we want to use the shell name/uid(?) instead of the full name?

FTR preferred_username should map to uid.

But users may not want to have their full name (cn?) in GitLab displayed (for example Rando McRandomface Jr). So the uid should be used for username and display name?

We currently use cn for display name. I think that part is fine to continue.

Change 943563 merged by Jelto:

[operations/puppet@production] gitlab: remove cas support

https://gerrit.wikimedia.org/r/943563

Change #1043247 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] idp: remove gitlab from the CAS protocol section

https://gerrit.wikimedia.org/r/1043247

Change #1043247 merged by Dzahn:

[operations/puppet@production] idp: remove gitlab from the CAS protocol section

https://gerrit.wikimedia.org/r/1043247