Thanks @jbond
Adding a datahub_staging oidc entry with service_id: 'https://datahub-frontend\.k8s-staging\.discovery\.wmnet(/.*)?' which we access via tunnel and mainly use for testing.

Change 944231 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] idp_test: add datahub_staging as a OIDC service

https://gerrit.wikimedia.org/r/944231

Moving to blocked whilst we carry out T343236: Get datahub-staging.wikimedia.org working with the staging deployment of datahub

Change 944231 merged by Stevemunene:

[operations/puppet@production] idp_test: add datahub_staging as a OIDC service

https://gerrit.wikimedia.org/r/944231

Actively working on this, thus moving it back in progress as we plan on implementing the solutions defined on https://phabricator.wikimedia.org/T343236#9079448

Change 949516 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub: Enable OIDC to idp_test

https://gerrit.wikimedia.org/r/949516

These are the values added for our initial idp test and a brief explanation on each.

- name: AUTH_OIDC_ENABLED
  value: "true"
- name: AUTH_OIDC_CLIENT_ID
  value: "datahub_staging"
- name: AUTH_OIDC_CLIENT_SECRET
  valueFrom:
    secretKeyRef:
      name: {{ template "base.name.release" $ }}-secret-config
      key: auth_oidc_client_secret
- name: AUTH_OIDC_PRE_PROVISIONING_REQUIRED
  value: "false"
- name: AUTH_OIDC_DISCOVERY_URI
  value: "https://idp-test.wikimedia.org/oidc/.well-known"
- name: AUTH_OIDC_BASE_URL
  value: "https://datahub-frontend.k8s-staging.discovery.wmnet"
- name: AUTH_OIDC_USER_NAME_CLAIM
  value: "preferred_username"
- name: AUTH_OIDC_JIT_PROVISIONING_ENABLED
  value: "true"
- name: AUTH_OIDC_EXTRACT_GROUPS_ENABLED
  value: "true"

• AUTH_OIDC_ENABLED: Delegates authentication to a OIDC identity provider if set to true.
• AUTH_OIDC_CLIENT_ID: The client id (name) we used in idp_test. datahub_staging.
• AUTH_OIDC_CLIENT_SECRET: Unique client secret received from identity provider. This has already been added to srv private deployment charts.
• AUTH_OIDC_PRE_PROVISIONING_REQUIRED: Whether the user should already exist in DataHub when they login, failing login if they are not, this is appropriate for situations where users and groups are batch ingested and tightly controlled inside your environment. Defaults to false.
• AUTH_OIDC_DISCOVERY_URI: Location of the identity provider OIDC discovery API.
• AUTH_OIDC_BASE_URL= URL on which our datahub instance is served.
• AUTH_OIDC_USER_NAME_CLAIM: The attribute that will contain the username used on the DataHub platform. We will initially use preferred_username (also the default), this shall be adjusted incase it differs with what the IDP provides.
• AUTH_OIDC_JIT_PROVISIONING_ENABLED: Whether Datahub users & groups should be provisioned on login if they do not exist. Defaults to true.

To test this out: from deployment server cd to the datahub deployment charts and helmfile -e staging -i apply
With the right credentials, tail the staging env datahub front-ed container as we try access frontend locally as below.

Try ssh tunnelling to reach frontend on the right 443 port by.

1. Add our base url to etc/hosts as 127.0.0.1 datahub-frontend.k8s-staging.discovery.wmnet
2. Initiate a connection to staging datahub front-end via ssh -N -L 30443:k8s-ingress-staging.svc.eqiad.wmnet:30443 deploy1002.eqiad.wmnet
3. Due to the rules for privileged ports like 443, use socat to locally forward to 443 via the command sudo socat TCP-LISTEN:443,fork TCP:0.0.0.0:30443

I have a feeling that for AUTH_OIDC_USER_NAME_CLAIM we may want to use cn, or possibly uid here.

At the moment, we use uid from the LDAP query response as the user's user name, whereas users expect to be able to use their Wikitech username as their login name for most things.

We tried using the cn attribute as the username for DataHub with LDAP before, but it had issues when there were spaces in the attribute. Therefore, we have had to issue users this guidance about using their shell login name, rather than their Wikitech user name.

Just adding a bit more context

In T305874#9098561, @BTullis wrote:

I have a feeling that for AUTH_OIDC_USER_NAME_CLAIM we may want to use cn, or possibly uid here.

preferred_username maps to uid

At the moment, we use uid from the LDAP query response as the user's user name, whereas users expect to be able to use their Wikitech username as their login name for most things.

Users will login to idp.w.o with there CN regardless of what we have here.

The AUTH_OIDC_USER_NAME_CLAIM attribute will be what is used as the primary key/local username in the auth database for the local app. id recommend using preferred_username (aka uid) here because the CN is case insensitive. In ldap we enforce the case insensitivcity but its unknown if datahub would enforce this on the local database which could open up avenues for an impersonation attack. Regardless of what we use for the username claim, OIDC also passes the following attributes which can be used by the datahub app, depending on how things have been integrated

• name: maps to cn
• groups: maps to memberOf
• email: maps: to email
• family_name: maps to givenName
• preferred_username: maps to uid
• memberOf

That's really helpful. Thanks @jbond

preferred_username maps to uid

Great, so we can keep with the default of preferred_username and this will map to the uid, which is what we want.

id recommend using uid here because the CN is case insensitive. In ldap we enforce the case insensitivcity but its unknown if datahub would enforce this on the local database which could open up avenues for an impersonation attack.

Yes, that makes perfect sense.
In fact, we've already seen an issue like this related to case sensitivity with our LDAP/JAAS implementation. T309382: DataHub rights assignment is case-sensitive

When we start explicitly using uid from LDAP as the DataHub user name record, I'm pretty sure that this issue will go away, as well as the main issue we currently face, which is: T327884: Datahub user records are not being created after login

In T305874#9098663, @BTullis wrote:

Great, so we can keep with the default of preferred_username and this will map to the uid, which is what we want.

exactly

id recommend using uid here because the CN is case insensitive. In ldap we enforce the case insensitivcity but its unknown if datahub would enforce this on the local database which could open up avenues for an impersonation attack.

Yes, that makes perfect sense.
In fact, we've already seen an issue like this related to case sensitivity with our LDAP/JAAS implementation. T309382: DataHub rights assignment is case-sensitive

When we start explicitly using uid from LDAP as the DataHub user name record, I'm pretty sure that this issue will go away, as well as the main issue we currently face, which is: T327884: Datahub user records are not being created after login

ack, i have subscribed to both of those issues, but yes lets see how things look after the migration

Initiate a connection to staging datahub front-end via ssh -N -L 30443:k8s-ingress-staging.svc.eqiad.wmnet:30443 deploy1002.eqiad.wmnet

should this instead be ssh -N -L 443:k8s-ingress-staging.svc.eqiad.wmnet:30443 deploy1002.eqiad.wmnet assuming the service on k8s-ingress-staging. is on port 443

Due to the rules for privileged ports like 443, use socat to locally forward to 443 via the command sudo socat TCP-LISTEN:443,fork TCP:0.0.0.0:30443

At this point you are already hijacking all 443 traffic so it might be easier to use CAP_NET_BIND_SERVICE e.g.

$ setcap 'cap_net_bind_service=+ep' /usr/bin/ssh
$ ssh -N -L 443:k8s-ingress-staging.svc.eqiad.wmnet:443 deploy1002.eqiad.wmnet

In T305874#9099070, @jbond wrote:

At this point you are already hijacking all 443 traffic so it might be easier to use CAP_NET_BIND_SERVICE e.g.

$ setcap 'cap_net_bind_service=+ep' /usr/bin/ssh
$ ssh -N -L 443:k8s-ingress-staging.svc.eqiad.wmnet:443 deploy1002.eqiad.wmnet

This is a nifty solution. Thanks.

Change 949516 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: Enable OIDC to idp_test

https://gerrit.wikimedia.org/r/949516

Change 951128 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub: fix cidr typo

https://gerrit.wikimedia.org/r/951128

Change 951128 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: fix cidr typo

https://gerrit.wikimedia.org/r/951128

Got some errors from the first test, but they're mostly related to the current setup. Looking into this

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/kubernetes/datahub-deploy-staging.config
  Error: UPGRADE FAILED: an error occurred while rolling back the release. original upgrade error: failed to create patch: The order in patch list:
  [map[name:AUTH_JAAS_ENABLED value:false] map[name:AUTH_OIDC_CLIENT_SECRET valueFrom:map[secretKeyRef:map[key:auth_oidc_client_secret name:datahub-frontend-main-secret-config]]] map[name:AUTH_OIDC_CLIENT_SECRET valueFrom:map[secretKeyRef:map[key:auth_oidc_client_secret name:datahub-frontend-main-secret-config]]] map[name:AUTH_OIDC_ENABLED value:true] map[name:AUTH_OIDC_CLIENT_ID value:datahub_staging] map[name:AUTH_OIDC_PRE_PROVISIONING_REQUIRED value:false] map[name:AUTH_OIDC_DISCOVERY_URI value:https://idp-test.wikimedia.org/oidc/.well-known] map[name:AUTH_OIDC_BASE_URL value:https://datahub-frontend.k8s-staging.discovery.wmnet] map[name:AUTH_OIDC_USER_NAME_CLAIM value:preferred_username]]
   doesn't match $setElementOrder list:
  [map[name:SERVICE_IDENTIFIER] map[name:AUTH_JAAS_ENABLED] map[name:AUTH_NATIVE_ENABLED] map[name:AUTH_OIDC_CLIENT_SECRET] map[name:DATAHUB_ENCRYPTION_KEY] map[name:ELASTICSEARCH_PASSWORD] map[name:MYSQL_PASSWORD] map[name:TOKEN_SERVICE_SIGNING_KEY] map[name:ENABLE_PROMETHEUS] map[name:DATAHUB_GMS_HOST] map[name:DATAHUB_GMS_PORT] map[name:DATAHUB_SECRET] map[name:DATAHUB_APP_VERSION] map[name:DATAHUB_PLAY_MEM_BUFFER_SIZE] map[name:DATAHUB_ANALYTICS_ENABLED] map[name:KAFKA_BOOTSTRAP_SERVER] map[name:ELASTIC_CLIENT_HOST] map[name:ELASTIC_CLIENT_PORT] map[name:INDEX_PREFIX] map[name:DATAHUB_TRACKING_TOPIC] map[name:DATAHUB_GMS_USE_SSL] map[name:AUTH_OIDC_ENABLED] map[name:AUTH_OIDC_CLIENT_ID] map[name:AUTH_OIDC_CLIENT_SECRET] map[name:AUTH_OIDC_PRE_PROVISIONING_REQUIRED] map[name:AUTH_OIDC_DISCOVERY_URI] map[name:AUTH_OIDC_BASE_URL] map[name:AUTH_OIDC_USER_NAME_CLAIM] map[name:AUTH_OIDC_JIT_PROVISIONING_ENABLED] map[name:AUTH_OIDC_EXTRACT_GROUPS_ENABLED]]
  : no ConfigMap with the name "datahub-frontend-main-jaas" found

Change 951474 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Remove duplicate definition of oidc client secret from datahub

https://gerrit.wikimedia.org/r/951474

Change 951474 merged by jenkins-bot:

[operations/deployment-charts@master] Remove duplicate definition of oidc client secret from datahub

https://gerrit.wikimedia.org/r/951474

Change 951485 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub:chart version bump

https://gerrit.wikimedia.org/r/951485

Change 951485 merged by jenkins-bot:

[operations/deployment-charts@master] datahub:chart version bump

https://gerrit.wikimedia.org/r/951485

Was able to get the deployment to staging done, login redirected to the right SSO page and I was able to enter my login details, however authentication failed with this from the logs.

2023-08-22 14:23:28,435 [proxyClient-akka.actor.default-dispatcher-4] INFO  akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2023-08-22 14:27:18,936 [application-akka.actor.default-dispatcher-9] ERROR controllers.SsoCallbackController - Caught exception while attempting to handle SSO callback! It's likely that SSO integration is mis-configured.
java.util.concurrent.CompletionException: org.pac4j.core.exception.TechnicalException: Unsigned ID tokens are not allowed: they must be explicitly enabled on client side and the response_type used must return no ID Token from the authorization endpoint

Which is likely caused by the AUTH_OIDC_CLIENT_AUTHENTICATION_METHOD that defaults to client_secret_basic on the datahub sso config. client_secret_basic uses HTTP Basic authentication which is likely not supported by our idp. Changing this to client_secret_post, which includes the client_id and secret_id as form parameters in the HTTP POST request.

This is discussed in detail here 1.2. Client Type. Datahub details discussed on here #2-configure-datahub-frontend-server

Change 951518 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub: set the oidc client authentication method

https://gerrit.wikimedia.org/r/951518

Change 951518 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: set the oidc client authentication method

https://gerrit.wikimedia.org/r/951518

Change 951839 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub:main chart version bump

https://gerrit.wikimedia.org/r/951839

Change 951839 merged by jenkins-bot:

[operations/deployment-charts@master] datahub:main chart version bump

https://gerrit.wikimedia.org/r/951839

Added the AUTH_OIDC_CLIENT_AUTHENTICATION_METHOD method and retested, the idp seems okay with everything, my user is authenticated and provided a token

2023-08-23 11:35:37,980 INFO [org.apereo.inspektr.audit.support.Slf4jLoggingAuditTrailManager] - <Audit trail record BEGIN
=============================================================
WHO: stevemunene
WHAT: {access_token=AT-4-********qObqPBK38z-o01q9XsTu, scope=email openid profile, id_token=********..., token_type=Bearer, expires_in=28800}
ACTION: OAUTH2_ACCESS_TOKEN_RESPONSE_CREATED
APPLICATION: CAS
WHEN: Wed Aug 23 11:35:37 UTC 2023
CLIENT IP ADDRESS: *redacted*
SERVER IP ADDRESS: 127.0.0.1

However I am still getting the same error from datahub

2023-08-23 11:35:37,987 [application-akka.actor.default-dispatcher-8] ERROR controllers.SsoCallbackController - Caught exception while attempting to handle SSO callback! It's likely that SSO integration is mis-configured.
java.util.concurrent.CompletionException: org.pac4j.core.exception.TechnicalException: Unsigned ID tokens are not allowed: they must be explicitly enabled on client side and the response_type used must return no ID Token from the authorization endpoint

Looking into the response datahub expects from the idp.

Thanks @Stevemunene - Good work. I can confirm that I get the same errors as you when I attempt to log into staging.

Found some info on what we might be missing,
We have so far verified that authentication on the IDP side is okay and that we do receive a signed id token. The challenge lies in how datahub is validating the token received. We must use the same ID token JWS algorithm used by the IDP to validate the token received.

If an OpenID Provider supports multiple ID token crypto algorithms, the one which has been selected for the client at registration time must be used source.

We can do this by introducing the AUTH_OIDC_PREFERRED_JWS_ALGORITHM env variable with the algorithm our IDP uses, an example below:

- name: AUTH_OIDC_PREFERRED_JWS_ALGORITHM
  value: "RS256"

@jbond would you have the right value for us to use?

Some discussions on this on the datahub slack as well.

In case it helps, I did a little digging into the CAS logs on idp-test1002 and stumbled upon this.

root@idp-test1002:/var/log/cas# grep ERROR cas.log 

2023-08-23 09:27:29,477 ERROR [org.apereo.cas.authentication.principal.WebApplicationServiceFactory] - <Unable to extract query parameters from [https://datahub-frontend\.k8s-staging\.discovery\.wmnet(/.*)?]: [java.net.URISyntaxException: Illegal character in authority at index 8: https://datahub-frontend\.k8s-staging\.discovery\.wmnet(/.*)?]>

I wasn't looking for that, I was searching for algorithm and I also spotted these two lines, which suggest that RS256 is what's in use.

2023-08-23 09:27:29,775 DEBUG [org.apereo.cas.support.oauth.web.response.accesstoken.OAuth20AccessTokenAtHashGenerator] - <Signing algorithm specified is [RS256]>
2023-08-23 09:27:29,775 DEBUG [org.apereo.cas.support.oauth.web.response.accesstoken.OAuth20AccessTokenAtHashGenerator] - <Digesting access token hash via algorithm [SHA-256]>

In T305874#9114087, @BTullis wrote:

In case it helps, I did a little digging into the CAS logs on idp-test1002 and stumbled upon this.

root@idp-test1002:/var/log/cas# grep ERROR cas.log 

2023-08-23 09:27:29,477 ERROR [org.apereo.cas.authentication.principal.WebApplicationServiceFactory] - <Unable to extract query parameters from [https://datahub-frontend\.k8s-staging\.discovery\.wmnet(/.*)?]: [java.net.URISyntaxException: Illegal character in authority at index 8: https://datahub-frontend\.k8s-staging\.discovery\.wmnet(/.*)?]>

Thanks @BTullis , I am looking into this.

Change 952208 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub: set preferred oidc jwt algotithm

https://gerrit.wikimedia.org/r/952208

Change 952208 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: set preferred oidc jwt algotithm

https://gerrit.wikimedia.org/r/952208

Adding the AUTH_OIDC_PREFERRED_JWS_ALGORITHM worked and resolved the unsiged token error we had. Datahub can now receive the token from the idp and successfully read it.
However, we still cannot login due to this error (some details and tokens redacted)

2023-08-24 16:34:01,040 [application-akka.actor.default-dispatcher-9] ERROR controllers.SsoCallbackController - Caught exception while attempting to handle SSO callback! It's likely that SSO integration is mis-configured.
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to resolve user name claim from profile provided by Identity Provider. Missing attribute. Attribute: 'preferred_username', Regex: '(.*)', Profile: {at_hash=w9dGLDsaQZn5Z3LCEBu4ew, sub=Stevemunene, amr=["LdapAuthenticationHandler"], id_token=very long redacted token, iss=https://idp-test.wikimedia.org/oidc, client_id=datahub_staging, sid=redacted, access_token=AT-10-redated, token_expiration_advance=-1, aud=[datahub_staging], nbf=Thu Aug 24 16:29:00 UTC 2023, service=https://datahub-frontend.k8s-staging.discovery.wmnet/callback/oidc, auth_time=Thu Aug 24 16:33:58 UTC 2023, expiration=Fri Aug 25 00:34:00 UTC 2023, attributes={"name":"Stevemunene","preferred_username":"stevemunene","email":"myemail@wikimedia.org"}, id=Stevemunene, state=e0701001d7, exp=Fri Aug 25 00:34:00 UTC 2023, iat=Thu Aug 24 16:34:00 UTC 2023, jti=TGT-9-fJmWQ1qgEBv6XELZRzreS9U6SHHBenm60AK-l8gVf3XcLW4xQDzf9pmuqumDPXL09Co-idp-test1002}

Here we see Failed to resolve user name claim from profile provided by Identity Provider. Missing attribute. Attribute: 'preferred_username' where despite the the preferred_username being available in the token it is not visible to the datahub login service. This is the data that will be part of the authenticated users profile/details or similar.
We could solve this by

1. Expanding the authorization scope by the below to env and increasing the amount of info available on authorization in the UserInfo claims and outside the token.

- name: AUTH_OIDC_SCOPE
  value: "openid profile email"

This would also contribute to T309382 since the preffered_username:preferred_username value in this case in in lowercase.

or

2. Using the info already provided at the end here id=Stevemunene the id and setting that as the oidc username claim as below

- name: AUTH_OIDC_USER_NAME_CLAIM
  value: "id"

In the context of T309382 using id provides us with a username where the first letter is in uppercase.

To avoid a case of user duplication or similar I feel option 1 would be the better option for this. However, option 2 provides a good backup incase option 1 does not work as expected.

Great work on identifying and solving that blocker!

I'm not sure that I quite understand yet why you have suggested the three scopes that you have in option 1, but I agree that this is likely to be the better option to pursue, as opposed to option 2.

The other problem with using the id value is that might contain spaces. It can be very different from the username. I ran into a similar issue here. T307711#7924203

So I would recommend continuing with option 1 and if value: "openid profile email" is right then great. I'm just not sure why preferred_username isn't working though.

Also, it looks like we made a mistake here:
https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/949516/6/charts/datahub/charts/datahub-frontend/templates/_containers.tpl

We've hard-coded some values for staging into the datahub-frontend chart, where they should have been references to values. Apologies that I didn't pick this up at the review stage.

Could you possibly update this so that it refers to values, with suitable 'if guards'? You can refer to the way that the upstream charts use values here, in case it helps. https://github.com/acryldata/datahub-helm/blob/master/charts/datahub/subcharts/datahub-frontend/templates/deployment.yaml#L177

@Stevemunene @BTullis great work on the progress you have made and sorry for my silence. Just want to say +1 to using option 1. the id value you have there looks like the cn which has caused issues in the past so should be. avoided, further the scopes you are suggesting in option one are pretty standard soi no issue there

Edit: i just checked The changed linked above and i see you currently have prefered_username as the scope. but this is not a scope it is contained in the profile scope

Change 952842 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub: Add the oidc scope

https://gerrit.wikimedia.org/r/952842

Change 952842 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: Add the oidc scope

https://gerrit.wikimedia.org/r/952842

Expanding/adding the AUTH_OIDC_SCOPE doesn't seem to have had much impact on the SSO process, we are still getting the same error.

2023-08-28 14:18:01,556 [application-akka.actor.default-dispatcher-10] ERROR controllers.SsoCallbackController - Caught exception while attempting to handle SSO callback! It's likely that SSO integration is mis-configured.
java.util.concurrent.CompletionException: java.lang.RuntimeException: Failed to resolve user name claim from profile provided by Identity Provider. Missing attribute. Attribute: 'preferred_username', Regex: '(.*)', Profile: {at_hash=_H_2rIIQ3Kj9mp9ZPMMVwQ, sub=Stevemunene, amr=["LdapAuthenticationHandler"], id_token=*redacted*, iss=https://idp-test.wikimedia.org/oidc, client_id=datahub_staging, sid=*redacted*, access_token=*redacted*, token_expiration_advance=-1, aud=[datahub_staging], nbf=Mon Aug 28 14:13:01 UTC 2023, service=https://datahub-frontend.k8s-staging.discovery.wmnet/callback/oidc, auth_time=Mon Aug 28 14:17:59 UTC 2023, expiration=Mon Aug 28 22:18:01 UTC 2023, attributes={"name":"Stevemunene","preferred_username":"stevemunene","email":"myemail@wikimedia.org"}, id=Stevemunene, state=55ba64be24, exp=Mon Aug 28 22:18:01 UTC 2023, iat=Mon Aug 28 14:18:01 UTC 2023, jti=TGT-12-*redacted*-idp-test1002}

Auth related env variables for datahub-frontend are

      AUTH_JAAS_ENABLED:                       false
      AUTH_NATIVE_ENABLED:                     false
      AUTH_OIDC_ENABLED:                       true
      AUTH_OIDC_CLIENT_ID:                     datahub_staging
      AUTH_OIDC_PRE_PROVISIONING_REQUIRED:     false
      AUTH_OIDC_DISCOVERY_URI:                 https://idp-test.wikimedia.org/oidc/.well-known
      AUTH_OIDC_BASE_URL:                      https://datahub-frontend.k8s-staging.discovery.wmnet
      AUTH_OIDC_SCOPE:                         openid profile email
      AUTH_OIDC_USER_NAME_CLAIM:               preferred_username
      AUTH_OIDC_JIT_PROVISIONING_ENABLED:      true
      AUTH_OIDC_EXTRACT_GROUPS_ENABLED:        true
      AUTH_OIDC_CLIENT_AUTHENTICATION_METHOD:  client_secret_post
      AUTH_OIDC_PREFERRED_JWS_ALGORITHM:       RS256
}

One issue we ran into with Gitlab also involved Gitlab not being able to locate OIDC attributes. This was as a result of how CAS returns the attributes. By default CAS will return the attributes in a nested format, which almost nothing expects.

You can try switching the format to "FLAT" as with Gitlab, that might help datahub locate the attributes

Example from the IDP configuration for Gitlab:

gitlab_oidc:
  id: 31
  service_class: 'OidcRegisteredService'
  service_id: 'https://gitlab\.wikimedia\.org(/.*)?'
  profile_format: 'FLAT'

In T305874#9125628, @SLyngshede-WMF wrote:

One issue we ran into with Gitlab also involved Gitlab not being able to locate OIDC attributes. This was as a result of how CAS returns the attributes. By default CAS will return the attributes in a nested format, which almost nothing expects.

You can try switching the format to "FLAT" as with Gitlab, that might help datahub locate the attributes

Example from the IDP configuration for Gitlab:

gitlab_oidc:
  id: 31
  service_class: 'OidcRegisteredService'
  service_id: 'https://gitlab\.wikimedia\.org(/.*)?'
  profile_format: 'FLAT'

Thanks @SLyngshede-WMF, Trying this out since the attribute preferred_username is already available and from the docs https://apereo.github.io/cas/6.6.x/authentication/OAuth-Authentication-UserProfiles.html#user-profiles---oauth-authentication should be available when we change to FLAT. Great insight

Change 953192 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] idp_test: change datahub_staging profile_format

https://gerrit.wikimedia.org/r/953192

Change 953192 merged by Stevemunene:

[operations/puppet@production] idp_test: change datahub_staging profile_format

https://gerrit.wikimedia.org/r/953192

You can try switching the format to "FLAT" as with Gitlab, that might help datahub locate the attributes

Example from the IDP configuration for Gitlab:

gitlab_oidc:
  id: 31
  service_class: 'OidcRegisteredService'
  service_id: 'https://gitlab\.wikimedia\.org(/.*)?'
  profile_format: 'FLAT'

Thanks @SLyngshede-WMF, Trying this out since the attribute preferred_username is already available and from the docs https://apereo.github.io/cas/6.6.x/authentication/OAuth-Authentication-UserProfiles.html#user-profiles---oauth-authentication should be available when we change to FLAT. Great insight

With this, we now have successful login to Datahub-staging via SSO idp-test \o/ . Next is standardising the charts and process then moving on to production.

Change 954009 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] idp: add datahub as oidc service

https://gerrit.wikimedia.org/r/954009

Change 954132 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] datahub: add oidc production settings

https://gerrit.wikimedia.org/r/954132

Change 954132 merged by jenkins-bot:

[operations/deployment-charts@master] datahub: add oidc production settings

https://gerrit.wikimedia.org/r/954132

Mentioned in SAL (#wikimedia-analytics) [2023-09-18T09:43:32Z] <stevemunene> disable auth_jaas and native login to datahub then enable oidc authentication to production in codfw T305874

Mentioned in SAL (#wikimedia-analytics) [2023-09-18T09:51:06Z] <stevemunene> disable auth_jaas and native login to datahub then enable oidc authentication to production in eqiad T305874

Change 954009 merged by Stevemunene:

[operations/puppet@production] idp: add datahub as oidc service

https://gerrit.wikimedia.org/r/954009

Mentioned in SAL (#wikimedia-analytics) [2023-09-18T10:42:06Z] <stevemunene> deploy datahub in codfw to pick up new changes T305874

Mentioned in SAL (#wikimedia-analytics) [2023-09-18T10:45:17Z] <stevemunene> deploy datahub in eqiad to pick up new changes T305874

We successfully implemented OIDC on production datahub and auth/login seems to be working great.
However there are some challenges with the user journey during login and logout that are specific to datahub. Due to the change, we see a couple of different login screens depending on where/how we log in, these are the expected SSO login page and the previous JAAS login page.
login in via http://datahub.wikimedia.org/login takes us to the previous user login page ie.

AUTH_JAAS_ENABLED:                       false
AUTH_NATIVE_ENABLED:                     false

On logout, the user is also redirected to the previous login page, we might want to change the redirect on logout to go to a different screen.
Currently working on a way to correct this login/logout journey on datahub.

Change 961094 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Add OIDC/datahub stub secret

https://gerrit.wikimedia.org/r/961094

Change 961094 merged by Muehlenhoff:

[labs/private@master] Add OIDC/datahub stub secret

https://gerrit.wikimedia.org/r/961094