JFTR, Puppet runs on idp-test1002 are currently failing, this is caused by the 'client_secret' that was pushed to the private repo, it needs a corresponding entry in hieradata/role/common/idp_test.yaml as well

In T343236#9067427, @MoritzMuehlenhoff wrote:

JFTR, Puppet runs on idp-test1002 are currently failing, this is caused by the 'client_secret' that was pushed to the private repo, it needs a corresponding entry in hieradata/role/common/idp_test.yaml as well

This was pushed together with 944231 which spawned some conversation on datahub-staging and possibly removing the datahub prod entry from idp_test and only having staging

Change 946851 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/dns@master] Add datahub_staging cname

https://gerrit.wikimedia.org/r/946851

We are probably going to have to ask for some guidance from the Service Ops team here, since I don't know of any other services that are actively using ingress on the staging cluster.
@JMeybohm or @akosiaris - can you think of any problems that we might face when trying to get this to work?

Namely, connecting up https://datahub-staging.wikimedia.org route to the frontend on: https://datahub-frontend.k8s-staging.discovery.wmnet

Are there any significant differences in the way that ingress works on staging, compared with the production wikikube clusters?
I know that there is guidance here: https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress#Add_a_new_service_under_Ingress but is there anything special that we need to consider if we would like this to work in staging?
Thanks.

IIUC you want to expose a staging service to the public internet. That is nothing we ever considered to do which is why there is not guidance.
I would strongly encourage to find a different solution for whatever the exact problem is with this (which I probably not fully understand as of now).

In T343236#9076462, @JMeybohm wrote:

IIUC you want to expose a staging service to the public internet. That is nothing we ever considered to do which is why there is not guidance.
I would strongly encourage to find a different solution for whatever the exact problem is with this (which I probably not fully understand as of now).

Thanks @JMeybohm - You raise a good point. I can try to explain what the problem is that we are facing and why we thought that adding an ingress route to the staging deployment of datahub was a reasonably good idea.
Maybe this will help us to work out together a better solution, as you suggest.

The immediate problem that we face is that we would like to:

1. enable OpenID Connect authentication for the DataHub web frontend, as opposed to LDAP via JAAS. Ref: T305874: Switch DataHub authentication to OIDC
2. be able to do this using the OIDC support of our CAS-SSO system, as has been achieved recently with Netbox and GitLab
3. be able to test the new OIDC authentication feature, before rolling it out to production

So we thought that it would be OK to enable the OIDC features on the staging deployment of datahub first, then promote them to production.
We can use the idp_test service with datahub staging and the production idp service with datahub production.

As a first step, we need to register a service_id URL to the staging idp service.
Our first attempt to do so for staging was here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/944231/1/hieradata/role/common/idp_test.yaml

However, it was pointed out by @jbond that our service_id of https://datahub-frontend.k8s-staging.discovery.wmnet may not work and might be difficult to use with SSH port forwarding.
We currently have to use SSH port forwarding along with a local edit to /etc/hosts in order to test the staging datahub frontend and we currently can't really test metadata ingestion through the staging gms service at all.

Adding https://datahub-staging.wikimedia.org to the CDN was put forward as a possible suggestion and it occurred to me that it could be a good way to solve these issues.
If you are strongly opposed to the idea of allowing any public-facing (even if authenticated by IDP) to the staging k8s cluster, then that's OK. We will just have to think of another way to solve the issue.

As I understand it the OIDC test can be worked around with by using an SSH forwarding (and editing the hosts file or simply manually re-issue the redirect request to the proper URL).
Can you please elaborate on the "metadata ingestion"/GMS issue a bit? What is the struggle with testing that?

Currently I would still oppose to making services on wikikube-staging publicly accessible as that is blurring a line we have right now. Also the intention of staging was/is to validate k8s deployments, not to do full fledged integration testing.

In T343236#9079372, @JMeybohm wrote:

Currently I would still oppose to making services on wikikube-staging publicly accessible as that is blurring a line we have right now. Also the intention of staging was/is to validate k8s deployments, not to do full fledged integration testing.

OK, I understand your concern. We'll find another way.

Maybe what we should do is:

• Move datahub to the dse-k8s cluster, deleting the deployments on wikikube and staging
• Create a datahub-test service (with its own namespace etc.) on dse-k8s and use this for our integration testing.
• Add https://datahub-test.wikimedia.org to the CDN and route it to this new service.

What do you think?

Can you please elaborate on the "metadata ingestion"/GMS issue a bit? What is the struggle with testing that?

Yes, gladly. It's all about end-to-end testing.

Currently, when we log into the staging deployment of datahub there is no content to be seen. i.e. There is no metadata on hive tables, kafka topics, druid tables, superset charts etc.
This means that there is limited utility in this staging development in verifying that everything is working as it should.

We have scheduled jobs in Airflow that do this regular metadata ingestion for the production DataHub, using the production Hive tables etc.
We could potentially have scheduled jobs that would allow us to to end-to-end validation of the staging deployment. We have a test hadoop cluster, with test airflow, test hive, test druid, test kafka etc. so it would seem like a good idea to link up these services with a test datahub service.

However, we haven't been able to do that at the moment because the GMS port on staging isn't exposed as the one on production is (at https://datahub-gms.discovery.wmnet:30443/)

As you mention that the staging cluster wasn't really intended for this kind of integration testing, but instead for validating kubernetes deployments, maybe this is the time to create a new service and to move it to dse-k8s.

In T343236#9068613, @Stevemunene wrote:

In T343236#9067427, @MoritzMuehlenhoff wrote:

JFTR, Puppet runs on idp-test1002 are currently failing, this is caused by the 'client_secret' that was pushed to the private repo, it needs a corresponding entry in hieradata/role/common/idp_test.yaml as well

This was pushed together with 944231 which spawned some conversation on datahub-staging and possibly removing the datahub prod entry from idp_test and only having staging

Removed the datahub_staging keys based on the discussions had and puppet is running ok on idptest1002 and 2002

In T343236#9079448, @BTullis wrote:

In T343236#9079372, @JMeybohm wrote:

Currently I would still oppose to making services on wikikube-staging publicly accessible as that is blurring a line we have right now. Also the intention of staging was/is to validate k8s deployments, not to do full fledged integration testing.

OK, I understand your concern. We'll find another way.

Maybe what we should do is:

• Move datahub to the dse-k8s cluster, deleting the deployments on wikikube and staging
• Create a datahub-test service (with its own namespace etc.) on dse-k8s and use this for our integration testing.
• Add https://datahub-test.wikimedia.org to the CDN and route it to this new service.

What do you think?

Can you please elaborate on the "metadata ingestion"/GMS issue a bit? What is the struggle with testing that?

Yes, gladly. It's all about end-to-end testing.

Currently, when we log into the staging deployment of datahub there is no content to be seen. i.e. There is no metadata on hive tables, kafka topics, druid tables, superset charts etc.
This means that there is limited utility in this staging development in verifying that everything is working as it should.

We have scheduled jobs in Airflow that do this regular metadata ingestion for the production DataHub, using the production Hive tables etc.
We could potentially have scheduled jobs that would allow us to to end-to-end validation of the staging deployment. We have a test hadoop cluster, with test airflow, test hive, test druid, test kafka etc. so it would seem like a good idea to link up these services with a test datahub service.

However, we haven't been able to do that at the moment because the GMS port on staging isn't exposed as the one on production is (at https://datahub-gms.discovery.wmnet:30443/)

As you mention that the staging cluster wasn't really intended for this kind of integration testing, but instead for validating kubernetes deployments, maybe this is the time to create a new service and to move it to dse-k8s.

Could we proceed with the OIDC test with the original SSH forwarding, since this might take some time to fully action?

In T343236#9082978, @Stevemunene wrote:

Could we proceed with the OIDC test with the original SSH forwarding, since this might take some time to fully action?

Yes, OK. If it's only a redirect as mentioned here then it might work for us. Feel free to keep trying.

In T343236#9079448, @BTullis wrote:

Maybe what we should do is:

• Move datahub to the dse-k8s cluster, deleting the deployments on wikikube and staging
• Create a datahub-test service (with its own namespace etc.) on dse-k8s and use this for our integration testing.
• Add https://datahub-test.wikimedia.org to the CDN and route it to this new service.

What do you think?

That sounds like a good plan to me as the dse cluster seems more a natural fit for datahub. IIRC we even said initially that it should move there when that cluster is ready to take production workloads.

In T343236#9079448, @BTullis wrote:

Can you please elaborate on the "metadata ingestion"/GMS issue a bit? What is the struggle with testing that?

Yes, gladly. It's all about end-to-end testing.

Currently, when we log into the staging deployment of datahub there is no content to be seen. i.e. There is no metadata on hive tables, kafka topics, druid tables, superset charts etc.
This means that there is limited utility in this staging development in verifying that everything is working as it should.

We have scheduled jobs in Airflow that do this regular metadata ingestion for the production DataHub, using the production Hive tables etc.
We could potentially have scheduled jobs that would allow us to to end-to-end validation of the staging deployment. We have a test hadoop cluster, with test airflow, test hive, test druid, test kafka etc. so it would seem like a good idea to link up these services with a test datahub service.

However, we haven't been able to do that at the moment because the GMS port on staging isn't exposed as the one on production is (at https://datahub-gms.discovery.wmnet:30443/)

As you mention that the staging cluster wasn't really intended for this kind of integration testing, but instead for validating kubernetes deployments, maybe this is the time to create a new service and to move it to dse-k8s.

So this is about making GMS reachable internally rather than exposing it to the internet, correct? IIRC you did configure a second ingress (datahub-gms.k8s-staging.discovery.wmnet) - so that should already work.

Change 946851 abandoned by Stevemunene:

[operations/dns@master] Add datahub_staging cname

Reason:

We are going to move the datahub deployment to the dse-k8s cluster and expose datahub-test from there. more details here https://phabricator.wikimedia.org/T343236#9079448

https://gerrit.wikimedia.org/r/946851