Page MenuHomePhabricator
Create Task
Maniphest T361185

Move datahub to dse-k8s cluster
Closed, ResolvedPublic
Actions

Description

[WIP]
Datahub is currently hosted on the wikiKube cluster and we would like to move it to the dse-k8s cluster. This task is to track any and all work related to the move.

  • Preferably ensure we are on v0.12.1 pending the move to Java 17 before further upgrades T361688 T355211
  • Provision staging and production namespaces on dse-k8s T363298
  • Add kubaedm configs for the namespaces T363832
  • Move datahub and datahub-staging helfile deployments to dse-k8s T363300
  • Provision datahub-next subdomain T365576
  • Cleanup datahub staging/next oidc settings and links T365674
  • Point datahub and datahub-next subdomains to traffic server T365668
  • Create Internal service DNS record for datahub services pointing to dse-k8s T363299
  • deploy datahub (next and prod) to dse-k8s-eqiad
  • Adjust airflow DAGS to point to new endpoint datahub-gms.svc.eqiad.wmnet datahub-gms T366135
  • Cleanup datahub LVS and WikiKube endpoints T366137
  • Monitor the availability of datahub deployment on dse-k8s T363301
  • Create saved views for datahub deployment logs on dse-k8s T363304
  • Delete the WikiKube datahub release. T366338

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Gehel triaged this task as Medium priority.Apr 4 2024, 7:06 PM
Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE.
Gehel edited projects, added Data-Platform-SRE (2024.04.15 - 2024.05.05); removed Data-Platform-SRE (2024.03.25 - 2024.04.14).Apr 15 2024, 12:50 PM
Stevemunene added a subtask: T363298: Create datahub namespaces on dse-k8s.Apr 24 2024, 8:38 AM
Stevemunene added a subtask: T363299: create Internal service DNS record for datahub pointing to dse-k8s.
Stevemunene added a subtask: T363300: Move datahub and datahub-staging helfile deployments to dse-k8s.Apr 24 2024, 8:44 AM
Stevemunene added a subtask: T363301: Monitor the availability of datahub deployment on dse-k8s.Apr 24 2024, 8:50 AM
Stevemunene added a subtask: T363304: Create saved views for datahub deployment logs on dse-k8s.Apr 24 2024, 8:52 AM
Stevemunene updated the task description. (Show Details)Apr 25 2024, 11:26 AM
Stevemunene added a subtask: T363832: Setup kubeconfigs for datahub-next on dse-k8s.Apr 30 2024, 1:56 PM
Stevemunene closed subtask T363298: Create datahub namespaces on dse-k8s as Resolved.Apr 30 2024, 4:21 PM
BTullis mentioned this in T363843: Datahub mae consumer spamming logstash in wikikube staging.Apr 30 2024, 4:52 PM
Stevemunene mentioned this in T364001: Datahub occasional 500 errors.May 2 2024, 12:50 PM
Stevemunene updated the task description. (Show Details)May 2 2024, 1:27 PM
bking subscribed.May 2 2024, 1:43 PM
Gehel edited projects, added Data-Platform-SRE (2024.05.06 - 2024.05.26); removed Data-Platform-SRE (2024.04.15 - 2024.05.05).May 3 2024, 3:40 PM
BTullis moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.05.06 - 2024.05.26) board.May 3 2024, 4:40 PM
Stevemunene closed subtask T363832: Setup kubeconfigs for datahub-next on dse-k8s as Resolved.May 6 2024, 3:55 PM
Stevemunene added a subtask: T365576: provision datahub-next subdomain.May 22 2024, 11:17 AM
Stevemunene updated the task description. (Show Details)May 22 2024, 11:38 AM
Stevemunene added a subtask: T365668: Point datahub and datahub-next subdomains to traffic server.May 23 2024, 7:05 AM
Stevemunene updated the task description. (Show Details)May 23 2024, 7:44 AM
Stevemunene updated the task description. (Show Details)May 23 2024, 8:27 AM
Stevemunene added a subtask: T365674: Cleanup datahub staging/next oidc settings and links.May 23 2024, 8:30 AM
Stevemunene closed subtask T365576: provision datahub-next subdomain as Resolved.May 24 2024, 9:02 AM
Stevemunene added a comment.May 24 2024, 9:05 AM

Adding here for visibility,
Upon some discussion on how best we can implement the move to k8s, we are first going to get datahub-next up and running on dse-k8s then set a time to get datahub production up and running this will require some downtime of say an hour that shall be communicated with the team beforehand

Gehel edited projects, added Data-Platform-SRE (2024.05.27 - 2024.06.16); removed Data-Platform-SRE (2024.05.06 - 2024.05.26).May 24 2024, 12:20 PM
Gehel moved this task from Backlog to In Progress on the Data-Platform-SRE (2024.05.27 - 2024.06.16) board.
gerritbot added a comment.May 27 2024, 10:01 AM

Change #1036202 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Add dse range to an-test coordinator

https://gerrit.wikimedia.org/r/1036202

gerritbot added a project: Patch-For-Review.May 27 2024, 10:02 AM
gerritbot added a comment.May 27 2024, 10:06 AM

Change #1036202 merged by Stevemunene:

[operations/puppet@production] Add dse range to an-test coordinator

https://gerrit.wikimedia.org/r/1036202

Maintenance_bot removed a project: Patch-For-Review.May 27 2024, 10:31 AM
gerritbot added a comment.May 27 2024, 11:10 AM

Change #1036237 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] Enable mesh for datahub-next

https://gerrit.wikimedia.org/r/1036237

gerritbot added a project: Patch-For-Review.May 27 2024, 11:10 AM
gerritbot added a comment.May 27 2024, 11:25 AM

Change #1036237 merged by jenkins-bot:

[operations/deployment-charts@master] Enable mesh for datahub-next

https://gerrit.wikimedia.org/r/1036237

Maintenance_bot removed a project: Patch-For-Review.May 27 2024, 11:30 AM
Stevemunene closed subtask T365674: Cleanup datahub staging/next oidc settings and links as Resolved.May 27 2024, 12:21 PM
gerritbot added a comment.May 27 2024, 12:50 PM

Change #1036263 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub-next: make sure subcharts get the environment default values

https://gerrit.wikimedia.org/r/1036263

gerritbot added a project: Patch-For-Review.May 27 2024, 12:50 PM
gerritbot added a comment.May 27 2024, 12:54 PM

Change #1036263 merged by Brouberol:

[operations/deployment-charts@master] datahub-next: make sure subcharts get the environment default values

https://gerrit.wikimedia.org/r/1036263

gerritbot added a comment.May 27 2024, 1:16 PM

Change #1036266 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] datahub-next: fix the ingress by restoring default gateway host

https://gerrit.wikimedia.org/r/1036266

gerritbot added a comment.May 27 2024, 1:21 PM

Change #1036266 abandoned by Brouberol:

[operations/deployment-charts@master] datahub-next: fix the ingress by restoring default gateway host

Reason:

Looking at the CI job output, this is not what we want after all

https://gerrit.wikimedia.org/r/1036266

Maintenance_bot removed a project: Patch-For-Review.May 27 2024, 1:31 PM
gerritbot added a comment.May 28 2024, 4:59 AM

Change #1036346 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] Configure datahub-gms-next not to wait for upgrade before starting

https://gerrit.wikimedia.org/r/1036346

gerritbot added a project: Patch-For-Review.May 28 2024, 4:59 AM
gerritbot added a comment.May 28 2024, 7:37 AM

Change #1036346 merged by jenkins-bot:

[operations/deployment-charts@master] Configure datahub-gms-next not to wait for upgrade before starting

https://gerrit.wikimedia.org/r/1036346

gerritbot added a comment.May 28 2024, 8:23 AM

Change #1036557 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] admin_ng: create datahub-next namespace tlsHostnames and extra SANs

https://gerrit.wikimedia.org/r/1036557

gerritbot added a comment.May 28 2024, 9:40 AM

Change #1036557 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: create datahub-next namespace tlsHostnames and extra SANs

https://gerrit.wikimedia.org/r/1036557

Maintenance_bot removed a project: Patch-For-Review.May 28 2024, 10:31 AM
Stevemunene updated the task description. (Show Details)May 28 2024, 2:25 PM
Stevemunene updated the task description. (Show Details)May 28 2024, 2:33 PM
Stevemunene closed subtask T363300: Move datahub and datahub-staging helfile deployments to dse-k8s as Resolved.May 28 2024, 3:00 PM
Stevemunene added a subtask: T366135: Change datahub ingestion endpoint for all airflow DAGS.May 29 2024, 6:49 AM
CodeReviewBot added a project: Patch-For-Review.May 29 2024, 6:51 AM

stevemunene opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/710

Change the datahub ingestion url

Stevemunene updated the task description. (Show Details)May 29 2024, 7:19 AM
Stevemunene added a subtask: T366137: Remove datahub from LVS.May 29 2024, 7:48 AM
gerritbot added a comment.May 29 2024, 7:49 AM

Change #1036994 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Remove datahub from LVS

https://gerrit.wikimedia.org/r/1036994

Stevemunene added a comment.May 29 2024, 12:36 PM

Hello, as DPE-SRE we have been working on miving one of our services datahub to the dse-k8s cluster datahub is currently hosted on wikiKube and accessed via LVS.
For some of the final steps towards availing datahub-wikimedia.org we need to provision datahub service records and avail the new endpoints datahub-frontend.svc.eqiad.wmnet while simultaneously removing the WikiKube records datahub-frontend.discovery.wmnet and datahub-gms.discovery.wmnet. This might cause some warnings/alerts however we do have page: false.
Next would be to Remove datahub from LVS which would require some input from wikimedia-traffic

Stashbot added a comment.May 29 2024, 2:04 PM

Mentioned in SAL (#wikimedia-analytics) [2024-05-29T14:04:07Z] <stevemunene> getting started on moving datahub to dse-k8s T361185

CodeReviewBot added a comment.May 29 2024, 4:57 PM

stevemunene merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/710

Change the datahub ingestion url

Stevemunene closed subtask T366135: Change datahub ingestion endpoint for all airflow DAGS as Resolved.May 30 2024, 8:54 AM
Stevemunene closed subtask T365668: Point datahub and datahub-next subdomains to traffic server as Resolved.
Stevemunene closed subtask T363299: create Internal service DNS record for datahub pointing to dse-k8s as Resolved.
Stevemunene updated the task description. (Show Details)
gerritbot added a comment.May 30 2024, 11:08 AM

Change #1037471 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Remove datahub from LVS

https://gerrit.wikimedia.org/r/1037471

gerritbot added a comment.May 30 2024, 11:28 AM

Change #1037479 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Remove datahub service entry

https://gerrit.wikimedia.org/r/1037479

gerritbot added a comment.May 30 2024, 1:43 PM

Change #1036994 merged by Stevemunene:

[operations/puppet@production] Set datahub LVS to state lvs_setup

https://gerrit.wikimedia.org/r/1036994

gerritbot added a comment.May 30 2024, 2:40 PM

Change #1037471 merged by Stevemunene:

[operations/puppet@production] Set datahub LVS to service_setup

https://gerrit.wikimedia.org/r/1037471

gerritbot added a comment.May 31 2024, 7:17 AM

Change #1037479 merged by Stevemunene:

[operations/puppet@production] Remove datahub service entry

https://gerrit.wikimedia.org/r/1037479

Maintenance_bot removed a project: Patch-For-Review.May 31 2024, 7:31 AM
Stevemunene closed subtask T366137: Remove datahub from LVS as Resolved.May 31 2024, 7:39 AM
Stevemunene closed subtask T363450: Adapt datahub Puppet service to point to the eqiad datacenter only as Invalid.May 31 2024, 7:43 AM
Stevemunene added a subtask: T366338: Delete datahub WikiKube release references.May 31 2024, 8:13 AM
Stevemunene updated the task description. (Show Details)May 31 2024, 8:19 AM
brouberol awarded a token.May 31 2024, 10:00 AM
Stevemunene closed subtask T363301: Monitor the availability of datahub deployment on dse-k8s as Resolved.Jun 4 2024, 9:44 AM
Stevemunene closed subtask T363304: Create saved views for datahub deployment logs on dse-k8s as Resolved.
Stashbot added a comment.Jun 4 2024, 12:30 PM

Mentioned in SAL (#wikimedia-analytics) [2024-06-04T12:30:46Z] <stevemunene> delete WikiKube datahub release T361185

gerritbot added a comment.Jun 4 2024, 1:06 PM

Change #1038773 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] Clean up datahub from main cluster

https://gerrit.wikimedia.org/r/1038773

gerritbot added a project: Patch-For-Review.Jun 4 2024, 1:06 PM
Stevemunene updated the task description. (Show Details)Jun 4 2024, 1:09 PM
gerritbot added a comment.Jun 6 2024, 8:10 AM

Change #1038773 merged by jenkins-bot:

[operations/deployment-charts@master] Clean up datahub from main cluster

https://gerrit.wikimedia.org/r/1038773

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2024, 8:31 AM
gerritbot added a comment.Jun 6 2024, 9:00 AM

Change #1039618 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Delete datahub kubeconfigs on main

https://gerrit.wikimedia.org/r/1039618

gerritbot added a project: Patch-For-Review.Jun 6 2024, 9:00 AM
gerritbot added a comment.Jun 11 2024, 11:57 AM

Change #1039618 merged by Stevemunene:

[operations/puppet@production] Delete datahub kubeconfigs on main

https://gerrit.wikimedia.org/r/1039618

Stevemunene closed this task as Resolved.Jun 11 2024, 11:58 AM
Stevemunene closed subtask T366338: Delete datahub WikiKube release references as Resolved.
Stevemunene moved this task from In Progress to Done on the Data-Platform-SRE (2024.05.27 - 2024.06.16) board.
Maintenance_bot removed a project: Patch-For-Review.Jun 11 2024, 12:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits