Page MenuHomePhabricator
Create Task
Maniphest T347710

Migrate the Analytics Superset instances to our DSE Kubernetes cluster
Closed, ResolvedPublic
Actions

Description

This is a proposal to migrate our Superset services to the DSE kubernetes cluster.

Two instances of Superset are within scope:

Currently:

  • superset is hosted on bare-metal on a single host: an-tool1010.eqiad.wmnet
  • superset-next is hosted on a VM on on a single host: an-tool1005.eqiad.wmnet

They each use a discrete database on analytics_meta MariaDB database on an-coord1001 for storing state.

They use have an instance of memcached local to the host, which is used for various metadata caching, but not query results caching.

The purpose of this ticket is to try to achieve consensus on the benefits, costs, and potential risks of moving Superset to the DSE Kubernetes cluster.

At this stage, we believe that the following steps will be required:

  • Write a lightweight design document describing how the Superset services are intened to work on Kubernetes T349396
  • Create a Superset container image using GitLab-CI and the Blubber/Kokkuri framework. T352165
  • Apply our patches not yet merged upstream to the supserset codebase in our Docker image - T356477
  • Create a helm chart for Superset T352166
  • Create two namespaces for superset and superset-next: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/983718
  • Add kubeadm config files for the two new namespaces: https://gerrit.wikimedia.org/r/c/operations/puppet/+/983720
  • Create two helmfile deployment for superset and superset-next T353790
  • Ensure that we are running an up-to-date version of Superset, facilitating the migration T335356
  • Ensure necessary firewall rules are open between the DSE worker nodes and external services - T356623
  • Create a keytab for each Superset deployment and make this available to the pods
  • Make configuration secrets available to helmfile - T356480
  • Configure ingress internal DNS records - T356481
  • Add entries to the puppet service catalog - T356483
  • Update public domain DNS records to make them point to the DSE Kubernetes ingress - T356482
  • Configure OIDC authentication for superset on dse-k8s - T353794
  • Write a migration plan for Superset to K8S - including what to do about the legacy instances.
  • Monitor the availability of the superset deployments - T356484
  • Create saved views for the superset deployment logs - T356485
  • Update the wikitech page with our production readiness checklist - T356486
  • Find a solution for the requestctl-generator html page - T356490
  • Serve Superset static assets from an optimised container - T357890

n.b. At present, we are not planning to move the metadata database (which is MariaDB in our case) to Kubernetetes.
The upstream helm charts declare a dependence on postgresql, which is what they tend to use with persistent volume claims, but for now we are not planning to use this.

We do have an option to migrate to PostgreSQL running on an-db100[1-2] (which is what Airflow uses) but we are not necessarily planning to take this option either. We have decided to stick with MariaDB for now.

Details

SubjectRepoBranchLines +/-
Upgrade the platform_eng instance of airflow to puppet 7operations/puppetproduction+3 -0
Add kubeadm files for superset namespacesoperations/puppetproduction+16 -0
Add superset namespaces to the dse-k8s clusteroperations/deployment-chartsmaster+2 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedbrouberolT353782 Decommission an-tool1010
 ResolvedbrouberolT347710 Migrate the Analytics Superset instances to our DSE Kubernetes cluster
 ResolvedBTullisT349396 Write a design document relating to superset on dse-k8s
 ResolvedBTullisT352165 Create a superset container image using the PipelineLib framework
 ResolvedbrouberolT352166 Create a helm chart for Superset
 ResolvedbrouberolT353790 Create helmfile deployment files for superset and superset-next
 ResolvedbrouberolT353794 Configure OIDC Authentication for Superset on K8S
 ResolvedbrouberolT297120 Assign users' real names and email addresses to their Superset user accounts automatically
Restricted Task
 ResolvedbrouberolT356477 Apply our patches not yet merged upstream to the supserset codebase in our Docker image
 ResolvedbrouberolT356480 Make configuration secrets available to helmfile
ResolvedbrouberolT356481 Configure ingress internal DNS records
 ResolvedbrouberolT356482 [superset k8s] Update public domain DNS records to make them point to the DSE Kubernetes ingress
 ResolvedbrouberolT356483 [superset k8s] Add entries to the puppet service catalog
 ResolvedStevemuneneT356484 Monitor the availability of the superset deployments
 ResolvedStevemuneneT356485 Create saved views for the superset deployment logs
 ResolvedStevemuneneT356486 [superset k8s] Update the wikitech page with our production readiness checklist
 ResolvedbrouberolT356490 [superset-k8s] Find a solution for the requestctl-generator html page
 ResolvedbrouberolT356623 Ensure necessary firewall rules are open between the DSE worker nodes and external services
 ResolvedBTullisT357890 Serve Superset static assets from an optimised container
 ResolvedbrouberolT358479 Declare the superset domains as alternate domains
 ResolvedbrouberolT358480 Remove all resources associated with the superset-(next-)k8s.wimedia.org domains
 ResolvedbrouberolT358510 Filter out DEBUG kerberos logs from the container output
 ResolvedbrouberolT358569 Migrate bare-metal superset services over to Kubernetes
 ResolvedbrouberolT358570 Cleanup superset related resources from puppet
 ResolvedBTullisT358650 Create superset-admins LDAP group and populate with the current list of Admins from the production instance.
 ResolvedbrouberolT358706 Decommission an-tool1005
 DeclinedbrouberolT358753 Re-evaluate data cache security issues, along with per user data caching
 ResolvedbrouberolT358778 Monitor the Superset Kubernetes application
DeclinedNoneT294772 Superset Timeout Logging

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Gehel added a subtask: T297120: Assign users' real names and email addresses to their Superset user accounts automatically.Dec 6 2023, 1:51 PM
Gehel mentioned this in T341895: Deprecate Hue and stop the services.Dec 7 2023, 3:54 PM
BTullis mentioned this in T352534: [airflow] Inserting task notes is not working since upgrade to version 2.7.3.Dec 11 2023, 1:10 PM
BTullis closed subtask T352165: Create a superset container image using the PipelineLib framework as Resolved.Dec 14 2023, 10:37 AM
BTullis updated the task description. (Show Details)
BTullis updated the task description. (Show Details)Dec 18 2023, 1:41 PM
gerritbot added a comment.Dec 18 2023, 2:38 PM

Change 983718 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Add superset namespaces to the dse-k8s cluster

https://gerrit.wikimedia.org/r/983718

gerritbot added a project: Patch-For-Review.Dec 18 2023, 2:38 PM
BTullis updated the task description. (Show Details)Dec 18 2023, 2:44 PM
BTullis added a comment.EditedDec 18 2023, 2:53 PM

Evaluate the upstream chart and if appropriate use our policy review to decide whether or not we should use it.

I had a good look at this chart, but I don't believe that there is enough in it of value to make it worth our while using it.

There are some interesting points if we were to wish to use celery for asynchronous queries and websockets for background chart updates, but these features are not part of our current roadmap. Therefore, I think that we will be better served by using our own in-house deploment chart templates that are managed with sextant.

gerritbot added a comment.Dec 18 2023, 2:59 PM

Change 983720 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Add kubeadm files for superset namespaces

https://gerrit.wikimedia.org/r/983720

BTullis updated the task description. (Show Details)Dec 18 2023, 3:03 PM
Gehel added a parent task: T353782: Decommission an-tool1010.Dec 20 2023, 10:38 AM
BTullis updated the task description. (Show Details)Dec 20 2023, 10:55 AM
BTullis updated the task description. (Show Details)Dec 20 2023, 11:29 AM
gerritbot added a comment.Dec 21 2023, 3:05 PM

Change 983718 merged by jenkins-bot:

[operations/deployment-charts@master] Add superset namespaces to the dse-k8s cluster

https://gerrit.wikimedia.org/r/983718

Stashbot added a comment.Dec 21 2023, 3:36 PM

Mentioned in SAL (#wikimedia-analytics) [2023-12-21T15:36:11Z] <btullis> creating superset and superset-next namespace on dse-k8s for T347710

gerritbot added a comment.Dec 21 2023, 3:37 PM

Change 983720 merged by Btullis:

[operations/puppet@production] Add kubeadm files for superset namespaces

https://gerrit.wikimedia.org/r/983720

BTullis updated the task description. (Show Details)Dec 21 2023, 3:43 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 21 2023, 4:31 PM
Gehel added a subtask: Restricted Task.Jan 24 2024, 9:44 AM
BTullis updated the task description. (Show Details)Jan 24 2024, 5:28 PM
Volans subscribed.Jan 31 2024, 6:39 PM
brouberol closed subtask Restricted Task as Resolved.Feb 1 2024, 10:57 AM
brouberol updated the task description. (Show Details)Feb 2 2024, 8:24 AM
BTullis updated the task description. (Show Details)Feb 2 2024, 9:41 AM
Volans updated the task description. (Show Details)Feb 2 2024, 10:08 AM
gerritbot added a comment.Feb 2 2024, 10:16 AM

Change 995187 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Upgrade the platform_eng instance of airflow to puppet 7

https://gerrit.wikimedia.org/r/995187

gerritbot added a project: Patch-For-Review.Feb 2 2024, 10:16 AM
gerritbot added a comment.Feb 2 2024, 10:36 AM

Change 995187 merged by Muehlenhoff:

[operations/puppet@production] Upgrade the platform_eng instance of airflow to puppet 7

https://gerrit.wikimedia.org/r/995187

brouberol closed subtask T356477: Apply our patches not yet merged upstream to the supserset codebase in our Docker image as Resolved.Feb 2 2024, 4:36 PM
brouberol updated the task description. (Show Details)Feb 2 2024, 5:32 PM
brouberol updated the task description. (Show Details)Feb 5 2024, 10:56 AM
brouberol closed subtask T356480: Make configuration secrets available to helmfile as Resolved.Feb 5 2024, 3:34 PM
brouberol updated the task description. (Show Details)
brouberol closed subtask T356623: Ensure necessary firewall rules are open between the DSE worker nodes and external services as Resolved.Feb 6 2024, 2:09 PM
brouberol updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Feb 6 2024, 2:32 PM
brouberol updated the task description. (Show Details)Feb 8 2024, 5:06 PM
lbowmaker moved this task from Incoming (new tickets) to Radar (External Teams) on the Data-Engineering board.Feb 8 2024, 7:06 PM
Stashbot added a comment.Feb 9 2024, 1:47 PM

Mentioned in SAL (#wikimedia-analytics) [2024-02-09T13:47:03Z] <brouberol> deploying superset/superset-next services in dse-k8s-eqiad - T347710

Stashbot added a comment.Feb 9 2024, 2:01 PM

Mentioned in SAL (#wikimedia-analytics) [2024-02-09T14:01:12Z] <brouberol> superset was successfully deployed once the MySQL password was updated - T347710

brouberol closed subtask T356481: Configure ingress internal DNS records as Resolved.Feb 9 2024, 2:15 PM
brouberol updated the task description. (Show Details)Feb 9 2024, 2:53 PM
brouberol updated the task description. (Show Details)
brouberol closed subtask T352166: Create a helm chart for Superset as Resolved.Feb 12 2024, 8:52 AM
brouberol closed subtask T353790: Create helmfile deployment files for superset and superset-next as Resolved.
brouberol closed subtask T356482: [superset k8s] Update public domain DNS records to make them point to the DSE Kubernetes ingress as Resolved.Feb 12 2024, 12:45 PM
brouberol closed subtask T356483: [superset k8s] Add entries to the puppet service catalog as Resolved.
brouberol updated the task description. (Show Details)
brouberol reopened subtask T356481: Configure ingress internal DNS records as Open.Feb 12 2024, 1:53 PM
brouberol closed subtask T356481: Configure ingress internal DNS records as Resolved.Feb 14 2024, 2:45 PM
BTullis updated the task description. (Show Details)Feb 19 2024, 11:32 AM
BTullis closed subtask T357890: Serve Superset static assets from an optimised container as Resolved.Feb 26 2024, 10:33 AM
brouberol closed subtask T356485: Create saved views for the superset deployment logs as Resolved.Feb 26 2024, 11:30 AM
brouberol closed subtask T358479: Declare the superset domains as alternate domains as Resolved.Feb 26 2024, 5:08 PM
Stevemunene closed subtask T356486: [superset k8s] Update the wikitech page with our production readiness checklist as Resolved.Feb 27 2024, 8:19 AM
brouberol updated the task description. (Show Details)Feb 27 2024, 9:44 AM
brouberol closed subtask T353794: Configure OIDC Authentication for Superset on K8S as Resolved.Feb 27 2024, 11:29 AM
brouberol closed subtask T297120: Assign users' real names and email addresses to their Superset user accounts automatically as Resolved.
brouberol updated the task description. (Show Details)
brouberol closed subtask T358510: Filter out DEBUG kerberos logs from the container output as Resolved.Feb 27 2024, 11:47 AM
brouberol closed subtask T356490: [superset-k8s] Find a solution for the requestctl-generator html page as Resolved.Feb 27 2024, 2:10 PM
brouberol updated the task description. (Show Details)Feb 27 2024, 2:15 PM
BTullis added a comment.Feb 27 2024, 2:18 PM

Added the new kerberos principals.

root@krb1001:~# kadmin.local addprinc -randkey superset/superset-next.svc.eqiad.wmnet@WIKIMEDIA
root@krb1001:~# kadmin.local addprinc -randkey superset/superset.svc.eqiad.wmnet@WIKIMEDIA
BTullis added a comment.Feb 27 2024, 2:25 PM

Created keytab files:

root@krb1001:/srv/kerberos/keytabs# mkdir -p superset-next.svc.eqiad.wmnet/superset
root@krb1001:/srv/kerberos/keytabs# mkdir -p superset.svc.eqiad.wmnet/superset

root@krb1001:/srv/kerberos/keytabs# kadmin.local ktadd -norandkey -k /srv/kerberos/keytabs/superset-next.svc.eqiad.wmnet/superset/superset.keytab superset/superset-next.svc.eqiad.wmnet@WIKIMEDIA
Entry for principal superset/superset-next.svc.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/superset-next.svc.eqiad.wmnet/superset/superset.keytab.

root@krb1001:/srv/kerberos/keytabs# kadmin.local ktadd -norandkey -k /srv/kerberos/keytabs/superset.svc.eqiad.wmnet/superset/superset.keytab superset/superset.svc.eqiad.wmnet@WIKIMEDIA
Entry for principal superset/superset.svc.eqiad.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/srv/kerberos/keytabs/superset.svc.eqiad.wmnet/superset/superset.keytab.
BTullis added a comment.Feb 27 2024, 2:51 PM

We rolled this out to superset-next and then we updated the database settings in the UI as shown.

image.png (859×546 px, 68 KB)

BTullis updated the task description. (Show Details)Feb 27 2024, 2:52 PM
BTullis closed subtask T358650: Create superset-admins LDAP group and populate with the current list of Admins from the production instance. as Resolved.Feb 28 2024, 12:08 PM
BTullis added a subtask: T358706: Decommission an-tool1005.Feb 28 2024, 6:42 PM
brouberol updated the task description. (Show Details)Feb 29 2024, 8:33 AM
brouberol closed subtask T358706: Decommission an-tool1005 as Resolved.Feb 29 2024, 10:48 AM
brouberol closed subtask T358753: Re-evaluate data cache security issues, along with per user data caching as Declined.Feb 29 2024, 11:42 AM
brouberol closed subtask T358778: Monitor the Superset Kubernetes application as Resolved.Mar 1 2024, 8:38 AM
Gehel mentioned this in T294772: Superset Timeout Logging.Mar 6 2024, 1:41 PM
Gehel added a subtask: T294772: Superset Timeout Logging.
brouberol closed subtask T358569: Migrate bare-metal superset services over to Kubernetes as Resolved.Mar 20 2024, 12:21 PM
brouberol closed subtask T358480: Remove all resources associated with the superset-(next-)k8s.wimedia.org domains as Resolved.Mar 20 2024, 1:22 PM
brouberol added a comment.Mar 20 2024, 1:26 PM

https://superset.wikimedia.org is now served by a service running in dse-k8s-eqiad. We only have a couple of cleanup tasks to perform, to remove resources from Puppet. As far as users are concerned, this is done!

brouberol updated the task description. (Show Details)Mar 20 2024, 1:27 PM
Gehel moved this task from Quarterly Goals to 2024.03.25 - 2024.04.14 on the Data-Platform-SRE board.Mar 22 2024, 8:54 AM
Gehel edited projects, added Data-Platform-SRE (2024.03.25 - 2024.04.14); removed Data-Platform-SRE.
BTullis added a comment.Mar 22 2024, 5:30 PM
In T347710#9645603, @brouberol wrote:

https://superset.wikimedia.org is now served by a service running in dse-k8s-eqiad. We only have a couple of cleanup tasks to perform, to remove resources from Puppet. As far as users are concerned, this is done!

I agree. This epic is almost ready to be closed. Great work!

brouberol closed subtask T358570: Cleanup superset related resources from puppet as Resolved.Mar 25 2024, 12:57 PM
brouberol closed this task as Resolved.Mar 26 2024, 10:39 AM
brouberol claimed this task.
brouberol closed subtask T356484: Monitor the availability of the superset deployments as Resolved.
brouberol moved this task from Backlog to Done on the Data-Platform-SRE (2024.03.25 - 2024.04.14) board.
BTullis closed subtask T294772: Superset Timeout Logging as Declined.Mar 26 2024, 10:42 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL