This is a proposal to migrate our [[https://wikitech.wikimedia.org/wiki/Superset|Superset services]] to the [[https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters#dse-k8s|DSE kubernetes]] cluster.
Two instances of Superset are within scope:
* https://superset.wikimedia.org
* https://superset-next.wikimedia.org
Currently:
* superset is hosted on **bare-metal** on a single host: **an-tool1010.eqiad.wmnet**
* superset-next is hosted on a **VM** on on a single host: **an-tool1005.eqiad.wmnet**
They each use a discrete database on [[https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Mysql_Meta|analytics_meta]] MariaDB database on an-coord1001 for storing state.
They use have an instance of `memcached` local to the host, which is used for various metadata caching, but **not** query results caching.
The purpose of this ticket is to try to achieve consensus on the benefits, costs, and potential risks of moving Superset to the DSE Kubernetes cluster.
At this stage, we believe that the following steps will be required:
[x] Write a lightweight design document describing how the Superset services are intened to work on Kubernetes T349396
[x] Create a Superset container image using GitLab-CI and the [[https://gitlab.wikimedia.org/repos/releng/kokkuri|Blubber/Kokkuri]] framework. T352165
[] Create a helm chart for Superset T352166
[] Evaluate the [[https://github.com/apache/superset/tree/master/helm/superset|upstream chart]] and if appropriate use our [[https://wikitech.wikimedia.org/wiki/Kubernetes/Upstream_Helm_charts_policy|policy review]] to decide whether or not we should use it.
or:
[] Create a new helm chart for Superset using https://gitlab.wikimedia.org/repos/sre/sextant
[] Create two namespaces for superset and superset-next
[] Create two helmfile deployment for superset and superset-next
[] Create a keytab for each Superset deployment and make this available to the pods
[] Configure OIDC authentication for superset on dse-k8s.
[] Write a migration plan for Superset to K8S - including what to do about the legacy instances.
**n.b.** At present, we are not planning to move the metadata database (which is MariaDB in our case) to Kubernetetes at present.
The upstream helm charts declare a dependence on postgresql, which is what they tend to use with persistent vaolume claims, but for now we are not planning to use this.
We **do** have an option to migrate to PostgreSQL running on `an-db100[1-2]` (which is what Airflow uses) but we are not necessarily planning to take this option either.