We have decided that we would like to migrate Airflow to Kubernetes.
The project proposal is here: [[https://docs.google.com/document/d/1peCptYVHtjVG825vXnZHOc2cicV3X83WT1u24iWIrHs/edit|Airflow: High-Availability Strategy]] (currently a Google Doc).
This epic ticket is intended to track the progress of the project. There are several phases or streams of work involved.
### Validate postgres operator and Ceph integration
[] Update ceph package versions to the most recent stable version T362993
[] Complete the Ceph container storage integration with the dse-k8s cluster T327259
[] Decide on which postgresql operator to use T362999
[] Build any requred container images for the postgresql operator
[] Create a helm chart for the postgresql operator and deploy it
[] Validate that postgresql clusters can be created as required
[] Test and validate the performance characteristics of postgresql clusters created by the operator
[] Test and validate the resilience and recovery characteristics of postgresql clusters created by the operator
### Adapt Airflow auth and DAG deployment method
[] Create an airflow container image using blubber/kokkuri T363000
[] Create an airflow chart that is appropriate to our needs T363001
[] Replicate airflow user group structure in LDAP T363003
[] Configure airflow authentication with OIDC
[] Decide how airflow DAG deployment will work under Kubernetes
[] Create DNS aliases for the new airflow instances
[] Allow inbound web traffic to the new airflow instances
### Migrate Airflow instances with LocalExecutor
The following list of instances assumes that we are not going to switch to a multi-tenant model, which might be practical with the new authentication scheme.
[] Migrate analytics_test airflow instance to dse-k8s
[] Migrate analytics airflow instance to dse-k8s
[] Migrate platform_eng airflow instance to dse-k8s
[] Migrate search airflow instance to dse-k8s
[] Migrate research airflow instance to dse-k8s
[] Migrate wmde airflow instance to dse-k8s
### Migrate the LocalExecutor to KubernetesExecutor
[] Decide between the KubernetesExecutor and/or LocalKubernetesExecutor
[] Migrate all instance to the new executor model