We have decided that we would like to migrate Airflow to Kubernetes.
The project proposal is here: Airflow: High-Availability Strategy (currently a Google Doc).
This epic ticket is intended to track the progress of the project. There are several phases or streams of work involved.
Validate postgres operator and Ceph integration
- Update ceph package versions to the most recent stable version T362993
- Complete the Ceph container storage integration with the dse-k8s cluster T327259
- Decide on which postgresql operator to use T362999
- Build any requred container images for the postgresql operator
- Create a helm chart for the postgresql operator and deploy it
- Validate that postgresql clusters can be created as required
- Test and validate the performance characteristics of postgresql clusters created by the operator
- Test and validate the resilience and recovery characteristics of postgresql clusters created by the operator
Adapt Airflow auth and DAG deployment method
- Create an airflow container image using blubber/kokkuri T363000
- Create an airflow chart that is appropriate to our needs T363001
- Replicate airflow user group structure in LDAP T363003
- Configure airflow authentication with OIDC
- Decide how airflow DAG deployment will work under Kubernetes
- Create DNS aliases for the new airflow instances
- Allow inbound web traffic to the new airflow instances
Migrate Airflow instances with LocalExecutor
The following list of instances assumes that we are not going to switch to a multi-tenant model, which might be practical with the new authentication scheme.
- Migrate analytics_test airflow instance to dse-k8s
- Migrate analytics airflow instance to dse-k8s
- Migrate platform_eng airflow instance to dse-k8s
- Migrate search airflow instance to dse-k8s
- Migrate research airflow instance to dse-k8s
- Migrate wmde airflow instance to dse-k8s
Migrate the LocalExecutor to KubernetesExecutor
- Decide between the KubernetesExecutor and/or LocalKubernetesExecutor
- Migrate all instance to the new executor model