Change Details

As part of "Migrate Airflow to the dse-k8s cluster" - T362788

As part of "Migrate Airflow to the dse-k8s cluster" - T362788 Once Airflow is running on Kubernetes, we also need to change the way tasks are being scheduled. By relying on the `LocalExecutor`, we kick off jobs as subprocesses, leaving us open to killing tasks when restarting the scheduler pod. Airflow has several other types of remote Executor we can rely on (Celery and Kubernetes being part of them). In our case, we'd like to rely on the `KubernetesExecutor`, causing Airflow to run tasks by deploying Pods to the cluster (in the same namespace).

As part of "Migrate Airflow to the dse-k8s cluster" - T362788 Once Airflow is running on Kubernetes, we also need to change the way tasks are being scheduled. By relying on the `LocalExecutor`, we kick off jobs as subprocesses, leaving us open to killing tasks when restarting the scheduler pod. Airflow has several other types of remote Executor we can rely on (Celery and Kubernetes being part of them). In our case, we'd like to rely on the `KubernetesExecutor`, causing Airflow to run tasks by deploying Pods to the cluster (in the same namespace).