The airflow-search migration is blocked by a failing data-purging DAG. Under the hood,
the dag runs a refinery script with BashOperator (via python_script_executor).
To be able to run refinery scripts k8s, we should add support for KubernetesPodOperator to python_script_executor.
AC:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| common: drop_older_then: check script only on bare metal. | repos/data-engineering/airflow-dags!1049 | gmodena | drop_older_than-path-check-1 | main | |
| common: drop_older_than: fix path existance check. | repos/data-engineering/airflow-dags!1047 | gmodena | drop_older_than-path-check | main | |
| Execute refinery scripts in a k8s pod. | repos/data-engineering/airflow-dags!1028 | gmodena | data-retention-with-k8s-operator | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | brouberol | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Resolved | brouberol | T364389 Migrate the airflow scheduler components to Kubernetes | |||
| Resolved | brouberol | T383430 Use the KubernetesPodOperator for tasks that require access to refine python scripts | |||
| Resolved | gmodena | T384255 python_script_executor should run with KubernetesPodOperator |
gmodena updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1028
Draft: wip: execute refinery scripts in a k8s pod.
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1028
Execute refinery scripts in a k8s pod.
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1047
common: drop_older_than: fix path existance check.
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1047
common: drop_older_than: fix path existance check.
gmodena opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1049
common: drop_older_then: check script only on bare metal.
gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1049
common: drop_older_then: check script only on bare metal.