| AKhatun_WMF | |
| Jan 29 2026, 7:26 PM |
Currently a lot of jobs and pipelines are running in an-launcher with SystemD timers. Some are spark jobs, some are python/java/scala scripts etc. To better manage and debug these jobs, we want to move them to airflow.
This task is currently concerned with moving Data Engineering related timers to airflow.
List of timers: T414107#11549475
The doc that groups the timers and does an initial assessment of moving the timers to Airflow: Google Doc
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T415941 [EPIC] Move SystemD timer based jobs to Airflow | |||
| Resolved | AKhatun_WMF | T414107 Inventory of SystemD timer based jobs and pipelines | |||
| Open | None | T411999 Migrate cleanup jobs for snapshot datasets from systemd timers to Airflow | |||
| Open | None | T415944 Move HDFSCleaner timers to Airflow | |||
| Open | None | T415945 Move import jobs from SystemD timers to Airflow | |||
| Open | Antoine_Quhen | T405360 Implement an Airflow operator for moving data from point A to B | |||
| Open | None | T415947 Move leftover Refine related timers to Airflow | |||
| Open | None | T311739 [Iceberg] Update Refine Sanitize to insert into Iceberg tables | |||
| Open | None | T415949 Move Refinery timers to Airflow | |||
| Declined | None | T415951 Move Sqoop timers to Airflow | |||
| Open | amastilovic | T409514 Migrate Sqoop jobs to Airflow |