Migrate the airflow scheduler components to Kubernetes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Gehel
	May 7 2024, 1:15 PM

Description

As part of "Migrate Airflow to the dse-k8s cluster" - T362788

We will migrate the webserver components first, then migrate the schedulers afterwards, once we have carried out more testing.

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
airflow: enable port 8600 to be reached from Kubernetes	operations/puppet	production	+6 -0
global_config: open port 8600 (webserver) for airflow services	operations/puppet	production	+4 -0
Enable the usage of the Kubernetes executor	operations/deployment-charts	master	+162 -14
Specify a custom deploy clusterrole for airflow namespaces in dse	operations/deployment-charts	master	+40 -0

Customize query in gerrit

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
Don't propagate the container SPARK_HOME to the hadoop workers	repos/data-engineering/airflow-dags!929	brouberol	T364389	main
Ensure SPARK_HOME=/usr/lib/spark3	repos/data-engineering/airflow!43	brouberol	T364389	main
Install missing libhdfs	repos/data-engineering/airflow!42	brouberol	T364389	main
Install libsasl libraries (take 3)	repos/data-engineering/airflow!41	brouberol	T364389	main
Install libsasl libraries (take 2)	repos/data-engineering/airflow!40	brouberol	T364389	main
Install libsasl libraries	repos/data-engineering/airflow!39	brouberol	T364389	main
Add missing dependencies	repos/data-engineering/airflow!38	brouberol	T364389	main
Cleanup scheduler logs as part of the purge_old_logs_from_s3 DAG	repos/data-engineering/airflow-dags!928	brouberol	cleanup-scheduler-logs	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	brouberol	T362788 Migrate Airflow to the dse-k8s cluster
Resolved	brouberol	T364389 Migrate the airflow scheduler components to Kubernetes
Resolved	brouberol	T368737 Deploy airflow scheduler via helm chart
Resolved	BTullis	T375895 Ensure that the filesystem permissions are correctly configured when Airflow jobs interact with Hadoop and HDFS
Resolved	brouberol	T377928 Ensure that we can submit spark jobs via `spark3-submit` from airflow
Resolved	brouberol	T377602 Validate that we can submit Spark jobs with Skein in Kubernetes
Resolved	brouberol	T377927 Enable the airflow task/scheduler pods to communicate with our Hadoop clusters
Declined	brouberol	T378920 Determine whether or not we need expose an internal service for yarn
Resolved	brouberol	T379265 Cleanup the UNIX runuser accross all the airflow instances
Resolved	brouberol	T379266 Make sure we install python 3.10 in our airflow worker image
Resolved	brouberol	T380284 Migrate the airflow-analytics-test scheduler to Kubernetes
Resolved	brouberol	T380619 Migrate the airflow-analytics scheduler to Kubernetes
Resolved	amastilovic	T386282 Migrate analytics Airflow DAGs to k8s Airflow deployment
Resolved	brouberol	T389172 Decommission airflow-analytics
Resolved	BTullis	T380620 Migrate the airflow-research scheduler to Kubernetes
Resolved	bking	T380621 Migrate the airflow-search scheduler to Kubernetes
Resolved	dcausse	T386097 Re-enable drop_old_data_daily in airflow-search
Resolved	brouberol	T380622 Migrate the airflow-wmde scheduler to Kubernetes
Resolved	brouberol	T380623 Migrate the airflow-analytics-product scheduler to Kubernetes
Resolved	• Stevemunene	T380624 Migrate the airflow-platform-eng scheduler to Kubernetes
Resolved	mfossati	T390880 [L] Verify all Airflow DAGs after migration to Kubernetes
Resolved	brouberol	T380733 Change the Airflow email From address so that it refers to the instance name instead of the k8s cluster
Resolved	brouberol	T380765 Integrate Kerberos login with RestExternalTaskSensor
Resolved	brouberol	T381234 Scheduler pods responding with a 503 to their health check were not restarted
Resolved	brouberol	T383430 Use the KubernetesPodOperator for tasks that require access to refine python scripts
Resolved	gmodena	T384255 python_script_executor should run with KubernetesPodOperator
Resolved	BTullis	T383417 Create a container image for analytics/refinery to be used with Airflow tasks

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2024, 1:15 PM

Gehel triaged this task as High priority.May 7 2024, 1:15 PM

Gehel moved this task from Incoming to Epics on the Data-Platform-SRE board.

BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.May 13 2024, 10:24 PM

Gehel moved this task from Epics to Quarterly Goals on the Data-Platform-SRE board.Jul 9 2024, 12:49 PM

Gehel removed a project: Epic.

Change #1075165 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] WIP: enable Kubernetes executor

https://gerrit.wikimedia.org/r/1075165

gerritbot added a project: Patch-For-Review.Sep 24 2024, 10:54 AM

brouberol updated the task description. (Show Details)Sep 25 2024, 7:42 AM

As the Airflow scheduler will need to be able to create/get/list/delete/watch Pods as part of the task lifecycle, it will need to have the associated permissions through a dedicated ServiceAccount. However, experience has shown that we can't create Role or ClusterRole resources within a chart, as the deploy role cannot manage them. These resources must be defined in admin_ng.

Having talked to @JMeybohm, it appears that a middleground solution could be:

declare a airflow-scheduler ClusterRole in admin_ng
declare a airflow-<instance-name>-scheduler ServiceAccount and RoleBinding resources in the airflow chart, binding the airflow-scheduler ClusterRole to the airflow-<instance-name>-scheduler ServiceAccount

Change #1075508 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Deploy an airflow-scheduler ClusteRole to dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1075508

After some testing, it appears that the deploy user cannot handle RoleBinding resources either, making the previously stated solution a dead end.

brouberol mentioned this in T375597: Debate whether we should grant the deploy ClusterRole permissions to create RoleBindings.Sep 25 2024, 8:40 AM

brouberol claimed this task.Sep 25 2024, 11:58 AM

brouberol edited projects, added Data-Platform-SRE (2024.09.06 - 2024.09.27); removed Data-Platform-SRE.

brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.09.06 - 2024.09.27) board.

Change #1075508 merged by Brouberol:

[operations/deployment-charts@master] Specify a custom deploy clusterrole for airflow namespaces in dse

https://gerrit.wikimedia.org/r/1075508

We have been able to trigger our first task to Kubernetes! Tailing of the task logs while it's still running is done via the Kube API, and the logs are then uploaded to s3.

Screenshot 2024-09-25 at 16.15.21.png (2×2 px, 551 KB)

Screenshot 2024-09-25 at 16.15.15.png (1×2 px, 420 KB)

Change #1075165 merged by Brouberol:

[operations/deployment-charts@master] Enable the usage of the Kubernetes executor

https://gerrit.wikimedia.org/r/1075165

Work has been done to allow us to migrate from LocalExecutor to KubernetesExecutor. I'm going to send this task back to our backlog, as the actual migration won't happen before a bit.

brouberol moved this task from In Progress to Backlog - project on the Data-Platform-SRE (2024.09.06 - 2024.09.27) board.Sep 26 2024, 9:13 AM

brouberol removed projects: Data-Platform-SRE (2024.09.06 - 2024.09.27), Patch-For-Review.

brouberol added a project: Data-Platform-SRE.Sep 26 2024, 9:23 AM

BTullis renamed this task from Migrate the LocalExecutor to KubernetesExecutor to Migrate the airflow scheduler components to Kubernetes.Sep 27 2024, 3:37 PM

BTullis updated the task description. (Show Details)

BTullis added a subtask: T368737: Deploy airflow scheduler via helm chart.

brouberol changed the status of subtask T377602: Validate that we can submit Spark jobs with Skein in Kubernetes from Open to In Progress.Oct 23 2024, 8:12 AM

brouberol added a subtask: T377927: Enable the airflow task/scheduler pods to communicate with our Hadoop clusters.Oct 23 2024, 9:30 AM

brouberol removed a subtask: T377602: Validate that we can submit Spark jobs with Skein in Kubernetes .Oct 23 2024, 9:34 AM

brouberol removed a subtask: T377927: Enable the airflow task/scheduler pods to communicate with our Hadoop clusters.Oct 23 2024, 9:47 AM

BTullis closed subtask T375895: Ensure that the filesystem permissions are correctly configured when Airflow jobs interact with Hadoop and HDFS as Resolved.Nov 6 2024, 10:39 AM

brouberol closed subtask T377928: Ensure that we can submit spark jobs via `spark3-submit` from airflow as Resolved.Nov 13 2024, 2:34 PM

BTullis mentioned this in T380258: Create an Airflow instance for ML.Nov 19 2024, 11:42 AM

brouberol changed the status of subtask T380284: Migrate the airflow-analytics-test scheduler to Kubernetes from Open to In Progress.Nov 19 2024, 4:01 PM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/928

Cleanup scheduler logs as part of the purge_old_logs_from_s3 DAG