Create a container image for analytics/refinery to be used with Airflow tasks
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	BTullis
	Jan 10 2025, 3:36 PM

Description

During the work to T362788: Migrate Airflow to the dse-k8s cluster we discovered that a number of tasks that are currently using the BashOperator are attempting to run some scripts that are part of analytics/refinery.

These tasks do not work in the Kubernetes environment, because refinery is only deployed to certain target hosts (1, 2, 3) and also to HDFS.

One approach that we may wish to use is to create a refinery job artifact using the conda and WMF Workflow Utils based approach.

However, another approach that is available to us is to create a refinery container image. This will contain all of the scripts and libraries required, as well as the underlying CLI utilities such as hdfs, hive, yarn, mysql etc.

We will then be able to launch this using the KubernetesPodOperator from within a DAG.

Details

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
Fix the refinery python module path	repos/data-engineering/refinery!3	btullis	fix_refinery_path	main
Add the PYTHONPATH and update the PATH variables	repos/data-engineering/refinery!2	btullis	add_env_vars	main
Add data-engineering/refinery to the trusted runners	repos/releng/gitlab-trusted-runner!106	btullis	add_refinery	main
Add data-engineering/refinery to the trusted runners	repos/releng/gitlab-trusted-runner!104	btullis	main	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Resolved	brouberol	T362788 Migrate Airflow to the dse-k8s cluster
Resolved	brouberol	T364389 Migrate the airflow scheduler components to Kubernetes
Resolved	BTullis	T383417 Create a container image for analytics/refinery to be used with Airflow tasks

Event Timeline

BTullis created this task.Jan 10 2025, 3:36 PM

BTullis claimed this task.Jan 10 2025, 3:38 PM

BTullis triaged this task as High priority.

BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2025.01.11 - 2025.01.31) board.

btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/104

Add data-engineering/refinery to the trusted runners

BTullis mentioned this in T380621: Migrate the airflow-search scheduler to Kubernetes.Jan 10 2025, 4:18 PM

btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/106

Add data-engineering/refinery to the trusted runners

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/106

Add data-engineering/refinery to the trusted runners

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/refinery/-/merge_requests/2

Add the PYTHONPATH and update the PATH variables

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/refinery/-/merge_requests/2

Add the PYTHONPATH and update the PATH variables

btullis opened https://gitlab.wikimedia.org/repos/data-engineering/refinery/-/merge_requests/3

Fix the refinery python module path

btullis merged https://gitlab.wikimedia.org/repos/data-engineering/refinery/-/merge_requests/3

Fix the refinery python module path

This image is now published and usable.

btullis@marlin:~$ docker run -it docker-registry.wikimedia.org/repos/data-engineering/refinery:2025-01-10-165424-6efbd5adbcf50d11d3be1cd20542308cad6e7ae5
Unable to find image 'docker-registry.wikimedia.org/repos/data-engineering/refinery:2025-01-10-165424-6efbd5adbcf50d11d3be1cd20542308cad6e7ae5' locally
2025-01-10-165424-6efbd5adbcf50d11d3be1cd20542308cad6e7ae5: Pulling from repos/data-engineering/refinery
7b45c6d330c8: Already exists 
ef2ebb48f9ce: Already exists 
526e23257365: Already exists 
a8d6e7c24a3f: Already exists 
c2665232a772: Already exists 
4f4fb700ef54: Already exists 
56c4fcf0234b: Pull complete 
b3542690eb1a: Pull complete 
Digest: sha256:2952e9d4eb2ab6e7c49c1f0cec5a6fe77fc30af012a1f8ed52942954fec4b9c0
Status: Downloaded newer image for docker-registry.wikimedia.org/repos/data-engineering/refinery:2025-01-10-165424-6efbd5adbcf50d11d3be1cd20542308cad6e7ae5
runuser@6e65fef54633:/opt/refinery$ refinery-drop-older-than --help
Drops Hive partitions and removes data directories older than a threshold.

Usage: refinery-drop-older-than [options]

BTullis mentioned this in T383430: Use the KubernetesPodOperator for tasks that require access to refine python scripts.Jan 10 2025, 5:21 PM

Gehel edited parent tasks, added: T364389: Migrate the airflow scheduler components to Kubernetes; removed: T368927: [Epic] Migrate Data Platform Engineering maintained git repos to GitLab.Jan 17 2025, 10:06 AM

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.01.11 - 2025.01.31) board.Jan 17 2025, 10:14 AM

btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/111

Add data-engineering/refinery to the trusted runners

btullis opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/136

Add data-engineering/refinery to the trusted runners

bd808 closed https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/104

Add data-engineering/refinery to the trusted runners