Maniphest T393603

Write a blog post about the recent Airflow migration to Kubernetes
Open, MediumPublic
Actions

Assigned To

Authored By

	BTullis
	May 7 2025, 2:24 PM

Tags

Referenced Files

None

Subscribers

• VirginiaPoundstone

Description

The Data-Platform-SRE has been working on a migration of our Airflow services from host-based to Kubernetes, for around a year.

The project was originally defined in Airflow: High-Availability Strategy - A design for maximum uptime of our pipelines

The work was tracked in: T362788: Migrate Airflow to the dse-k8s cluster

We presented our work at FOSDEM 2025 in Enhancing Airflow for Analytics, Data Engineering, and ML at Wikimedia

Slides
Video

We have also presented a deep-dive session internally to the Data-Platform group: in DPE Deep Dive - Airflow on K8s

We believe that it would make a good post for the tech blog.

Details

Other Assignee: brouberol

Related Objects

Mentioned Here: T362788: Migrate Airflow to the dse-k8s cluster

Event Timeline

BTullis created this task.May 7 2025, 2:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 7 2025, 2:24 PM

BTullis triaged this task as Medium priority.May 7 2025, 2:29 PM

Ahoelzl added a project: Data-Platform-SRE.May 13 2025, 5:49 PM

Ahoelzl moved this task from Incoming (new tickets) to Needs Clarification on the Data-Engineering board.

@BTullis great idea to write a post for the tech blog! What support do you need to advance this?

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.May 16 2025, 9:00 AM

Gehel edited projects, added Data-Platform-SRE (2025.05.02 - 2025.05.23); removed Data-Platform-SRE.

Gehel edited projects, added Data-Platform-SRE (2025.05.24 - 2025.06.13); removed Data-Platform-SRE (2025.05.02 - 2025.05.23).May 23 2025, 1:08 PM

Gehel edited projects, added Data-Platform-SRE (2025.06.13 - 2025.07.04); removed Data-Platform-SRE (2025.05.24 - 2025.06.13).Jun 13 2025, 8:49 AM

BTullis edited projects, added Data-Platform-SRE (2025.07.05 - 2025.07.25); removed Data-Platform-SRE (2025.06.13 - 2025.07.04).Jul 4 2025, 3:59 PM

BTullis edited projects, added Data-Platform-SRE (2025.07.26 - 2025.08.15); removed Data-Platform-SRE (2025.07.05 - 2025.07.25).Jul 28 2025, 4:49 PM

Cool idea! Let DE know if you need anything from us!

BTullis edited projects, added Data-Platform-SRE (2025.08.16 - 2025.09.05); removed Data-Platform-SRE (2025.07.26 - 2025.08.15).Aug 20 2025, 10:28 AM

Gehel edited projects, added Data-Platform-SRE (2025.09.05 - 2025.09.26); removed Data-Platform-SRE (2025.08.16 - 2025.09.05).Sep 5 2025, 2:24 PM

Gehel added a project: Essential-Work.Sep 11 2025, 12:13 PM

Gehel edited projects, added Data-Platform-SRE (2025.09.26 - 2025.10.17); removed Data-Platform-SRE (2025.09.05 - 2025.09.26).Sep 26 2025, 1:51 PM

Gehel edited projects, added Data-Platform-SRE (2025.10.17 - 2025.11.07); removed Data-Platform-SRE (2025.09.26 - 2025.10.17).Oct 17 2025, 8:52 AM

Gehel edited projects, added Data-Platform-SRE (2025.11.07 - 2025.11.28); removed Data-Platform-SRE (2025.10.17 - 2025.11.07).Nov 7 2025, 1:32 PM

Gehel edited projects, added Data-Platform-SRE (2026.01.05 - 2026.01.23); removed Data-Platform-SRE (2025.11.07 - 2025.11.28).Jan 8 2026, 1:15 PM

Gehel edited projects, added Data-Platform-SRE (2026.01.23 - 2026.02.13); removed Data-Platform-SRE (2026.01.05 - 2026.01.23).Jan 23 2026, 10:05 AM

Gehel edited projects, added Data-Platform-SRE (2026-02-13 - 2026-03-06); removed Data-Platform-SRE (2026.01.23 - 2026.02.13).Feb 16 2026, 3:36 PM

Gehel edited projects, added Data-Platform-SRE (2026-03-06 - 2026-03-27); removed Data-Platform-SRE (2026-02-13 - 2026-03-06).Mar 6 2026, 2:04 PM

Gehel edited projects, added Data-Platform-SRE (2026-03-27 - 2026-04-17); removed Data-Platform-SRE (2026-03-06 - 2026-03-27).Mar 27 2026, 9:51 AM

Gehel edited projects, added Data-Platform-SRE (2026-04-24 - 2026-05-15); removed Data-Platform-SRE (2026-03-27 - 2026-04-17).Fri, Apr 24, 9:17 AM