Maniphest T379267

Migrate the airflow webservers to Kubernetes
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	brouberol
	Nov 7 2024, 3:24 PM

Tags

Referenced Files

None

Subscribers

Description

Umbrella task for all webserver migration tasks.

The overarching goal is to allow teams to:

reach their airflow UI via a public domain
manage roles and permissions via LDAP group management
get working links in alert emails

even if the scheduler and tasks still run on the airflow hosts.

Details

	Subject	Repo	Branch	Lines +/-
	airflow: define the webserver.base_url configuration	operations/puppet	production	+5 -0
	Add missing stub secrets for airflow_wmde and airflow_research	labs/private	master	+4 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T362788 Migrate Airflow to the dse-k8s cluster
		Resolved		brouberol	T379267 Migrate the airflow webservers to Kubernetes
		Resolved		bking	T374948 Migrate airflow-analytics-test webserver to Kubernetes
		Resolved		brouberol	T374936 Allow Airflow subcomponents to be deployed to Kubernetes
		Resolved		brouberol	T376385 Expose the airflow analytics hosts in external services
		Declined		bking	T377601 Allow the selective enabling/disabling of each airflow component systemd service
		Resolved		brouberol	T378377 Fix intermitent envoy errors when accessing the airflow UI
		Resolved		Stevemunene	T378438 Migrate airflow-wmde instance webserver to kubernetes
		Resolved		Stevemunene	T378440 Migrate airflow-analytics-product instance webserver to kubernetes
		Resolved		brouberol	T378441 Migrate airflow-search instance webserver to kubernetes
		Resolved		brouberol	T378442 Migrate the airflow research instance webserver to dse-k8s
		Resolved		brouberol	T378443 Migrate the airflow platform-engineering instance webserver to dse-k8s
		Resolved		brouberol	T378439 Migrate the analytics airflow instance webserver to dse-k8s

Event Timeline

brouberol created this task.Nov 7 2024, 3:24 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 7 2024, 3:24 PM

brouberol added a subtask: T374948: Migrate airflow-analytics-test webserver to Kubernetes.Nov 7 2024, 3:25 PM

brouberol added a subtask: T378438: Migrate airflow-wmde instance webserver to kubernetes.

brouberol added a subtask: T378440: Migrate airflow-analytics-product instance webserver to kubernetes.

brouberol added a subtask: T378441: Migrate airflow-search instance webserver to kubernetes.

brouberol added a subtask: T378442: Migrate the airflow research instance webserver to dse-k8s.

brouberol added a subtask: T378443: Migrate the airflow platform-engineering instance webserver to dse-k8s.

brouberol added a subtask: T378439: Migrate the analytics airflow instance webserver to dse-k8s.

Gehel moved this task from Incoming to Scratch on the Data-Platform-SRE board.Nov 8 2024, 9:59 AM

Gehel moved this task from Scratch to Incoming on the Data-Platform-SRE board.

BTullis triaged this task as High priority.Nov 8 2024, 10:13 AM

BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.

BTullis edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE.

Change #1090810 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] Add missing stub secrets for airflow_wmde and airflow_research

https://gerrit.wikimedia.org/r/1090810

gerritbot added a project: Patch-For-Review.Nov 13 2024, 10:10 AM

Change #1090810 merged by Muehlenhoff:

[labs/private@master] Add missing stub secrets for airflow_wmde and airflow_research

https://gerrit.wikimedia.org/r/1090810

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2024, 10:30 AM

brouberol closed subtask T378442: Migrate the airflow research instance webserver to dse-k8s as Resolved.Nov 13 2024, 3:18 PM

Stevemunene closed subtask T378438: Migrate airflow-wmde instance webserver to kubernetes as Resolved.Nov 13 2024, 3:44 PM

brouberol changed the status of subtask T378441: Migrate airflow-search instance webserver to kubernetes from Open to In Progress.Nov 14 2024, 8:17 AM

brouberol closed subtask T378441: Migrate airflow-search instance webserver to kubernetes as Resolved.Nov 14 2024, 4:36 PM

brouberol changed the status of subtask T378439: Migrate the analytics airflow instance webserver to dse-k8s from Open to In Progress.Nov 14 2024, 4:39 PM

Change #1091654 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow: define the webserver.base_url configuration

https://gerrit.wikimedia.org/r/1091654

gerritbot added a project: Patch-For-Review.Nov 15 2024, 12:53 PM

brouberol closed subtask T378439: Migrate the analytics airflow instance webserver to dse-k8s as Resolved.Nov 15 2024, 1:10 PM

brouberol claimed this task.Nov 15 2024, 3:45 PM

brouberol moved this task from Backlog - project to Needs Review on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

brouberol closed subtask T378443: Migrate the airflow platform-engineering instance webserver to dse-k8s as Resolved.Nov 15 2024, 5:47 PM

Change #1091654 merged by Brouberol:

[operations/puppet@production] airflow: define the webserver.base_url configuration

https://gerrit.wikimedia.org/r/1091654

Maintenance_bot removed a project: Patch-For-Review.Nov 18 2024, 1:31 PM

brouberol closed this task as Resolved.Nov 18 2024, 1:44 PM

brouberol closed subtask T378440: Migrate airflow-analytics-product instance webserver to kubernetes as Resolved.

brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 29 2024, 9:29 AM