Page MenuHomePhabricator

Migrate the airflow-platform-eng database to Kubernetes
Closed, ResolvedPublic

Description

Following the instructions on https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes/Operations#Deploy_a_CloudnativePG_cluster_in_the_same_Kubernetes_namespace_as_Airflow_and_import_the_data

  • Create the postgresql-airflow-search S3 user.
  • Create the S3 bucket in which the PG data will be stored.
  • add the secrets
  • Add the helfile.d changes to add cloudnative-pg namespace and deploy (to be deployed)
  • deploy the pods
  • remove the secrets
  • remove db details from values file
  • redeploy airflow

Event Timeline

Gehel triaged this task as High priority.Nov 25 2024, 1:34 PM

Change #1125424 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] airflow-product-eng: migrate scheduler and db to k8s

https://gerrit.wikimedia.org/r/1125424

Instance owners have requested us to postpone our move to dse-k8s until they are done testing and running currently blocked pipelines related to https://phabricator.wikimedia.org/T387470 and https://phabricator.wikimedia.org/T386255

Stevemunene renamed this task from Migrate the airflow-product-eng database to Kubernetes to Migrate the airflow-platform-eng database to Kubernetes.Apr 2 2025, 11:52 AM
Stevemunene updated the task description. (Show Details)

Change #1125424 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-product-eng: migrate scheduler and db to k8s

https://gerrit.wikimedia.org/r/1125424

Change #1133397 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] airflow-platform-eng: remove db values used for db import

https://gerrit.wikimedia.org/r/1133397

Change #1133397 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-platform-eng: remove db values used for db import

https://gerrit.wikimedia.org/r/1133397

Change #1133420 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Enable the airflow-platform-eng instance to gitsync the analytics DAGs

https://gerrit.wikimedia.org/r/1133420

Change #1133420 merged by jenkins-bot:

[operations/deployment-charts@master] Enable the airflow-platform-eng instance to gitsync the analytics DAGs

https://gerrit.wikimedia.org/r/1133420