Page MenuHomePhabricator
Create Task
Maniphest T362788

Migrate Airflow to the dse-k8s cluster
Closed, ResolvedPublic
Actions

Description

We have decided that we would like to migrate Airflow to Kubernetes.

The project proposal is here: Airflow: High-Availability Strategy (currently a Google Doc).

This epic ticket is intended to track the progress of the project. There are several phases or streams of work involved.

Ensure that the cephosd cluster is production-ready

T364386: Validate postgres operator and Ceph integration

  • Update ceph package versions to the most recent stable version T362993
  • Complete the Ceph container storage integration with the dse-k8s cluster T327259
  • Decide on which postgresql operator to use T362999 - We have decided on https://cloudnative-pg.io/
  • Build the requred container images for the cloudnativepg postgresql operator T364795
  • Create a helm chart for the cloudnativepg postgresql operator T364797
  • Deploy the cloudnativepg postgresql operator
  • Validate that postgresql clusters can be created as required
  • Test and validate the performance characteristics of postgresql clusters created by the operator
  • Test and validate the resilience and recovery characteristics of postgresql clusters created by the operator

T364387: Adapt Airflow auth and DAG deployment method

  • Create an airflow container image using blubber/kokkuri T363000
  • Create an airflow chart that is appropriate to our needs T363001
  • Create an LDAP group to use for testing Airflow on k8s T363003
  • Create a git-sync container image to be used with airflow T368757
  • Configure airflow authentication with OIDC
  • Decide how airflow DAG deployment will work under Kubernetes T368033
  • Create DNS aliases for the new airflow instances
  • Allow inbound web traffic to the new airflow instances

T379267: Migrate the airflow webservers to Kubernetes

We will be migrating the Airflow webserver services first, whilst leaving the schedulers running on their existing hosts

  • Deploy a new test-k8s instance to dse-k8s
  • Migrate analytics_test airflow webserver to dse-k8s T374948
  • Migrate analytics airflow webserver to dse-k8s T378439
  • Migrate platform_eng airflow webserver to dse-k8s T378443
  • Migrate search airflow webserver to dse-k8s T378441
  • Migrate research airflow webserver to dse-k8s T378442
  • Migrate product analytics airflow webserver to dse-k8s T378440
  • Migrate wmde airflow webserver to dse-k8s T378438

T375871: Integrate Airflow with Kerberos

  • Deploy the kerberos token renewer in the Airflow chart T375875
  • Ensure that the filesystem permissions are correctly configured when Airflow jobs interact with Hadoop and HDFS T375895
  • Ensure that we can submit spark jobs via spark3-submit from airflow T377928
  • Validate that we can submit Spark jobs with Skein in Kubernetes T377602

T364389: Migrate the airflow scheduler components to Kubernetes

  • Decide between the KubernetesExecutor and/or LocalKubernetesExecutor
  • Add the airflow scheduler to its helm chart T368737
  • Migrate all instance to the new executor model

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Reduce the verbosity of pgbouncer logs in airflow deploymentsoperations/deployment-chartsmaster+3 -1
airflow: enable the support of multiple executorsoperations/deployment-chartsmaster+3 -3
airflow: reflect recent changes in MX server hostnamesoperations/deployment-chartsmaster+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedbrouberolT362788 Migrate Airflow to the dse-k8s cluster
 ResolvedBTullisT364386 Validate postgres operator and Ceph integration
 ResolvedbrouberolT362999 Decide on which postgresql operator to use
 ResolvedBTullisT327259 Enable the Container Storage Interface (CSI) and the Ceph CSI plugin on dse-k8s cluster
 ResolvedBTullisT364472 Assess the suitability of the upstream ceph-csi-rbd helm chart for deployment
 ResolvedBTullisT362993 Update cephosd100[1-5] with the most recent stable version of Ceph
 ResolvedbrouberolT364795 Build the required container images for the cloudnativepg postgresql operator
 ResolvedbrouberolT364797 Create a helm chart for the cloudnativepg postgresql operator
 ResolvedbrouberolT368240 Create a helm chart for cloudnative-pg clusters
 ResolvedbrouberolT372256 Validate PGBouncer integration and use of Poolers
 ResolvedbrouberolT372276 Validate that we can resize the ceph persistent volume
 ResolvedbrouberolT372278 Adjust the cluster default size and number of instances
 ResolvedbrouberolT372282 Validate manual backups
 ResolvedbrouberolT372283 Test a full-scale backup recovery.
 ResolvedbrouberolT372284 Create PostgreSQL monitors
ResolvedbrouberolT372285 Create a PostgreSQL/cloudnative-pg Grafana dashboard
 ResolvedbrouberolT372286 Test the integration of the airflow chart with the cloudnative-pg-cluster chart
 ResolvedbrouberolT372901 Validate that the PG master is automatically failed over
ResolvedBTullisT372908 Setup out-of-band bacula DB/PG backups
 ResolvedbrouberolT373503 Investigate regular crashes of CloudnativePG test-airflow-k8s cluster
 ResolvedbrouberolT374950 Design a process to migrate from an-db1001 to a cloudnativePG cluster
 ResolvedbrouberolT375846 Adjust WAL volume setting for cloudnative PG clusters
 ResolvedbrouberolT375850 Prevent a cloudnative PG cluster from being deployed without s3 credentials
 ResolvedbrouberolT375853 Allow the cloudnative PG s3 credentials to be updated on the fly
 ResolvedbrouberolT364387 Adapt Airflow auth and DAG deployment method
 ResolvedbkingT363000 Create an airflow container image using blubber/kokkuri
 ResolvedbkingT363001 Create a helm chart for airflow that is appropriate to our needs
 ResolvedBTullisT368738 Add pgbouncer to our airflow helm chart
 ResolvedbrouberolT384275 Bug: Updates to the pod templates configmap data does not change the configuration checksum
 ResolvedBTullisT363003 Create an LDAP group to use for testing Airflow on K8s
 ResolvedBTullisT368030 Create a pgbouncer image for use with Airflow -> PostgreSQL
 ResolvedbrouberolT368033 Design a suitable DAG deployment method
 ResolvedbkingT368757 Create a git-sync container image to be used with airflow
 ResolvedbrouberolT379279 Make it possible to select the DAG deployment method
 Resolved• StevemuneneT368760 Configure airflow webserver under Kubernetes to use OIDC authentication
 Resolved• StevemuneneT371209 Create airflow-test-k8s OIDC configuration
 ResolvedbrouberolT369098 Enable metrics collection for airflow-k8s
 ResolvedMoritzMuehlenhoffT375729 Create LDAP groups to use for OIDC permission mapping with corresponding airflow DAG Authors groups
OpenNoneT380059 Enable 2FA for Airflow UI
 DuplicateNoneT364388 Migrate Airflow instances with LocalExecutor
 ResolvedbrouberolT364389 Migrate the airflow scheduler components to Kubernetes
 ResolvedbrouberolT368737 Deploy airflow scheduler via helm chart
 ResolvedBTullisT375895 Ensure that the filesystem permissions are correctly configured when Airflow jobs interact with Hadoop and HDFS
 ResolvedbrouberolT377928 Ensure that we can submit spark jobs via `spark3-submit` from airflow
ResolvedbrouberolT377602 Validate that we can submit Spark jobs with Skein in Kubernetes
ResolvedbrouberolT377927 Enable the airflow task/scheduler pods to communicate with our Hadoop clusters
 DeclinedbrouberolT378920 Determine whether or not we need expose an internal service for yarn
ResolvedbrouberolT379265 Cleanup the UNIX runuser accross all the airflow instances
ResolvedbrouberolT379266 Make sure we install python 3.10 in our airflow worker image
 ResolvedbrouberolT380284 Migrate the airflow-analytics-test scheduler to Kubernetes
 ResolvedbrouberolT380619 Migrate the airflow-analytics scheduler to Kubernetes
 ResolvedamastilovicT386282 Migrate analytics Airflow DAGs to k8s Airflow deployment
 ResolvedbrouberolT389172 Decommission airflow-analytics
 ResolvedBTullisT380620 Migrate the airflow-research scheduler to Kubernetes
 ResolvedbkingT380621 Migrate the airflow-search scheduler to Kubernetes
 ResolveddcausseT386097 Re-enable drop_old_data_daily in airflow-search
 ResolvedbrouberolT380622 Migrate the airflow-wmde scheduler to Kubernetes
 ResolvedbrouberolT380623 Migrate the airflow-analytics-product scheduler to Kubernetes
 Resolved• StevemuneneT380624 Migrate the airflow-platform-eng scheduler to Kubernetes
 ResolvedmfossatiT390880 [L] Verify all Airflow DAGs after migration to Kubernetes
 ResolvedbrouberolT380733 Change the Airflow email From address so that it refers to the instance name instead of the k8s cluster
 ResolvedbrouberolT380765 Integrate Kerberos login with RestExternalTaskSensor
 ResolvedbrouberolT381234 Scheduler pods responding with a 503 to their health check were not restarted
ResolvedbrouberolT383430 Use the KubernetesPodOperator for tasks that require access to refine python scripts
 ResolvedgmodenaT384255 python_script_executor should run with KubernetesPodOperator
 ResolvedBTullisT383417 Create a container image for analytics/refinery to be used with Airflow tasks
 Resolved• StevemuneneT369582 Enable prometheus metrics on the cephosd cluster
 Resolved• StevemuneneT369583 Configure availability and health monitoring for the cephosd cluster
 ResolvedbrouberolT371208 Create remaining airflow DNS records to be used for access and oidc
 Resolved• StevemuneneT371210 Create the airflow-analytics-test domain and certs
 ResolvedBTullisT330152 Deploy ceph radosgw processes to data-engineering cluster
 ResolvedBTullisT330153 Configure Anycast load-balancing ceph radosgw services on the data-engineering cluster
 ResolvedBTullisT369634 Decide how to do DAG logging on dse-k8s
 ResolvedbrouberolT372787 Implement S3 based logging for Airflow tasks on dse-k8s
 DeclinedNoneT372788 Implement elasticsearch/opensearch based logging for Airflow on dse-k8s
 ResolvedbkingT373034 Build Opensearch2 Docker image for testing
 ResolvedbrouberolT372281 Configure scheduled backups and WAL archiving to use our S3 endpoint
 ResolvedbrouberolT373026 Configure an Airflow/Datahub connection
ResolvedbrouberolT373837 Create airflow dags specific to data platform SRE to help us test our setup
 ResolvedbrouberolT374938 Assign an individual S3 bucket for each airflow and cloudnative PG release
 ResolvedbrouberolT375715 Add maintenance DAGs to wmf_airflow_common
 Resolved• StevemuneneT375714 Cleanup obsolete DAGs from Datahub
 ResolvedbrouberolT375770 Enforce a s3 retention policy for Airflow dags
 ResolvedbrouberolT375878 Clean Airflow database of old entries
 ResolvedbrouberolT383860 Airflow database cleanup job only archives the deleted data
 ResolvedbrouberolT376699 Create a DAG that ensures that email alerting is working
 ResolvedbrouberolT375713 Fix Datahub -> Airflow backlinks
ResolvedBTullisT375739 Hide the "Airflow is accessible over the public internet" warning
 ResolvedBTullisT375871 Integrate Airflow with Kerberos
 ResolvedbrouberolT375872 Decide whether we will use our own kerberos-kinit image or the `airflow kerberos` command
 ResolvedbrouberolT375875 Deploy the kerberos token renewer in the Airflow chart
 Resolved• StevemuneneT375877 Issue a principal and a keytab to the airflow-test-k8s instance
 ResolvedbrouberolT375716 Ensure the Airflow API can be reached out to from within Kubernetes and is authenticated
 DeclinedbrouberolT377952 Ensure that the airflow API can only be called from our production subnet
ResolvedbrouberolT377178 Monitor the availability of the airflow pods
 Resolved• StevemuneneT377926 Enable the definition of pod-scoped "external services" network policies
Restricted Task
 ResolvedbrouberolT379267 Migrate the airflow webservers to Kubernetes
 ResolvedbkingT374948 Migrate airflow-analytics-test webserver to Kubernetes
 ResolvedbrouberolT374936 Allow Airflow subcomponents to be deployed to Kubernetes
ResolvedbrouberolT376385 Expose the airflow analytics hosts in external services
 ResolvedbkingT377601 Allow the selective enabling/disabling of each airflow component systemd service
 ResolvedbrouberolT378377 Fix intermitent envoy errors when accessing the airflow UI
 Resolved• StevemuneneT378438 Migrate airflow-wmde instance webserver to kubernetes
 Resolved• StevemuneneT378440 Migrate airflow-analytics-product instance webserver to kubernetes
 ResolvedbrouberolT378441 Migrate airflow-search instance webserver to kubernetes
 ResolvedbrouberolT378442 Migrate the airflow research instance webserver to dse-k8s
 ResolvedbrouberolT378443 Migrate the airflow platform-engineering instance webserver to dse-k8s
 ResolvedbrouberolT378439 Migrate the analytics airflow instance webserver to dse-k8s
 ResolvedbrouberolT379363 Point the test airflow instances to the test Hadoop cluster
 ResolvedbrouberolT379659 Airflow jobs warn that Native hadoop libraries cannot be loaded
ResolvedbrouberolT380590 Migrate the airflow databases to Kubernetes
 ResolvedbrouberolT380591 Migrate the airflow-analytics-test database to Kubernetes
 ResolvedbrouberolT380613 Migrate the airflow-wmde database to Kubernetes
 DeclinedbrouberolT380614 Migrate the airflow-analytics database to Kubernetes
 ResolvedbkingT380615 Migrate the airflow-search database to Kubernetes
 ResolvedBTullisT380616 Migrate the airflow-research database to Kubernetes
 Resolved• StevemuneneT380618 Migrate the airflow-platform-eng database to Kubernetes
 ResolvedbrouberolT380617 Migrate the airflow-analytics-product database to Kubernetes
 ResolvedbrouberolT380727 Restore original behavior for Airflow variable management
 ResolvedbrouberolT381160 Export the backups of cloudnative-pg databases to Bacula
 ResolvedbrouberolT381161 Export the postgresql-airflow-analytics-test backups to Bacula
 ResolvedbrouberolT381162 Export the postgresql-airflow-analytics-product backups to Bacula
 ResolvedbrouberolT381163 Export the postgresql-airflow-wmde backups to Bacula
 ResolvedbrouberolT381164 Export the postgresql-airflow-analytics backups to Bacula
 ResolvedbrouberolT381165 Export the postgresql-airflow-search backups to Bacula
 ResolvedbrouberolT381166 Export the postgresql-airflow-research backups to Bacula
 Resolved• StevemuneneT381167 Export the postgresql-airflow-platform-eng backups to Bacula
 ResolvedbrouberolT381264 Connectivity issues with the ceph CSI provisioner pods
 ResolvedbrouberolT383651 Scheduler pods are restarted due to the PG pooler DNS name being non-resolvable
 ResolvedbrouberolT385785 airflow-test-k8s logs are no longer stored in s3
 ResolvedbrouberolT386103 Monitor that Airflow task logs are being stored in S3
 ResolvedbrouberolT387700 Enable using the yarn CLI on Kubernetes
 Resolved• StevemuneneT391075 Airflow migration to DSE k8s - cleanup once complete
 ResolvedbrouberolT390941 Decommission Airflow VMs
 ResolvedbrouberolT395296 Cleanup previous deployment methodology
 ResolvedamastilovicT392244 Facilitate automatic artifact cache warming for airflow-dags artifacts
 Resolved• StevemuneneT390940 Update all airflow docs to reflect changes.
 Resolved• StevemuneneT391359 Cleanup deployment charts references
 ResolvedbrouberolT391657 Deprecate running python scripts via the BashOperator
 ResolvedamastilovicT393111 Update airflow-dags wmf_airflow_common library rename analytics to main
 ResolvedJAllemandouT394015 Remove `analytics` instance folder in airflow repo
 ResolvedBTullisT395495 Archive and decommission the analytics Airflow instance
 ResolvedbrouberolT395557 Re-purpose an-db100[1-2] as dse-k8s-worker nodes
 ResolvedBTullisT394398 Enable the use of LocalExecutor in addition to the KubernetesExecutor
 ResolvedBTullisT396197 Enable xcom functionality for pods using the KubernetesPodOperator

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
BTullis mentioned this in T304615: Airflow scheduler and webserver logs should be readable by airflow instance admins.Sep 27 2024, 11:16 PM
brouberol closed subtask T371208: Create remaining airflow DNS records to be used for access and oidc as Resolved.Sep 30 2024, 7:36 AM
BTullis closed subtask T375739: Hide the "Airflow is accessible over the public internet" warning as Resolved.Sep 30 2024, 10:25 AM
BTullis mentioned this in T376113: Investigate alerts coming from the analytics_product airflow instance.Oct 1 2024, 11:20 AM
BTullis mentioned this in T373630: Investigate recent rise in Unique Devices.Oct 1 2024, 1:38 PM
KCVelaga_WMF subscribed.Oct 1 2024, 2:33 PM
BTullis updated the task description. (Show Details)Oct 1 2024, 3:09 PM
mfossati subscribed.Oct 10 2024, 3:22 PM
brouberol changed the status of subtask T377178: Monitor the availability of the airflow pods from Open to In Progress.Oct 15 2024, 8:34 AM
BTullis updated the task description. (Show Details)Oct 16 2024, 9:35 AM
brouberol closed subtask T377178: Monitor the availability of the airflow pods as Resolved.Oct 16 2024, 4:40 PM
gerritbot added a comment.Oct 17 2024, 9:15 AM

Change #1081082 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: reflect recent changes in MX server hostnames

https://gerrit.wikimedia.org/r/1081082

gerritbot added a project: Patch-For-Review.Oct 17 2024, 9:15 AM
gerritbot added a comment.Oct 17 2024, 9:18 AM

Change #1081082 merged by Brouberol:

[operations/deployment-charts@master] airflow: reflect recent changes in MX server hostnames

https://gerrit.wikimedia.org/r/1081082

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2024, 9:30 AM
BTullis updated the task description. (Show Details)Oct 18 2024, 1:14 AM
• Stevemunene closed subtask T369582: Enable prometheus metrics on the cephosd cluster as Resolved.Oct 18 2024, 10:07 AM
BTullis closed subtask T364386: Validate postgres operator and Ceph integration as Resolved.Oct 18 2024, 10:57 AM
brouberol closed subtask T375715: Add maintenance DAGs to wmf_airflow_common as Resolved.Oct 22 2024, 11:59 AM
brouberol removed a subtask: T375716: Ensure the Airflow API can be reached out to from within Kubernetes and is authenticated.Oct 23 2024, 7:45 AM
brouberol removed a subtask: T377927: Enable the airflow task/scheduler pods to communicate with our Hadoop clusters.Oct 23 2024, 9:30 AM
• Stevemunene added a subtask: T378438: Migrate airflow-wmde instance webserver to kubernetes.Oct 29 2024, 8:08 AM
• Stevemunene added a subtask: T378440: Migrate airflow-analytics-product instance webserver to kubernetes.Oct 29 2024, 8:15 AM
• Stevemunene added a subtask: T378441: Migrate airflow-search instance webserver to kubernetes.
• Stevemunene added a subtask: T378442: Migrate the airflow research instance webserver to dse-k8s.
brouberol added a subtask: T378443: Migrate the airflow platform-engineering instance webserver to dse-k8s.Nov 5 2024, 1:46 PM
brouberol added a subtask: T378439: Migrate the analytics airflow instance webserver to dse-k8s.
brouberol removed a subtask: T378438: Migrate airflow-wmde instance webserver to kubernetes.Nov 7 2024, 3:25 PM
brouberol removed a subtask: T378440: Migrate airflow-analytics-product instance webserver to kubernetes.
brouberol removed a subtask: T378441: Migrate airflow-search instance webserver to kubernetes.
brouberol removed a subtask: T378442: Migrate the airflow research instance webserver to dse-k8s.
brouberol removed a subtask: T378443: Migrate the airflow platform-engineering instance webserver to dse-k8s.
brouberol removed a subtask: T378439: Migrate the analytics airflow instance webserver to dse-k8s.
BTullis updated the task description. (Show Details)Nov 8 2024, 10:13 AM
brouberol changed the status of subtask Restricted Task from Open to In Progress.Nov 8 2024, 10:29 AM
brouberol added a subtask: T379363: Point the test airflow instances to the test Hadoop cluster.Nov 8 2024, 1:49 PM
brouberol closed subtask T379363: Point the test airflow instances to the test Hadoop cluster as Resolved.Nov 12 2024, 10:38 AM
brouberol closed subtask Restricted Task as Resolved.Nov 12 2024, 4:55 PM
brouberol added a subtask: T379659: Airflow jobs warn that Native hadoop libraries cannot be loaded .Nov 12 2024, 5:39 PM
brouberol closed subtask T379659: Airflow jobs warn that Native hadoop libraries cannot be loaded as Resolved.Nov 13 2024, 6:59 AM
BTullis closed subtask T375871: Integrate Airflow with Kerberos as Resolved.Nov 13 2024, 1:27 PM
BTullis updated the task description. (Show Details)
mpopov updated the task description. (Show Details)Nov 13 2024, 8:06 PM
brouberol updated the task description. (Show Details)Nov 15 2024, 7:45 AM
brouberol updated the task description. (Show Details)
brouberol closed subtask T379267: Migrate the airflow webservers to Kubernetes as Resolved.Nov 18 2024, 1:44 PM
brouberol updated the task description. (Show Details)Nov 22 2024, 1:44 PM
brouberol changed the status of subtask T380727: Restore original behavior for Airflow variable management from Open to In Progress.Nov 25 2024, 11:25 AM
brouberol updated the task description. (Show Details)Nov 25 2024, 11:28 AM
brouberol closed subtask T380727: Restore original behavior for Airflow variable management as Resolved.Nov 26 2024, 3:45 PM
brouberol added a subtask: T381264: Connectivity issues with the ceph CSI provisioner pods.Dec 2 2024, 4:10 PM
brouberol closed subtask T377926: Enable the definition of pod-scoped "external services" network policies as Resolved.Dec 2 2024, 7:10 PM
brouberol closed subtask T381264: Connectivity issues with the ceph CSI provisioner pods as Resolved.Dec 11 2024, 9:36 AM
gerritbot added a comment.Dec 11 2024, 1:45 PM

Change #1102312 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: enable the support of multiple executors

https://gerrit.wikimedia.org/r/1102312

gerritbot added a project: Patch-For-Review.Dec 11 2024, 1:45 PM
gerritbot added a comment.Dec 12 2024, 10:50 AM

Change #1102312 merged by Brouberol:

[operations/deployment-charts@master] airflow: enable the support of multiple executors

https://gerrit.wikimedia.org/r/1102312

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2024, 11:31 AM
BTullis mentioned this in T383417: Create a container image for analytics/refinery to be used with Airflow tasks.Jan 10 2025, 3:36 PM
brouberol changed the status of subtask T383651: Scheduler pods are restarted due to the PG pooler DNS name being non-resolvable from Open to In Progress.Jan 20 2025, 2:08 PM
fgiunchedi mentioned this in T384214: Port or deprecate Icinga airflow checks.Jan 20 2025, 2:29 PM
brouberol closed subtask T383651: Scheduler pods are restarted due to the PG pooler DNS name being non-resolvable as Resolved.Jan 21 2025, 9:03 AM
brouberol changed the status of subtask T385785: airflow-test-k8s logs are no longer stored in s3 from Open to In Progress.Feb 6 2025, 11:28 AM
brouberol closed subtask T385785: airflow-test-k8s logs are no longer stored in s3 as Resolved.Feb 10 2025, 10:30 AM
brouberol added a subtask: T386103: Monitor that Airflow task logs are being stored in S3.Feb 11 2025, 2:48 PM
brouberol closed subtask T386103: Monitor that Airflow task logs are being stored in S3 as Resolved.Feb 11 2025, 3:52 PM
brouberol closed subtask T387700: Enable using the yarn CLI on Kubernetes as Resolved.Mar 11 2025, 1:04 PM
dr0ptp4kt subscribed.Mar 18 2025, 11:15 AM
Gehel removed a subtask: T390941: Decommission Airflow VMs.Apr 4 2025, 8:18 AM
Gehel removed a subtask: T390940: Update all airflow docs to reflect changes..Apr 4 2025, 8:49 AM
gerritbot added a comment.Apr 4 2025, 7:32 PM

Change #1134283 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Reduce the verbosity of pgbouncer logs in airflow deployments

https://gerrit.wikimedia.org/r/1134283

gerritbot added a project: Patch-For-Review.Apr 4 2025, 7:32 PM
gerritbot added a comment.Apr 7 2025, 9:49 AM

Change #1134283 merged by jenkins-bot:

[operations/deployment-charts@master] Reduce the verbosity of pgbouncer logs in airflow deployments

https://gerrit.wikimedia.org/r/1134283

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2025, 10:31 AM
brouberol closed this task as Resolved.Apr 10 2025, 3:09 PM
brouberol claimed this task.
brouberol closed subtask T364389: Migrate the airflow scheduler components to Kubernetes as Resolved.
brouberol closed subtask T380590: Migrate the airflow databases to Kubernetes as Resolved.
brouberol closed subtask T381160: Export the backups of cloudnative-pg databases to Bacula as Resolved.
brouberol closed subtask T364387: Adapt Airflow auth and DAG deployment method as Resolved.
brouberol moved this task from Quarterly Goals to Done on the Data-Platform-SRE board.
brouberol reopened this task as Open.Apr 10 2025, 3:11 PM
brouberol moved this task from Done to Quarterly Goals on the Data-Platform-SRE board.
BTullis mentioned this in T393603: Write a blog post about the recent Airflow migration to Kubernetes.May 7 2025, 2:24 PM
BTullis updated the task description. (Show Details)May 29 2025, 11:23 AM
kevinbazira mentioned this in T396495: Build model training pipeline for tone check using WMF ML Airflow instance.Jun 11 2025, 6:03 AM
brouberol closed subtask T396197: Enable xcom functionality for pods using the KubernetesPodOperator as Resolved.Jun 19 2025, 7:33 PM
BTullis closed subtask T394398: Enable the use of LocalExecutor in addition to the KubernetesExecutor as Resolved.Jul 3 2025, 4:13 PM
BTullis added a comment.Jul 4 2025, 4:45 PM

We're so close to closing this epic. How can we best get over the line?
Shall we move some tickets out to a follow-up tracking ticket, or can we just close this with some tasks still open?

brouberol added a comment.Jul 5 2025, 10:57 AM

I'd say:

brouberol closed this task as Resolved.Aug 4 2025, 4:21 PM
brouberol closed subtask T391075: Airflow migration to DSE k8s - cleanup once complete as Resolved.
brouberol moved this task from Backlog - project to Done on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.
brouberol edited projects, added Data-Platform-SRE (2025.07.26 - 2025.08.15); removed Data-Platform-SRE.
dr0ptp4kt mentioned this in T396672: Request for dedicated Airflow instance for WME.Aug 4 2025, 6:05 PM
BTullis moved this task from Done to Reported on the Data-Platform-SRE (2025.07.26 - 2025.08.15) board.Aug 20 2025, 10:32 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits