Page MenuHomePhabricator
Create Task
Maniphest T374948

Migrate airflow-analytics-test webserver to Kubernetes
Closed, ResolvedPublic
Actions

Description

We can start delivering value to the teams even if we haven't figured everything out. By deploying the webserver pods to Kubernetes and integrating them with the an-db1001 database, we provide everyone with a public wikimedia.org subdomain and OIDC authentication, instead of relying on ssh tunneling to access the airflow UI.

For each instance:

  • allow ingress traffic to the airflow instances coming from the DSE_KUBEPODS subnet
  • Create the k8s namespaces
  • Create the k8s user kubeconfigs
  • Create the wikimedia.org public subdomains
  • Create the OIDC/CAS configuration
  • Create the config section in the private repo
  • Deploy the application
    • modify modules/profile/manifests/airflow.pp to support an optional secret secret_key and populate the webserver.secret_key config with it if found
    • add the secret key already found in /etc/helmfile-defaults/private/dse-k8s_services/airflow-analytics-test/dse-k8s-eqiad.yaml on the deployment secret to /srv/git/private/hieradata
  • Enable ATS traffic redirection and caching

Instances:

  • airflow-analytics-test

[ ] airflow-analytics
[ ] airflow-analytics-product
[ ] airflow-search
[ ] airflow-research
[ ] airflow-platform-eng
[ ] airflow-wmde

Update: I've reduced the scope to the initial test instance and will split out the production instances to their own tasks (or logically associate multiple instances in the same task, if that makes sense).

Procedure to follow: https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#Migrating_an_existing_instance

Details

Other Assignee
brouberol
SubjectRepoBranchLines +/-
airflow: make 'secret_key' configurableoperations/puppetproduction+16 -6
analytics_test_cluster: add secretlabs/privatemaster+3 -2
airflow-analytic-test: disable remote loggingoperations/deployment-chartsmaster+4 -2
airflow-analytic-test: fix OIDC client idoperations/deployment-chartsmaster+1 -1
airflow: disable network egress by defaultoperations/deployment-chartsmaster+1 -1
ATS: add mapping for airflow-analytics-testoperations/puppetproduction+5 -0
airflow: fix missing configmap when the scheduler isn't deployedoperations/deployment-chartsmaster+1 -3
airflow: remove custom network policy when the scheduler is running outside Kubernetesoperations/deployment-chartsmaster+1 -28
admin_ng (dse-k8s): remove unused namespacesoperations/deployment-chartsmaster+0 -24
Remove unused airflow kubernetes user credentialsoperations/puppetproduction+3 -51
airflow-analytics-test: correct oidc mappingoperations/deployment-chartsmaster+1 -1
airflow-analytic-test: comment out the postgresql deploymentoperations/deployment-chartsmaster+19 -15
airflow-analytics-test: define admin LDAP group -> role mappingoperations/deployment-chartsmaster+8 -2
idp.yaml: Add airflow serviceoperations/puppetproduction+8 -0
idp: add dummy client secret for aitflow_analytics_testlabs/privatemaster+2 -0
dse-k8s-services: add airflow helmfile directoryoperations/deployment-chartsmaster+100 -0
airflow: allow traffic to webserver port from dse-k8s podsoperations/puppetproduction+18 -1
admin-ng: add airflow namespaces to dse-k8s-eqiadoperations/deployment-chartsmaster+29 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 17 2024, 12:43 PM
brouberol added a subtask: T374936: Allow Airflow subcomponents to be deployed to Kubernetes .Sep 17 2024, 12:44 PM
brouberol reassigned this task from brouberol to bking.Sep 17 2024, 3:15 PM
brouberol added a comment.EditedSep 19 2024, 1:53 PM

Here's a brain dump of the things I think we need to do once:

  • allow ingress traffic to the airflow instances coming from the DSE_KUBEPODS subnet

Here are the things we need to do for each airflow instance.

  • Create the k8s namespaces
  • Create the k8s user kubeconfigs
  • Create the wikimedia.org public subdomains
  • Create the OIDC/CAS configuration
  • Create the config section in the private repo
dse-k8s:
    ...
    airflow-<instance-name>:
      dse-k8s-eqiad:
        config:
          private:
            airflow__core__fernet_key: <random 64 characters>
            airflow__webserver__secret_key: <random 64 characters>
          airflow:
            postgresqlPass: <PG password>
          oidc:
            client_secret: <OIDC client secret>
  • Create the airflow helmfile/values by taking inspiration from airflow-test-k8s. We need to specify the following values:
config:
  airflow:
    dbName: '<name of DB on an-db1001>'
    dbUser: '<user used on an-db1001>'
    dags_folder: <dags folder that should be cloned from https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/>
    instance_name: <dash separated name without the 'airflow-' prefix. Ex: analytics-product>

scheduler: 
  enabled: false # we don't currently run the scheduler in Kubernetes
  remote_host: <ip of the airflow instance>
  remote_port: <port of the airflow scheduler on the instance>

postgresql:
  cloudnative: false  # we don't currently run PG in Kubernetes

kerberos:
  enabled: false  # we don't currently run the Kerberos token renewer in Kubernetes. This config does not do anything atm, but will be useful in the future, when the chart supports it

external_services:
  postgresql: [analytics]

postgresql:
  cloudnative: false

ingress:
  gatewayHosts:
    default: "airflow-<instance-name>"
    extraFQDNs:
    - airflow-<instance-name>.wikimedia.org

oidc:
  client_id: <OIDC client id>

Feel free to copy the helmfile from helmfile.d/dse-k8s-services/airflow-test-k8s/ and adapt the namespace / instance

  • Deploy the application
  • Enable ATS traffic redirection and caching
NOTE: Instructions to create a new Airflow instance from scratch are available at https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#Creating_a_new_instance, but these are slightly different because we're migrating an existing instance.
brouberol added a comment.Sep 19 2024, 2:04 PM

We also need to think about how we assign Airflow roles, and to whom. The way that it works would be by creating LDAP groups, and mapping them to Airflow roles via the config.airflow.auth.role_mappings value.

Do we want to create an admin LDAP group for each airflow instance, and assign a couple of people to that group for each instance? Do we want to only have SRE be admins? Do we want anyone from nda and wmf to be able to access every Airflow instance?

I think needs to be figured out as well before we can call the task done.

brouberol closed subtask T374936: Allow Airflow subcomponents to be deployed to Kubernetes as Resolved.Sep 19 2024, 3:03 PM
Gehel triaged this task as High priority.Sep 20 2024, 7:53 AM
BTullis updated the task description. (Show Details)Sep 23 2024, 3:29 PM
gerritbot added a comment.Sep 24 2024, 5:21 PM

Change #1075278 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] admin-ng: add airflow namespaces to dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1075278

gerritbot added a project: Patch-For-Review.Sep 24 2024, 5:21 PM
gerritbot added a comment.Sep 24 2024, 9:48 PM

Change #1075321 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] airflow: allow traffic to webserver port from dse-k8s pods

https://gerrit.wikimedia.org/r/1075321

CodeReviewBot added a comment.Sep 25 2024, 7:22 AM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/841

test_k8s: define a dag that streams logs for a minute

CodeReviewBot added a comment.Sep 25 2024, 9:36 AM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/841

test_k8s: define a dag that streams logs for a minute

CodeReviewBot added a comment.Sep 25 2024, 12:22 PM

brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/842

Fix dag name

CodeReviewBot added a comment.Sep 25 2024, 12:31 PM

brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/842

Fix dag name

bking updated the task description. (Show Details)Sep 25 2024, 7:06 PM
BTullis added a parent task: T375729: Create LDAP groups to use for OIDC permission mapping with corresponding airflow DAG Authors groups .Sep 26 2024, 11:06 AM
gerritbot added a comment.Sep 26 2024, 6:46 PM

Change #1075278 merged by jenkins-bot:

[operations/deployment-charts@master] admin-ng: add airflow namespaces to dse-k8s-eqiad

https://gerrit.wikimedia.org/r/1075278

Gehel edited projects, added Data-Platform-SRE (2024.09.28 - 2024.10.18); removed Data-Platform-SRE (2024.09.06 - 2024.09.27).Sep 27 2024, 1:00 PM
BTullis mentioned this in T358137: Review the Airflow instance security settings to ensure that they are still suitable.Sep 27 2024, 3:13 PM
BTullis merged a task: T364388: Migrate Airflow instances with LocalExecutor.Sep 27 2024, 3:22 PM
BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.
BTullis removed a parent task: T362788: Migrate Airflow to the dse-k8s cluster.Sep 27 2024, 3:42 PM
gerritbot added a comment.Sep 27 2024, 4:56 PM

Change #1075321 merged by Bking:

[operations/puppet@production] airflow: allow traffic to webserver port from dse-k8s pods

https://gerrit.wikimedia.org/r/1075321

Maintenance_bot removed a project: Patch-For-Review.Sep 27 2024, 5:30 PM
brouberol updated the task description. (Show Details)Sep 30 2024, 7:55 AM
bking updated the task description. (Show Details)Sep 30 2024, 1:28 PM
gerritbot added a comment.Sep 30 2024, 2:34 PM

Change #1076775 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] dse-k8s-services: add airflow helmfile directory

https://gerrit.wikimedia.org/r/1076775

gerritbot added a project: Patch-For-Review.Sep 30 2024, 2:34 PM
gerritbot added a comment.Sep 30 2024, 3:35 PM

Change #1076775 abandoned by Bking:

[operations/deployment-charts@master] dse-k8s-services: add airflow helmfile directory

Reason:

will start over with recommended approach

https://gerrit.wikimedia.org/r/1076775

Maintenance_bot removed a project: Patch-For-Review.Sep 30 2024, 4:30 PM
brouberol closed subtask T376385: Expose the airflow analytics hosts in external services as Resolved.Oct 3 2024, 1:51 PM
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.09.28 - 2024.10.18) board.Oct 10 2024, 2:24 PM
gerritbot added a comment.Oct 10 2024, 2:25 PM

Change #1079292 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] idp.yaml: Add airflow service

https://gerrit.wikimedia.org/r/1079292

gerritbot added a project: Patch-For-Review.Oct 10 2024, 2:25 PM
gerritbot added a comment.Oct 10 2024, 2:37 PM

Change #1079296 had a related patch set uploaded (by Brouberol; author: Brouberol):

[labs/private@master] idp: add dummy client secret for aitflow_analytics_test

https://gerrit.wikimedia.org/r/1079296

gerritbot added a comment.Oct 10 2024, 2:39 PM

Change #1079296 merged by Bking:

[labs/private@master] idp: add dummy client secret for aitflow_analytics_test

https://gerrit.wikimedia.org/r/1079296

brouberol mentioned this in rLPRI2dad2e1f88f3: idp: add dummy client secret for aitflow_analytics_test.Oct 10 2024, 2:40 PM
gerritbot added a comment.Oct 10 2024, 2:48 PM

Change #1079292 merged by Bking:

[operations/puppet@production] idp.yaml: Add airflow service

https://gerrit.wikimedia.org/r/1079292

Maintenance_bot removed a project: Patch-For-Review.Oct 10 2024, 3:30 PM
bking updated the task description. (Show Details)Oct 10 2024, 7:18 PM
bking updated the task description. (Show Details)Oct 10 2024, 8:01 PM
gerritbot added a comment.Oct 10 2024, 9:30 PM

Change #1079361 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] ATS: add mapping for airflow-analytics-test

https://gerrit.wikimedia.org/r/1079361

gerritbot added a project: Patch-For-Review.Oct 10 2024, 9:30 PM
BTullis mentioned this in T375729: Create LDAP groups to use for OIDC permission mapping with corresponding airflow DAG Authors groups .Oct 15 2024, 11:02 AM
gerritbot added a comment.Oct 15 2024, 12:39 PM

Change #1080273 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflo-analytics-test: define admin LDAP group -> role mapping

https://gerrit.wikimedia.org/r/1080273

brouberol updated the task description. (Show Details)Oct 15 2024, 12:43 PM
gerritbot added a comment.Oct 15 2024, 1:56 PM

Change #1080273 merged by Brouberol:

[operations/deployment-charts@master] airflow-analytics-test: define admin LDAP group -> role mapping

https://gerrit.wikimedia.org/r/1080273

gerritbot added a comment.Oct 16 2024, 3:22 PM

Change #1080742 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytic-test: comment out the postgresql deployment

https://gerrit.wikimedia.org/r/1080742

gerritbot added a comment.Oct 16 2024, 5:11 PM

Change #1080742 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytic-test: comment out the postgresql deployment

https://gerrit.wikimedia.org/r/1080742

gerritbot added a comment.Oct 17 2024, 1:34 PM

Change #1081152 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] airflow-analytics-test: correct oidc mapping

https://gerrit.wikimedia.org/r/1081152

gerritbot added a comment.Oct 17 2024, 1:37 PM

Change #1081152 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytics-test: correct oidc mapping

https://gerrit.wikimedia.org/r/1081152

gerritbot added a comment.Oct 17 2024, 1:45 PM

Change #1081157 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Remove unused airflow kubernetes user credentials

https://gerrit.wikimedia.org/r/1081157

gerritbot added a comment.Oct 17 2024, 1:49 PM

Change #1081161 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] admin_ng (dse-k8s): remove unused namespaces

https://gerrit.wikimedia.org/r/1081161

gerritbot added a comment.Oct 17 2024, 1:49 PM

Change #1081157 merged by Bking:

[operations/puppet@production] Remove unused airflow kubernetes user credentials

https://gerrit.wikimedia.org/r/1081157

gerritbot added a comment.Oct 17 2024, 1:53 PM

Change #1081161 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng (dse-k8s): remove unused namespaces

https://gerrit.wikimedia.org/r/1081161

gerritbot added a comment.Oct 17 2024, 2:42 PM

Change #1081178 had a related patch set uploaded (by Bking; author: Bking):

[operations/deployment-charts@master] airflow: disable network egress by default

https://gerrit.wikimedia.org/r/1081178

gerritbot added a comment.Oct 17 2024, 4:10 PM

Change #1081200 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: remove custom network policy when the scheduler is running outside Kubernetes

https://gerrit.wikimedia.org/r/1081200

gerritbot added a comment.Oct 17 2024, 4:24 PM

Change #1081200 merged by Brouberol:

[operations/deployment-charts@master] airflow: remove custom network policy when the scheduler is running outside Kubernetes

https://gerrit.wikimedia.org/r/1081200

gerritbot added a comment.Oct 17 2024, 4:37 PM

Change #1081210 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: fix missing configmap when the scheduler isn't deployed

https://gerrit.wikimedia.org/r/1081210

gerritbot added a comment.Oct 17 2024, 4:39 PM

Change #1081210 merged by Brouberol:

[operations/deployment-charts@master] airflow: fix missing configmap when the scheduler isn't deployed

https://gerrit.wikimedia.org/r/1081210

gerritbot added a comment.Oct 17 2024, 4:43 PM

Change #1079361 merged by Brouberol:

[operations/puppet@production] ATS: add mapping for airflow-analytics-test

https://gerrit.wikimedia.org/r/1079361

gerritbot added a comment.Oct 17 2024, 5:45 PM

Change #1081228 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytic-test: fix OIDC client id

https://gerrit.wikimedia.org/r/1081228

gerritbot added a comment.Oct 17 2024, 5:50 PM

Change #1081178 abandoned by Bking:

[operations/deployment-charts@master] airflow: disable network egress by default

Reason:

no longer needed

https://gerrit.wikimedia.org/r/1081178

gerritbot added a comment.Oct 17 2024, 5:51 PM

Change #1081228 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytic-test: fix OIDC client id

https://gerrit.wikimedia.org/r/1081228

gerritbot added a comment.Oct 17 2024, 6:01 PM

Change #1081230 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytic-test: disable remote logging

https://gerrit.wikimedia.org/r/1081230

gerritbot added a comment.Oct 17 2024, 6:25 PM

Change #1081230 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytic-test: disable remote logging

https://gerrit.wikimedia.org/r/1081230

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2024, 6:31 PM
bking updated the task description. (Show Details)Oct 17 2024, 7:07 PM
gerritbot added a comment.Oct 17 2024, 8:37 PM

Change #1081261 had a related patch set uploaded (by Bking; author: Bking):

[labs/private@master] analytics_test_cluster: add secret

https://gerrit.wikimedia.org/r/1081261

gerritbot added a project: Patch-For-Review.Oct 17 2024, 8:37 PM
bking updated the task description. (Show Details)Oct 17 2024, 8:45 PM
gerritbot added a comment.Oct 17 2024, 9:55 PM

Change #1081268 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] airflow: make 'secret_key' configurable

https://gerrit.wikimedia.org/r/1081268

gerritbot added a comment.Oct 18 2024, 1:55 PM

Change #1081261 merged by Bking:

[labs/private@master] analytics_test_cluster: add secret

https://gerrit.wikimedia.org/r/1081261

bking mentioned this in rLPRIc5c1e80cf716: analytics_test_cluster: add secret.Oct 18 2024, 1:55 PM
BTullis edited projects, added Data-Platform-SRE (2024.10.19 - 2024.11.08); removed Data-Platform-SRE (2024.09.28 - 2024.10.18).Oct 18 2024, 3:08 PM
BTullis moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.10.19 - 2024.11.08) board.
gerritbot added a comment.Oct 18 2024, 3:15 PM

Change #1081268 merged by Brouberol:

[operations/puppet@production] airflow: make 'secret_key' configurable

https://gerrit.wikimedia.org/r/1081268

brouberol added a comment.Oct 18 2024, 3:26 PM

The airflow-analytics-test webserver migration is now complete! The UI is reachable (and OIDC-authenticated) at https://airflow-analytics-test.wikimedia.org.

We have also documented the migration process here to make it easier to migrate the other instances.

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2024, 3:30 PM
brouberol updated the task description. (Show Details)Oct 18 2024, 3:51 PM
brouberol updated the task description. (Show Details)
brouberol updated the task description. (Show Details)
bking updated the task description. (Show Details)Mon, Oct 28, 6:57 PM
bking closed this task as Resolved.Mon, Oct 28, 6:59 PM
bking moved this task from In Progress to Done on the Data-Platform-SRE (2024.10.19 - 2024.11.08) board.

I think migrating the test instance is a good AC for this task; we can create a new task or tasks for migrating the production instances. Closing...

bking updated Other Assignee, added: brouberol.Mon, Oct 28, 7:00 PM
brouberol added a subtask: T378377: Fix intermitent envoy errors when accessing the airflow UI.Tue, Oct 29, 8:28 AM
brouberol closed subtask T378377: Fix intermitent envoy errors when accessing the airflow UI as Resolved.Mon, Nov 4, 4:16 PM
brouberol renamed this task from Migrate airflow webservers to Kubernetes to Migrate airflow-analytics-test webserver to Kubernetes.Thu, Nov 7, 3:22 PM
brouberol edited parent tasks, added: T379267: Migrate the airflow webservers to Kubernetes; removed: T375729: Create LDAP groups to use for OIDC permission mapping with corresponding airflow DAG Authors groups .Thu, Nov 7, 3:25 PM
brouberol closed subtask T377601: Allow the selective enabling/disabling of each airflow component systemd service as Declined.Tue, Nov 12, 8:57 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits