Page MenuHomePhabricator
Create Task
Maniphest T378438

Migrate airflow-wmde instance webserver to kubernetes
Closed, ResolvedPublic
Actions

Description

After the migration of the airflow-analytics-test instance webserver to kubernetes, we are ready to begin the migration of the remaining airflow instance webservers to kubernetes following the guide on https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#Migrating_an_existing_instance
allow ingress traffic to the airflow instances coming from the DSE_KUBEPODS subnet

  • Create the k8s namespaces
  • Create the k8s user kubeconfigs
  • Create the wikimedia.org public subdomains
  • Create the OIDC/CAS configuration
  • create aiflow-wmde-ops ldap group
  • create admin_ng relevant config (in progress)
  • deploy admin_ng
  • Defining the webserver configuration (in progress)
  • Deploy the application
    • modify modules/profile/manifests/airflow.pp to support an optional secret secret_key and populate the webserver.secret_key config with it if found
    • add the secret key already found in /etc/helmfile-defaults/private/dse-k8s_services/airflow-analytics-test/dse-k8s-eqiad.yaml on the deployment secret to /srv/git/private/hieradata
  • Enable ATS traffic redirection and caching

Details

SubjectRepoBranchLines +/-
airflow-wmde: fix network policyoperations/deployment-chartsmaster+1 -1
fix values symlink and ops groupoperations/deployment-chartsmaster+2 -1
ATS: add mapping for airflow-wmdeoperations/puppetproduction+5 -0
airflow: add airflow-wmde filesoperations/deployment-chartsmaster+121 -0
airflow: add wmde namespaceoperations/puppetproduction+8 -0
idp:Add airflow-wmde to idpoperations/puppetproduction+8 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Stevemunene created this task.Oct 29 2024, 8:07 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2024, 8:07 AM
Stevemunene added a parent task: T362788: Migrate Airflow to the dse-k8s cluster.Oct 29 2024, 8:08 AM
bking subscribed.Oct 29 2024, 4:05 PM
Gehel triaged this task as High priority.Oct 29 2024, 4:33 PM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.10.19 - 2024.11.08) board.
brouberol edited parent tasks, added: T379267: Migrate the airflow webservers to Kubernetes; removed: T362788: Migrate Airflow to the dse-k8s cluster.Nov 7 2024, 3:25 PM
gerritbot added a comment.Nov 8 2024, 6:26 AM

Change #1088404 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] airflow: add wmde namespace

https://gerrit.wikimedia.org/r/1088404

gerritbot added a project: Patch-For-Review.Nov 8 2024, 6:26 AM
gerritbot added a comment.Nov 8 2024, 6:46 AM

Change #1088405 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] airflow: add airflow-wmde files

https://gerrit.wikimedia.org/r/1088405

gerritbot added a comment.Nov 8 2024, 6:57 AM

Change #1088407 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] idp:Add aiflow-wmde to idp

https://gerrit.wikimedia.org/r/1088407

gerritbot added a comment.Nov 8 2024, 8:35 AM

Change #1088407 merged by Stevemunene:

[operations/puppet@production] idp:Add airflow-wmde to idp

https://gerrit.wikimedia.org/r/1088407

Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
Gehel moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
BTullis renamed this task from Migrate airflow-wmde instance to kubernetes to Migrate airflow-wmde instance webserver to kubernetes.Nov 8 2024, 10:01 AM
BTullis updated the task description. (Show Details)
BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.Nov 8 2024, 10:13 AM
gerritbot added a comment.Nov 8 2024, 1:21 PM

Change #1088404 merged by Stevemunene:

[operations/puppet@production] airflow: add wmde namespace

https://gerrit.wikimedia.org/r/1088404

Stevemunene updated the task description. (Show Details)Nov 12 2024, 7:56 AM
Stevemunene mentioned this in T375729: Create LDAP groups to use for OIDC permission mapping with corresponding airflow DAG Authors groups .Nov 12 2024, 12:01 PM
Stevemunene added a comment.Nov 12 2024, 12:33 PM

Adding the airflow-wmde-ops group as per https://phabricator.wikimedia.org/T375729#10228506

stevemunene@seaborgium:~$ ldapadd -f airflow-wmde-ops.ldif -D "cn=admin,dc=wikimedia,dc=org" -x -W -H "ldap://ldap-rw.eqiad.wikimedia.org:389"
Enter LDAP Password: 
adding new entry "cn=airflow-wmde-ops,ou=groups,dc=wikimedia,dc=org"

stevemunene@seaborgium:~$ ldapsearch -x cn=airflow-wmde-ops
# extended LDIF
#
# LDAPv3
# base <dc=wikimedia,dc=org> (default) with scope subtree
# filter: cn=airflow-wmde-ops
# requesting: ALL
#

# airflow-wmde-ops, groups, wikimedia.org
dn: cn=airflow-wmde-ops,ou=groups,dc=wikimedia,dc=org
cn: airflow-wmde-ops
objectClass: groupOfNames
member: uid=lucaswerkmeister-wmde,ou=people,dc=wikimedia,dc=org
member: uid=conniecc1,ou=people,dc=wikimedia,dc=org
member: uid=tarrow,ou=people,dc=wikimedia,dc=org
member: uid=itamar,ou=people,dc=wikimedia,dc=org
member: uid=awight,ou=people,dc=wikimedia,dc=org
member: uid=wmde-fisch,ou=people,dc=wikimedia,dc=org
member: uid=wmde-leszek,ou=people,dc=wikimedia,dc=org
member: uid=jakob,ou=people,dc=wikimedia,dc=org
member: uid=sihe,ou=people,dc=wikimedia,dc=org
member: uid=ollieshotton,ou=people,dc=wikimedia,dc=org
member: uid=muja,ou=people,dc=wikimedia,dc=org
member: uid=andrewtavis-wmde,ou=people,dc=wikimedia,dc=org
member: uid=kpayne,ou=people,dc=wikimedia,dc=org

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
gerritbot added a comment.Nov 12 2024, 3:46 PM

Change #1088405 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: add airflow-wmde files

https://gerrit.wikimedia.org/r/1088405

Stevemunene moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 12 2024, 4:05 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 12 2024, 4:31 PM
brouberol updated the task description. (Show Details)Nov 12 2024, 4:42 PM
gerritbot added a comment.Nov 13 2024, 8:24 AM

Change #1090794 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] ATS: add mapping for airflow-wmde

https://gerrit.wikimedia.org/r/1090794

gerritbot added a project: Patch-For-Review.Nov 13 2024, 8:24 AM
gerritbot added a comment.Nov 13 2024, 12:04 PM

Change #1090794 merged by Stevemunene:

[operations/puppet@production] ATS: add mapping for airflow-wmde

https://gerrit.wikimedia.org/r/1090794

gerritbot added a comment.Nov 13 2024, 12:18 PM

Change #1090838 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] fix values symlink and ops group

https://gerrit.wikimedia.org/r/1090838

gerritbot added a comment.Nov 13 2024, 12:25 PM

Change #1090838 merged by jenkins-bot:

[operations/deployment-charts@master] fix values symlink and ops group

https://gerrit.wikimedia.org/r/1090838

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2024, 12:31 PM
gerritbot added a comment.Nov 13 2024, 12:43 PM

Change #1090840 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] airflow-wmde: fix network policy

https://gerrit.wikimedia.org/r/1090840

gerritbot added a project: Patch-For-Review.Nov 13 2024, 12:43 PM
gerritbot added a comment.Nov 13 2024, 1:05 PM

Change #1090840 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-wmde: fix network policy

https://gerrit.wikimedia.org/r/1090840

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2024, 1:31 PM
Stevemunene updated the task description. (Show Details)Nov 13 2024, 2:54 PM
Stevemunene closed this task as Resolved.Nov 13 2024, 3:44 PM
Stevemunene moved this task from To Be Deployed to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

now available via https://airflow-wmde.wikimedia.org/ on dse-k8s cluster

image.png (516×1 px, 70 KB)

AndrewTavis_WMDE subscribed.Nov 13 2024, 5:00 PM

Thanks so much for the support here, @Stevemunene!

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 15 2024, 2:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits