Page MenuHomePhabricator
Create Task
Maniphest T378443

Migrate the airflow platform-engineering instance webserver to dse-k8s
Closed, ResolvedPublic
Actions

Description

After the migration of the airflow-analytics-test instance webserver to kubernetes, we are ready to begin the migration of the remaining airflow instance webservers to kubernetes following the guide on https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#Migrating_an_existing_instance

This tracks the migration of the airflow platform-eng instance to dse-k8s

  • Create the k8s namespaces
  • Create the k8s user kubeconfigs
  • Create the wikimedia.org public subdomains
  • Create the OIDC/CAS configuration
  • add the secret key already found in /etc/helmfile-defaults/private/dse-k8s_services/airflow-analytics-test/dse-k8s-eqiad.yaml on the deployment secret to /srv/git/private/hieradata
  • add keytab to the puppet private repo
  • create aiflow-platform-engineering-ops ldap group
  • create admin_ng relevant config
  • deploy admin_ng
  • Defining the webserver configuration
  • Deploy the application
  • Enable ATS traffic redirection and caching

Details

SubjectRepoBranchLines +/-
airflow_platform_eng: Add missing stub secretlabs/privatemaster+2 -0
airflow-analytics-product: define namespaceoperations/deployment-chartsmaster+4 -0
airflow-platform-eng: create ATS mapping and caching configoperations/puppetproduction+5 -0
airflow-platform-eng: define helmfile and valuesoperations/deployment-chartsmaster+116 -0
airflow-platform-eng: register kube namespace in ceph csi / cloudnative pg tenantsoperations/deployment-chartsmaster+3 -0
airflow-platform-eng: define kube namespaceoperations/deployment-chartsmaster+4 -0
airflow-platform-eng: create OIDC configoperations/puppetproduction+8 -0
airflow-platform-eng: create user kubeconfigsoperations/puppetproduction+8 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Stevemunene created this task.Oct 29 2024, 8:17 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2024, 8:17 AM
brouberol added a parent task: T362788: Migrate Airflow to the dse-k8s cluster.Nov 5 2024, 1:46 PM
brouberol edited parent tasks, added: T379267: Migrate the airflow webservers to Kubernetes; removed: T362788: Migrate Airflow to the dse-k8s cluster.Nov 7 2024, 3:26 PM
Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
BTullis renamed this task from Migrate the airflow platform-engineering instance to dse-k8s to Migrate the airflow platform-engineering instance webserver to dse-k8s.Nov 8 2024, 10:03 AM
BTullis updated the task description. (Show Details)
BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.Nov 8 2024, 10:13 AM
Gehel triaged this task as High priority.Nov 8 2024, 2:23 PM
brouberol claimed this task.Nov 15 2024, 1:17 PM
brouberol updated the task description. (Show Details)
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
gerritbot added a comment.Nov 15 2024, 1:26 PM

Change #1091697 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-platform-eng: define kube namespace

https://gerrit.wikimedia.org/r/1091697

gerritbot added a comment.Nov 15 2024, 1:26 PM

Change #1091694 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-platform-eng: create user kubeconfigs

https://gerrit.wikimedia.org/r/1091694

gerritbot added a project: Patch-For-Review.Nov 15 2024, 1:26 PM

Change #1091698 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-platform-eng: register kube namespace in ceph csi / cloudnative pg tenants

https://gerrit.wikimedia.org/r/1091698

gerritbot added a comment.Nov 15 2024, 1:26 PM

Change #1091695 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-platform-eng: create OIDC config

https://gerrit.wikimedia.org/r/1091695

gerritbot added a comment.Nov 15 2024, 1:26 PM

Change #1091699 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-platform-eng: define helmfile and values

https://gerrit.wikimedia.org/r/1091699

gerritbot added a comment.Nov 15 2024, 1:26 PM

Change #1091696 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-platform-eng: create ATS mapping and caching config

https://gerrit.wikimedia.org/r/1091696

brouberol added a comment.Nov 15 2024, 1:30 PM
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey analytics-platform-eng/airflow-platform-eng.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey airflow/airflow-platform-eng.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey HTTP/airflow-platform-eng.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local ktadd -norandkey -k analytics-platform-eng.keytab analytics-platform-eng/airflow-platform-eng.discovery.wmnet airflow/airflow-platform-eng.discovery.wmnet@WIKIMEDIA HTTP/airflow-platform-eng.discovery.wmnet@WIKIMEDIA
Entry for principal analytics-platform-eng/airflow-platform-eng.discovery.wmnet with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:analytics-platform-eng.keytab.
Entry for principal airflow/airflow-platform-eng.discovery.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:analytics-platform-eng.keytab.
Entry for principal HTTP/airflow-platform-eng.discovery.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:analytics-platform-eng.keytab.
brouberol updated the task description. (Show Details)Nov 15 2024, 1:32 PM
brouberol updated the task description. (Show Details)Nov 15 2024, 2:26 PM
gerritbot added a comment.Nov 15 2024, 3:26 PM

Change #1091694 merged by Brouberol:

[operations/puppet@production] airflow-platform-eng: create user kubeconfigs

https://gerrit.wikimedia.org/r/1091694

gerritbot added a comment.Nov 15 2024, 3:27 PM

Change #1091695 merged by Brouberol:

[operations/puppet@production] airflow-platform-eng: create OIDC config

https://gerrit.wikimedia.org/r/1091695

gerritbot added a comment.Nov 15 2024, 3:33 PM

Change #1091697 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-platform-eng: define kube namespace

https://gerrit.wikimedia.org/r/1091697

gerritbot added a comment.Nov 15 2024, 3:33 PM

Change #1091698 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-platform-eng: register kube namespace in ceph csi / cloudnative pg tenants

https://gerrit.wikimedia.org/r/1091698

gerritbot added a comment.Nov 15 2024, 3:33 PM

Change #1091699 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-platform-eng: define helmfile and values

https://gerrit.wikimedia.org/r/1091699

gerritbot added a comment.Nov 15 2024, 3:39 PM

Change #1091696 merged by Brouberol:

[operations/puppet@production] airflow-platform-eng: create ATS mapping and caching config

https://gerrit.wikimedia.org/r/1091696

brouberol updated the task description. (Show Details)Nov 15 2024, 3:39 PM
brouberol moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 15 2024, 3:45 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 15 2024, 4:30 PM
brouberol closed this task as Resolved.Nov 15 2024, 5:47 PM
brouberol moved this task from To Be Deployed to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

Screenshot 2024-11-15 at 18.47.20.png (1×2 px, 488 KB)

gerritbot added a comment.Nov 18 2024, 10:10 AM

Change #1092197 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/deployment-charts@master] airflow-analytics-product: define namespace

https://gerrit.wikimedia.org/r/1092197

gerritbot added a project: Patch-For-Review.Nov 18 2024, 10:10 AM
gerritbot added a comment.Nov 18 2024, 11:23 AM

Change #1092197 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytics-product: define namespace

https://gerrit.wikimedia.org/r/1092197

Maintenance_bot removed a project: Patch-For-Review.Nov 18 2024, 11:30 AM
gerritbot added a comment.Nov 19 2024, 10:32 AM

Change #1092785 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[labs/private@master] airflow_platform_eng: Add missing stub secret

https://gerrit.wikimedia.org/r/1092785

gerritbot added a project: Patch-For-Review.Nov 19 2024, 10:32 AM
gerritbot added a comment.Nov 19 2024, 11:20 AM

Change #1092785 merged by Muehlenhoff:

[labs/private@master] airflow_platform_eng: Add missing stub secret

https://gerrit.wikimedia.org/r/1092785

Maintenance_bot removed a project: Patch-For-Review.Nov 19 2024, 11:31 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 29 2024, 9:26 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits