Page MenuHomePhabricator
Create Task
Maniphest T378442

Migrate the airflow research instance webserver to dse-k8s
Closed, ResolvedPublic
Actions

Description

After the migration of the airflow-analytics-test instance webserver to kubernetes, we are ready to begin the migration of the remaining airflow instance webservers to kubernetes following the guide on https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#Migrating_an_existing_instance

This tracks the migration of the airflow research instance to dse-k8s.

  • Create the k8s namespaces
  • Create the k8s user kubeconfigs
  • Create the wikimedia.org public subdomains
  • Create the OIDC/CAS configuration
  • add the secret key already found in /etc/helmfile-defaults/private/dse-k8s_services/airflow-analytics-test/dse-k8s-eqiad.yaml on the deployment secret to /srv/git/private/hieradata
  • add keytab to the puppet private repo
  • create aiflow-research-ops ldap group
  • create admin_ng relevant config
  • deploy admin_ng
  • Defining the webserver configuration
  • Deploy the application
  • Enable ATS traffic redirection and caching

Details

SubjectRepoBranchLines +/-
Add airflow oidc clients for pcclabs/privatemaster+6 -0
airflow-research: change user identitiy files owner to analytics-deployoperations/puppetproduction+2 -2
airflow-research: define ATS redirectionoperations/puppetproduction+5 -0
airflow-research: provision kubernetes usersoperations/puppetproduction+8 -0
airflow-research: provision values and helmfilesoperations/deployment-chartsmaster+116 -0
add airflow-research namespace to the list of ceph/cloudnative tenant namespacesoperations/deployment-chartsmaster+3 -0
Provision airflow-research namespaceoperations/deployment-chartsmaster+4 -0
airflow-research: define OIDC configurationoperations/puppetproduction+8 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Stevemunene created this task.Oct 29 2024, 8:16 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2024, 8:16 AM
Stevemunene added a parent task: T362788: Migrate Airflow to the dse-k8s cluster.Oct 29 2024, 8:16 AM
brouberol edited parent tasks, added: T379267: Migrate the airflow webservers to Kubernetes; removed: T362788: Migrate Airflow to the dse-k8s cluster.Nov 7 2024, 3:26 PM
Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
BTullis renamed this task from Migrate the airflow research instance to dse-k8s to Migrate the airflow research instance webserver to dse-k8s.Nov 8 2024, 10:02 AM
BTullis updated the task description. (Show Details)
BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.Nov 8 2024, 10:13 AM
Gehel triaged this task as High priority.Nov 8 2024, 2:22 PM
brouberol claimed this task.Nov 13 2024, 8:28 AM
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
gerritbot added a comment.Nov 13 2024, 8:33 AM

Change #1090796 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-research: provision values and helmfiles

https://gerrit.wikimedia.org/r/1090796

gerritbot added a project: Patch-For-Review.Nov 13 2024, 8:33 AM
brouberol updated the task description. (Show Details)Nov 13 2024, 8:36 AM
brouberol updated the task description. (Show Details)Nov 13 2024, 8:47 AM
brouberol updated the task description. (Show Details)Nov 13 2024, 8:52 AM
brouberol updated the task description. (Show Details)
gerritbot added a comment.Nov 13 2024, 8:54 AM

Change #1090801 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Provision airflow-research namespace

https://gerrit.wikimedia.org/r/1090801

gerritbot added a comment.Nov 13 2024, 8:54 AM

Change #1090802 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] add airflow-research namespace to the list of ceph/cloudnative tenant namespaces

https://gerrit.wikimedia.org/r/1090802

gerritbot added a comment.Nov 13 2024, 8:58 AM

Change #1090803 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-research: define OIDC configuration

https://gerrit.wikimedia.org/r/1090803

gerritbot added a comment.Nov 13 2024, 8:58 AM

Change #1090804 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-research: define ATS redirection

https://gerrit.wikimedia.org/r/1090804

brouberol mentioned this in T375729: Create LDAP groups to use for OIDC permission mapping with corresponding airflow DAG Authors groups .Nov 13 2024, 9:22 AM
gerritbot added a comment.Nov 13 2024, 10:17 AM

Change #1090803 merged by Brouberol:

[operations/puppet@production] airflow-research: define OIDC configuration

https://gerrit.wikimedia.org/r/1090803

gerritbot added a comment.Nov 13 2024, 1:07 PM

Change #1090801 merged by Brouberol:

[operations/deployment-charts@master] Provision airflow-research namespace

https://gerrit.wikimedia.org/r/1090801

gerritbot added a comment.Nov 13 2024, 1:09 PM

Change #1090802 merged by Brouberol:

[operations/deployment-charts@master] add airflow-research namespace to the list of ceph/cloudnative tenant namespaces

https://gerrit.wikimedia.org/r/1090802

gerritbot added a comment.Nov 13 2024, 1:14 PM

Change #1090796 merged by Brouberol:

[operations/deployment-charts@master] airflow-research: provision values and helmfiles

https://gerrit.wikimedia.org/r/1090796

gerritbot added a comment.Nov 13 2024, 1:23 PM

Change #1090845 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-research: provision kubernetes users

https://gerrit.wikimedia.org/r/1090845

brouberol updated the task description. (Show Details)Nov 13 2024, 1:24 PM
brouberol updated the task description. (Show Details)
brouberol updated the task description. (Show Details)Nov 13 2024, 1:59 PM
gerritbot added a comment.Nov 13 2024, 2:00 PM

Change #1090845 merged by Brouberol:

[operations/puppet@production] airflow-research: provision kubernetes users

https://gerrit.wikimedia.org/r/1090845

brouberol updated the task description. (Show Details)Nov 13 2024, 2:27 PM
gerritbot added a comment.Nov 13 2024, 2:27 PM

Change #1090804 merged by Brouberol:

[operations/puppet@production] airflow-research: define ATS redirection

https://gerrit.wikimedia.org/r/1090804

Maintenance_bot removed a project: Patch-For-Review.Nov 13 2024, 2:31 PM
brouberol moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 13 2024, 2:32 PM
brouberol closed this task as Resolved.Nov 13 2024, 3:18 PM
brouberol moved this task from To Be Deployed to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

Screenshot 2024-11-13 at 16.18.38.png (2×2 px, 847 KB)

brouberol updated the task description. (Show Details)Nov 13 2024, 4:44 PM
gerritbot added a comment.Nov 14 2024, 8:28 AM

Change #1091178 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-research: change user identitiy files owner to analytics-deploy

https://gerrit.wikimedia.org/r/1091178

gerritbot added a project: Patch-For-Review.Nov 14 2024, 8:28 AM
gerritbot added a comment.Nov 14 2024, 9:25 AM

Change #1091178 merged by Brouberol:

[operations/puppet@production] airflow-research: change user identitiy files owner to analytics-deploy

https://gerrit.wikimedia.org/r/1091178

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2024, 9:31 AM
gerritbot added a comment.Nov 14 2024, 9:47 AM

Change #1091193 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[labs/private@master] Add airflow oidc clients for pcc

https://gerrit.wikimedia.org/r/1091193

gerritbot added a project: Patch-For-Review.Nov 14 2024, 9:47 AM
gerritbot added a comment.Nov 14 2024, 9:52 AM

Change #1091193 merged by Stevemunene:

[labs/private@master] Add airflow oidc clients for pcc

https://gerrit.wikimedia.org/r/1091193

Maintenance_bot removed a project: Patch-For-Review.Nov 14 2024, 10:31 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 15 2024, 2:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits