Page MenuHomePhabricator
Create Task
Maniphest T378441

Migrate airflow-search instance webserver to kubernetes
Closed, ResolvedPublic
Actions

Description

After the migration of the airflow-analytics-test instance webserver to kubernetes, we are ready to begin the migration of the remaining airflow instance webservers to kubernetes following the guide on https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#Migrating_an_existing_instance

This tracks the migration of the search instance to dse-k8s

  • Create the k8s namespaces
  • Create the k8s user kubeconfigs
  • Create the wikimedia.org public subdomains
  • Create the OIDC/CAS configuration
  • add the secret key already found in /etc/helmfile-defaults/private/dse-k8s_services/airflow-analytics-test/dse-k8s-eqiad.yaml on the deployment secret to /srv/git/private/hieradata
  • add keytab to the puppet private repo
  • create aiflow-search-ops ldap group
  • create admin_ng relevant config
  • deploy admin_ng
  • Defining the webserver configuration
  • Deploy the application
  • Enable ATS traffic redirection and caching

Details

SubjectRepoBranchLines +/-
airflow-search: set kubeconfig owner group to analytics-delployersoperations/puppetproduction+2 -2
airflow-search: define ATS mapping and cache configoperations/puppetproduction+5 -0
airflow-search: define helmfile and valuesoperations/deployment-chartsmaster+116 -0
airflow-search: define OIDC configoperations/puppetproduction+8 -0
Add airflow oidc clients for pcclabs/privatemaster+6 -0
airflow-search: register tenant namespace in ceph-csi and cloudnative-pg operator configsoperations/deployment-chartsmaster+3 -0
airflow-search: define k8s namespaceoperations/deployment-chartsmaster+4 -0
airflow-search: define user kubeconfigsoperations/puppetproduction+8 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Stevemunene created this task.Oct 29 2024, 8:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2024, 8:14 AM
Stevemunene added a parent task: T362788: Migrate Airflow to the dse-k8s cluster.Oct 29 2024, 8:15 AM
Stevemunene updated the task description. (Show Details)
brouberol edited parent tasks, added: T379267: Migrate the airflow webservers to Kubernetes; removed: T362788: Migrate Airflow to the dse-k8s cluster.Nov 7 2024, 3:26 PM
Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
BTullis renamed this task from Migrate airflow-search instance to kubernetes to Migrate airflow-search instance webserver to kubernetes.Nov 8 2024, 10:02 AM
BTullis updated the task description. (Show Details)
BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.Nov 8 2024, 10:13 AM
Gehel triaged this task as High priority.Nov 8 2024, 2:22 PM
brouberol changed the task status from Open to In Progress.Nov 14 2024, 8:17 AM
brouberol claimed this task.
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
gerritbot added a comment.Nov 14 2024, 8:27 AM

Change #1091175 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-search: define k8s namespace

https://gerrit.wikimedia.org/r/1091175

gerritbot added a project: Patch-For-Review.Nov 14 2024, 8:27 AM

Change #1091176 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-search: register tenant namespace in ceph-csi and cloudnative-pg operator configs

https://gerrit.wikimedia.org/r/1091176

gerritbot added a comment.Nov 14 2024, 8:27 AM

Change #1091177 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-search: define helmfile and values

https://gerrit.wikimedia.org/r/1091177

gerritbot added a comment.Nov 14 2024, 8:28 AM

Change #1091179 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-search: define user kubeconfigs

https://gerrit.wikimedia.org/r/1091179

gerritbot added a comment.Nov 14 2024, 8:28 AM

Change #1091180 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-search: define OIDC config

https://gerrit.wikimedia.org/r/1091180

gerritbot added a comment.Nov 14 2024, 8:28 AM

Change #1091181 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-search: define ATS mapping and cache config

https://gerrit.wikimedia.org/r/1091181

brouberol added a comment.EditedNov 14 2024, 8:31 AM
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey analytics-search/airflow-search.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey airflow/airflow-search.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey HTTP/airflow-search.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local ktadd -norandkey -k search.keytab analytics-search/airflow-search.discovery.wmnet airflow/airflow-search.discovery.wmnet@WIKIMEDIA HTTP/airflow-search.discovery.wmnet@WIKIMEDIA
Entry for principal analytics-search/airflow-search.discovery.wmnet with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:search.keytab.
Entry for principal airflow/airflow-search.discovery.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:search.keytab.
Entry for principal HTTP/airflow-search.discovery.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:search.keytab.
brouberol added a subscriber: BTullis.Nov 14 2024, 8:41 AM

@BTullis who should we add to the analytics-search-ops LDAP group? These users?

# puppet/modules/admin/data/data.yaml
...
  analytics-search-users:
    gid: 771
    description: Group of users for managing search related analytics jobs
    members: [ebernhardson, dcausse, gehel, bearloga, tjones, pfischer, dr0ptp4kt] # <--
    privileges: ['ALL = (analytics-search) NOPASSWD: ALL']
    system_members: [analytics-search]
brouberol moved this task from In Progress to To Be Deployed on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 14 2024, 8:51 AM
brouberol moved this task from To Be Deployed to Needs Review on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
brouberol updated the task description. (Show Details)Nov 14 2024, 9:03 AM
gerritbot added a comment.Nov 14 2024, 9:25 AM

Change #1091179 merged by Brouberol:

[operations/puppet@production] airflow-search: define user kubeconfigs

https://gerrit.wikimedia.org/r/1091179

gerritbot added a comment.Nov 14 2024, 9:33 AM

Change #1091175 merged by Brouberol:

[operations/deployment-charts@master] airflow-search: define k8s namespace

https://gerrit.wikimedia.org/r/1091175

gerritbot added a comment.Nov 14 2024, 9:35 AM

Change #1091176 merged by Brouberol:

[operations/deployment-charts@master] airflow-search: register tenant namespace in ceph-csi and cloudnative-pg operator configs

https://gerrit.wikimedia.org/r/1091176

brouberol updated the task description. (Show Details)Nov 14 2024, 9:37 AM
gerritbot added a comment.Nov 14 2024, 9:47 AM

Change #1091193 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[labs/private@master] Add airflow oidc clients for pcc

https://gerrit.wikimedia.org/r/1091193

gerritbot added a comment.Nov 14 2024, 9:52 AM

Change #1091193 merged by Stevemunene:

[labs/private@master] Add airflow oidc clients for pcc

https://gerrit.wikimedia.org/r/1091193

gerritbot added a comment.Nov 14 2024, 3:08 PM

Change #1091180 merged by Brouberol:

[operations/puppet@production] airflow-search: define OIDC config

https://gerrit.wikimedia.org/r/1091180

gerritbot added a comment.Nov 14 2024, 3:13 PM

Change #1091177 merged by Brouberol:

[operations/deployment-charts@master] airflow-search: define helmfile and values

https://gerrit.wikimedia.org/r/1091177

gerritbot added a comment.Nov 14 2024, 3:16 PM

Change #1091181 merged by Brouberol:

[operations/puppet@production] airflow-search: define ATS mapping and cache config

https://gerrit.wikimedia.org/r/1091181

brouberol updated the task description. (Show Details)Nov 14 2024, 3:17 PM
brouberol updated the task description. (Show Details)
Maintenance_bot removed a project: Patch-For-Review.Nov 14 2024, 3:31 PM
brouberol updated the task description. (Show Details)Nov 14 2024, 4:09 PM
brouberol closed this task as Resolved.Nov 14 2024, 4:36 PM
brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
gerritbot added a comment.Nov 15 2024, 7:22 AM

Change #1091523 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-search: set kubeconfig owner group to analytics-delployers

https://gerrit.wikimedia.org/r/1091523

gerritbot added a project: Patch-For-Review.Nov 15 2024, 7:22 AM
brouberol updated the task description. (Show Details)Nov 15 2024, 7:35 AM
gerritbot added a comment.Nov 15 2024, 12:07 PM

Change #1091523 merged by Brouberol:

[operations/puppet@production] airflow-search: set kubeconfig owner group to analytics-delployers

https://gerrit.wikimedia.org/r/1091523

Maintenance_bot removed a project: Patch-For-Review.Nov 15 2024, 12:31 PM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 15 2024, 2:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits