Page MenuHomePhabricator
Create Task
Maniphest T378439

Migrate the analytics airflow instance webserver to dse-k8s
Closed, ResolvedPublic
Actions

Description

After the migration of the airflow-analytics-test instance webserver to kubernetes, we are ready to begin the migration of the remaining airflow instance webservers to kubernetes following the guide on https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes#Migrating_an_existing_instance

This tracks the migration of the analytics airflow instance to dse-k8s

  • Create the k8s namespaces
  • Create the k8s user kubeconfigs
  • Create the wikimedia.org public subdomains
  • Create the OIDC/CAS configuration
  • add the secret key already found in /etc/helmfile-defaults/private/dse-k8s_services/airflow-analytics-test/dse-k8s-eqiad.yaml on the deployment secret to /srv/git/private/hieradata
  • add keytab to the puppet private repo
  • create aiflow-analytics-ops ldap group
  • create admin_ng relevant config
  • deploy admin_ng
  • Defining the webserver configuration
  • Deploy the application
  • Enable ATS traffic redirection and caching

Details

SubjectRepoBranchLines +/-
airflow-analytics: define ATS mapping and cache configoperations/puppetproduction+5 -0
airflow-analytics: fix typo in db usernameoperations/deployment-chartsmaster+1 -1
airflow-analytics: define helmfile and valuesoperations/deployment-chartsmaster+116 -0
airflow-analytics: add namespace to tenant list of ceph csi and cloudnative-pgoperations/deployment-chartsmaster+3 -0
airflow-analytics: define namespaceoperations/deployment-chartsmaster+4 -0
airflow-analytics: define OIDC configoperations/puppetproduction+8 -0
airflow-analytics: define user kubeconfigoperations/puppetproduction+8 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Stevemunene created this task.Oct 29 2024, 8:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 29 2024, 8:10 AM
Stevemunene renamed this task from Migrate the analytics airflow webserver to dse-k8s to Migrate the analytics airflow instance to dse-k8s.Oct 29 2024, 8:14 AM
Stevemunene updated the task description. (Show Details)
brouberol added a parent task: T362788: Migrate Airflow to the dse-k8s cluster.Nov 5 2024, 1:47 PM
brouberol edited parent tasks, added: T379267: Migrate the airflow webservers to Kubernetes; removed: T362788: Migrate Airflow to the dse-k8s cluster.Nov 7 2024, 3:26 PM
Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
BTullis renamed this task from Migrate the analytics airflow instance to dse-k8s to Migrate the analytics airflow instance webserver to dse-k8s.Nov 8 2024, 10:03 AM
BTullis updated the task description. (Show Details)
BTullis mentioned this in T362788: Migrate Airflow to the dse-k8s cluster.Nov 8 2024, 10:13 AM
Gehel triaged this task as High priority.Nov 8 2024, 2:22 PM
brouberol changed the task status from Open to In Progress.Nov 14 2024, 4:39 PM
brouberol claimed this task.
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
brouberol updated the task description. (Show Details)Nov 14 2024, 4:42 PM
gerritbot added a comment.Nov 15 2024, 7:22 AM

Change #1091527 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics: define namespace

https://gerrit.wikimedia.org/r/1091527

gerritbot added a project: Patch-For-Review.Nov 15 2024, 7:22 AM

Change #1091528 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics: add namespace to tenant list of ceph csi and cloudnative-pg

https://gerrit.wikimedia.org/r/1091528

gerritbot added a comment.Nov 15 2024, 7:22 AM

Change #1091524 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-analytics: define user kubeconfig

https://gerrit.wikimedia.org/r/1091524

gerritbot added a comment.Nov 15 2024, 7:22 AM

Change #1091529 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics: define helmfile and values

https://gerrit.wikimedia.org/r/1091529

gerritbot added a comment.Nov 15 2024, 7:22 AM

Change #1091525 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-analytics: define OIDC config

https://gerrit.wikimedia.org/r/1091525

gerritbot added a comment.Nov 15 2024, 7:22 AM

Change #1091526 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] airflow-analytics: define ATS mapping and cache config

https://gerrit.wikimedia.org/r/1091526

brouberol added a comment.EditedNov 15 2024, 7:24 AM
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey analytics/airflow-analytics.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey airflow/airflow-analytics.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local addprinc -randkey HTTP/airflow-analytics.discovery.wmnet@WIKIMEDIA
brouberol@krb1001:~$ sudo kadmin.local ktadd -norandkey -k analytics.keytab analytics/airflow-analytics.discovery.wmnet airflow/airflow-analytics.discovery.wmnet@WIKIMEDIA HTTP/airflow-analytics.discovery.wmnet@WIKIMEDIA
Entry for principal analytics/airflow-analytics.discovery.wmnet with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:analytics.keytab.
Entry for principal airflow/airflow-analytics.discovery.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:analytics.keytab.
Entry for principal HTTP/airflow-analytics.discovery.wmnet@WIKIMEDIA with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:analytics.keytab.
brouberol updated the task description. (Show Details)Nov 15 2024, 7:35 AM
gerritbot added a comment.Nov 15 2024, 12:07 PM

Change #1091524 merged by Brouberol:

[operations/puppet@production] airflow-analytics: define user kubeconfig

https://gerrit.wikimedia.org/r/1091524

gerritbot added a comment.Nov 15 2024, 12:10 PM

Change #1091525 merged by Brouberol:

[operations/puppet@production] airflow-analytics: define OIDC config

https://gerrit.wikimedia.org/r/1091525

gerritbot added a comment.Nov 15 2024, 12:16 PM

Change #1091527 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytics: define namespace

https://gerrit.wikimedia.org/r/1091527

gerritbot added a comment.Nov 15 2024, 12:16 PM

Change #1091528 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytics: add namespace to tenant list of ceph csi and cloudnative-pg

https://gerrit.wikimedia.org/r/1091528

gerritbot added a comment.Nov 15 2024, 12:16 PM

Change #1091529 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytics: define helmfile and values

https://gerrit.wikimedia.org/r/1091529

gerritbot added a comment.Nov 15 2024, 12:24 PM

Change #1091642 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics: fix typo in db username

https://gerrit.wikimedia.org/r/1091642

gerritbot added a comment.Nov 15 2024, 12:25 PM

Change #1091642 merged by Brouberol:

[operations/deployment-charts@master] airflow-analytics: fix typo in db username

https://gerrit.wikimedia.org/r/1091642

gerritbot added a comment.Nov 15 2024, 12:27 PM

Change #1091526 merged by Brouberol:

[operations/puppet@production] airflow-analytics: define ATS mapping and cache config

https://gerrit.wikimedia.org/r/1091526

brouberol updated the task description. (Show Details)Nov 15 2024, 12:27 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 15 2024, 12:31 PM
brouberol updated the task description. (Show Details)Nov 15 2024, 12:32 PM
brouberol closed this task as Resolved.Nov 15 2024, 1:10 PM
brouberol moved this task from In Progress to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.

Screenshot 2024-11-15 at 14.10.00.png (2×2 px, 912 KB)

Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 15 2024, 2:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits