Page MenuHomePhabricator
Create Task
Maniphest T380284

Migrate the airflow-analytics-test scheduler to Kubernetes
Closed, ResolvedPublic
Actions

Description

Now that the webserver is running in Kubernetes, we'll migrate the kerberos and scheduler components to Kubernetes, to see whether everything still works as expected.

  • Create the s3 user for the airflow logs
  • Create the s3 bucket for the airflow logs
  • Store the S3 keys in the private repo
  • Stop the kerberos and scheduler systemd services on an-test-client1002
  • Copy all airflow logs to the s3 log bucket
  • Deploy airflow-analytics-test with kerberos, scheduler and remote logging enabled
  • Disable the airflow systemd services on the airflow host

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
airflow-analytics-test: use the cloudnative PG clusteroperations/deployment-chartsmaster+0 -18
airflow-analytics-test: add namespace to the cloudnativePG tenant namespacesoperations/deployment-chartsmaster+1 -0
airflow-analytics-test: deploy the scheduler and kerberos componentsoperations/deployment-chartsmaster+4 -10
airflow: allow multiple DAG folders to be pulled inoperations/deployment-chartsmaster+8 -1
hotfix: prevent puppet resource creation when no airflow instances are specifiedoperations/puppetproduction+1 -1
an-test-client1002: disable puppet management of airflow servicesoperations/puppetproduction+168 -165
an-test-client1002: ensure that airflow services are absentoperations/puppetproduction+26 -26
airflow: upgrade base imageoperations/deployment-chartsmaster+1 -1
Airflow: add missing hive connectionsoperations/deployment-chartsmaster+18 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

brouberol created this task.Nov 19 2024, 3:32 PM
brouberol updated the task description. (Show Details)Nov 19 2024, 3:37 PM
brouberol added a comment.Nov 19 2024, 3:47 PM
brouberol@cephosd1001:~$ sudo radosgw-admin user create --uid=airflow-analytics-test --display-name="airflow-analytics-test"
{
    "user_id": "airflow-analytics-test",
    "display_name": "airflow-analytics-test",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "airflow-analytics-test",
            "access_key": "REDACTED",
            "secret_key": "REDACTED"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
brouberol updated the task description. (Show Details)Nov 19 2024, 3:47 PM
brouberol added a comment.Nov 19 2024, 3:50 PM
brouberol@stat1008:~$ s3cmd --access_key=$access_key --secret_key=$secret_key --host=rgw.eqiad.dpe.anycast.wmnet --region=dpe --host-bucket=no mb s3://logs.airflow-analytics-test.dse-k8s-eqiad
Bucket 's3://logs.airflow-analytics-test.dse-k8s-eqiad/' created
brouberol updated the task description. (Show Details)Nov 19 2024, 3:50 PM
brouberol changed the task status from Open to In Progress.Nov 19 2024, 4:01 PM
brouberol moved this task from Backlog - project to In Progress on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
gerritbot added a comment.Nov 20 2024, 7:00 AM

Change #1093177 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics-test: deploy the scheduler and kerberos components

https://gerrit.wikimedia.org/r/1093177

gerritbot added a project: Patch-For-Review.Nov 20 2024, 7:00 AM
brouberol added a comment.Nov 20 2024, 8:17 AM
brouberol@an-test-client1002:/srv/airflow-analytics_test/logs$ s3cmd \
    --access_key=$access_key \
    --secret_key=$secret_key \
    --host=rgw.eqiad.dpe.anycast.wmnet \
    --region=dpe \
    --host-bucket=no \
    sync -r ./* s3://logs.airflow-analytics-test.dse-k8s-eqiad/
...
Done. Uploaded 3914855088 bytes in 886.2 seconds, 4.21 MB/s.
brouberol updated the task description. (Show Details)Nov 20 2024, 8:18 AM
brouberol updated the task description. (Show Details)Nov 20 2024, 9:16 AM
brouberol@an-test-client1002 $ sudo puppet agent --disable "brouberol: WIP migrating the airflow scheduler to Kubernetes"
brouberol@an-test-client1002 $ sudo systemctl stop airflow-{webserver,kerberos,scheduler}@analytics_test.service
brouberol updated the task description. (Show Details)Nov 20 2024, 3:18 PM
gerritbot added a comment.Nov 20 2024, 3:24 PM

Change #1093364 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] Airflow: add missing hive connections

https://gerrit.wikimedia.org/r/1093364

gerritbot added a comment.Nov 20 2024, 3:24 PM

Change #1093365 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: upgrade base image

https://gerrit.wikimedia.org/r/1093365

gerritbot added a comment.Nov 20 2024, 3:24 PM

Change #1093366 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow: allow multiple DAG folders to be pulled in

https://gerrit.wikimedia.org/r/1093366

gerritbot added a comment.Nov 20 2024, 3:40 PM

Change #1093368 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] an-test-client1002: ensure that airflow services are absent

https://gerrit.wikimedia.org/r/1093368

brouberol updated the task description. (Show Details)Nov 20 2024, 3:45 PM
brouberol removed a subscriber: JMeybohm.
gerritbot added a comment.Nov 20 2024, 3:51 PM

Change #1093364 merged by jenkins-bot:

[operations/deployment-charts@master] Airflow: add missing hive connections

https://gerrit.wikimedia.org/r/1093364

gerritbot added a comment.Nov 20 2024, 3:51 PM

Change #1093365 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: upgrade base image

https://gerrit.wikimedia.org/r/1093365

gerritbot added a comment.Nov 20 2024, 3:57 PM

Change #1093368 merged by Brouberol:

[operations/puppet@production] an-test-client1002: ensure that airflow services are absent

https://gerrit.wikimedia.org/r/1093368

gerritbot added a comment.Nov 20 2024, 4:01 PM

Change #1093373 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] an-test-client1002: disable puppet management of airflow services

https://gerrit.wikimedia.org/r/1093373

gerritbot added a comment.Nov 20 2024, 4:14 PM

Change #1093373 merged by Brouberol:

[operations/puppet@production] an-test-client1002: disable puppet management of airflow services

https://gerrit.wikimedia.org/r/1093373

gerritbot added a comment.Nov 20 2024, 4:21 PM

Change #1093378 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/puppet@production] hotfix: prevent puppet resource creation when no airflow instances are specified

https://gerrit.wikimedia.org/r/1093378

gerritbot added a comment.Nov 20 2024, 4:23 PM

Change #1093378 merged by Brouberol:

[operations/puppet@production] hotfix: prevent puppet resource creation when no airflow instances are specified

https://gerrit.wikimedia.org/r/1093378

brouberol closed this task as Resolved.Nov 20 2024, 4:26 PM
brouberol updated the task description. (Show Details)
brouberol moved this task from In Progress to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
brouberol reopened this task as In Progress.Nov 20 2024, 4:59 PM

We still have some unmerged patches

brouberol moved this task from Done to Needs Review on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 20 2024, 4:59 PM
gerritbot added a comment.Nov 22 2024, 2:05 PM

Change #1094436 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics-test: use the cloudnative PG cluster

https://gerrit.wikimedia.org/r/1094436

gerritbot added a comment.Nov 22 2024, 2:46 PM

Change #1093366 merged by jenkins-bot:

[operations/deployment-charts@master] airflow: allow multiple DAG folders to be pulled in

https://gerrit.wikimedia.org/r/1093366

gerritbot added a comment.Nov 22 2024, 2:46 PM

Change #1093177 merged by jenkins-bot:

[operations/deployment-charts@master] airflow-analytics-test: deploy the scheduler and kerberos components

https://gerrit.wikimedia.org/r/1093177

brouberol closed this task as Resolved.Nov 22 2024, 3:08 PM
brouberol moved this task from Needs Review to Done on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
gerritbot added a comment.Nov 25 2024, 9:37 AM

Change #1097323 had a related patch set uploaded (by Brouberol; author: Brouberol):

[operations/deployment-charts@master] airflow-analytics-test: add namespace to the cloudnativePG tenant namespaces

https://gerrit.wikimedia.org/r/1097323

gerritbot added a comment.Nov 25 2024, 9:42 AM

Change #1097323 merged by Brouberol:

[operations/deployment-charts@master] airflow-analytics-test: add namespace to the cloudnativePG tenant namespaces

https://gerrit.wikimedia.org/r/1097323

gerritbot added a comment.Nov 25 2024, 10:02 AM

Change #1094436 merged by Brouberol:

[operations/deployment-charts@master] airflow-analytics-test: use the cloudnative PG cluster

https://gerrit.wikimedia.org/r/1094436

Maintenance_bot removed a project: Patch-For-Review.Nov 25 2024, 10:30 AM
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.Nov 29 2024, 9:33 AM
Gehel edited projects, added Data-Platform-SRE (2023.12.01 - 2023.12.31); removed Data-Platform-SRE (2024.11.09 - 2024.11.29).Jan 10 2025, 2:28 PM
Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2023.12.01 - 2023.12.31).Jan 10 2025, 3:28 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits