| BTullis | |
| Apr 19 2024, 4:55 PM |
| F55498133: image.png | |
| Thu, Jun 20, 3:48 PM |
We are planning to migrate Airflow to the dse-k8s cluster.
As such, we will need a helm chart for the purpose.
There is an official upstream chart for Airflow available here:
https://airflow.apache.org/docs/helm-chart/stable/index.html
We may want to consider whether or not to use this chart as it is, or whether it serves merely as inspiration.
There is a process to follow if we wish to use the chart as it is: https://wikitech.wikimedia.org/wiki/Kubernetes/Upstream_Helm_charts_policy
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| blubber: add wmf-certificates deb pkg and work around image build failures | repos/data-engineering/airflow!7 | bking | wmf-certificates | main | |
| Enable kubernetes provider | repos/data-engineering/airflow!6 | bking | kubernetes | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Open | None | T364387 Adapt Airflow auth and DAG deployment method | |||
| Open | bking | T363001 Create a helm chart for airflow that is appropriate to our needs |
Change #1034951 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] data-engineering: add scaffolding for airflow service
Change #1034961 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] dse-k8s: add new airflow service to k8s cluster
Change #1034951 abandoned by Bking:
[operations/deployment-charts@master] data-engineering: add scaffolding for airflow service
Reason:
forgot to add prometheus statsd support
Change #1035013 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] dse-k8s: Add net-new service scaffolding for airflow
Change #1035015 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] dse-k8s: add airflow namespace
Change #1034961 merged by Bking:
[operations/puppet@production] dse-k8s: add new airflow service to k8s cluster
Change #1037077 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] dse-k8s: add new airflow service to k8s cluster
Change #1037077 merged by Bking:
[operations/puppet@production] dse-k8s: add new airflow service to k8s cluster
Change #1035015 merged by Bking:
[operations/deployment-charts@master] dse-k8s: add airflow-analytics-test namespace
bking opened https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/6
Enable kubernetes provider
bking merged https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/6
Enable kubernetes provider
Mentioned in SAL (#wikimedia-operations) [2024-06-05T13:45:31Z] <inflatador> bking@an-db1001 install acl pkg T363001
Mentioned in SAL (#wikimedia-operations) [2024-06-05T13:48:56Z] <inflatador> bking@an-db1001 install python3-psycopg2 pkg T363001
Change #1039260 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] an-db1001: add `airflow_test_k8s` user and db
Change #1039260 merged by Bking:
[operations/puppet@production] an-db1001: add `airflow_test_k8s` user and db
Change #1039838 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] dse-k8s: replace 'airflow-analytics-test' ns with 'airflow'
Change #1039838 merged by jenkins-bot:
[operations/deployment-charts@master] dse-k8s: replace 'airflow-analytics-test' ns with 'airflow'
Change #1035013 abandoned by Bking:
[operations/deployment-charts@master] dse-k8s: Add net-new service scaffolding for airflow
Reason:
starting over and using WMF scaffold
Change #1041759 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] dse-k8s-services: Add net-new chart for Airflow
bking opened https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/7
blubber: add wmf-certificates deb pkg and work around image build failures
bking merged https://gitlab.wikimedia.org/repos/data-engineering/airflow/-/merge_requests/7
blubber: add wmf-certificates deb pkg and work around image build failures
Change #1043275 had a related patch set uploaded (by Bking; author: Bking):
[operations/deployment-charts@master] dse-k8s: harmonize airflow user/namespace/db names
Change #1043277 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] dse-k8s: harmonize airflow user/namespace/db names
Change #1043277 merged by Bking:
[operations/puppet@production] dse-k8s: harmonize airflow user/namespace/db names
Change #1043275 merged by jenkins-bot:
[operations/deployment-charts@master] dse-k8s: harmonize airflow user/namespace/db names
Change #1047189 had a related patch set uploaded (by Bking; author: Bking):
[operations/puppet@production] analytics: allow dse-k8s pod network to reach an-db1001
Change #1047189 merged by Bking:
[operations/puppet@production] analytics: allow dse-k8s pod network to reach an-db1001
Just as a quick progress update, we have now succeeded in getting the airflow webserver running for a test instance, using our own airflow image.
I'm mounting an emptyDir for the logs directory and I will do the same for the gunicorn pid file, unless I can find a way to disable that.