Page MenuHomePhabricator

[Airflow] Set up scap deployment
Open, HighPublic

Description

Make it possible to deploy https://gitlab.wikimedia.org/data-engineering/airflow-dags with scap.
The deployment process should allow to specify a parameter containing the team name,
with which scap would choose the corresponding Airflow instance to deploy to,
and also point Airflow configuration (AIRFLOW_HOME, and maybe other properties) to the right place.

Event Timeline

Change 737764 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add gitlab support for scap_source

https://gerrit.wikimedia.org/r/737764

Change 737770 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] [WIP] declare airflow/data_eng scap source and target for airflow analytics instance.

https://gerrit.wikimedia.org/r/737770

Change 737783 had a related patch set uploaded (by Ottomata; author: Ottomata):

[labs/private@master] Add dummy ssh keypair for deploy_airflow keyholder agent

https://gerrit.wikimedia.org/r/737783

Change 737783 merged by Ottomata:

[labs/private@master] Add dummy ssh keypair for deploy_airflow keyholder agent

https://gerrit.wikimedia.org/r/737783

Change 737764 merged by Ottomata:

[operations/puppet@production] Add gitlab support for scap_source

https://gerrit.wikimedia.org/r/737764

Change 737770 merged by Ottomata:

[operations/puppet@production] Declare airflow-dags scap for analytics-test instance

https://gerrit.wikimedia.org/r/737770

Change 738436 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] scap::target - add $manage_ssh_key parameter

https://gerrit.wikimedia.org/r/738436

Change 738436 merged by Ottomata:

[operations/puppet@production] scap::target - add $manage_ssh_key parameter

https://gerrit.wikimedia.org/r/738436

Mentioned in SAL (#wikimedia-operations) [2021-11-12T17:15:17Z] <ottomata> restarting and arming keyholder on deploy1002 - T295380

Change 738440 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Use airflow-dags/analytics on an-launcher

https://gerrit.wikimedia.org/r/738440

Change 738440 merged by Ottomata:

[operations/puppet@production] Use airflow-dags/analytics on an-launcher

https://gerrit.wikimedia.org/r/738440

Alright we are ready to go!

Scap is being used for the analytics (an-launcher1002) and analytics-test (an-test-client1001) instances. I updated documentation:

We can switch the research and platform_eng instances to use airflow-dags whenever we are ready.

Change 742813 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/airflow@debian] Airflow 2.2.2 with extra dependencies

https://gerrit.wikimedia.org/r/742813

odimitrijevic moved this task from Incoming to Transform on the Data-Engineering board.