Page MenuHomePhabricator
Create Task
Maniphest T295380

[Airflow] Set up scap deployment
Closed, ResolvedPublic
Actions

Description

Make it possible to deploy https://gitlab.wikimedia.org/data-engineering/airflow-dags with scap.
The deployment process should allow to specify a parameter containing the team name,
with which scap would choose the corresponding Airflow instance to deploy to,
and also point Airflow configuration (AIRFLOW_HOME, and maybe other properties) to the right place.

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+7 -13airflow - Set up research instance and deployment of airflow-dags
operations/debs/airflowdebian+366 -274Airflow 2.1.4 with extra dependencies
operations/puppetproduction+0 -10Remove now unneeded analytics-test airflow scap target override
operations/puppetproduction+4 -3Use distinct deployment and config for analytics-test airflow instances
operations/puppetproduction+0 -19Use airflow-dags/analytics on an-launcher
operations/puppetproduction+18 -0scap::target - add $manage_ssh_key parameter
operations/puppetproduction+176 -16Declare airflow-dags scap for analytics-test instance
operations/puppetproduction+7 -3Add gitlab support for scap_source
labs/privatemaster+4 -0Add dummy ssh keypair for deploy_airflow keyholder agent
Customize query in gerrit
ReferenceSource BranchDest BranchAuthorTitle
repos/data-engineering/airflow-dags!16researchmainottoAdd research/ instance directory
repos/data-engineering/airflow-dags!1otto-main-patch-13902mainottoAdd analytics/dags directory for testing scap deployment and configs of analytics-test instance.
Customize query in GitLab

Related Objects

Event Timeline

mforns created this task.Nov 9 2021, 4:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 9 2021, 4:38 PM
mforns moved this task from Backlog to Estimated on the Data Pipelines board.Nov 9 2021, 5:16 PM
gerritbot added a comment.Nov 9 2021, 6:52 PM

Change 737764 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add gitlab support for scap_source

https://gerrit.wikimedia.org/r/737764

gerritbot added a project: Patch-For-Review.Nov 9 2021, 6:52 PM
gerritbot added a comment.Nov 9 2021, 7:26 PM

Change 737770 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] [WIP] declare airflow/data_eng scap source and target for airflow analytics instance.

https://gerrit.wikimedia.org/r/737770

gerritbot added a comment.Nov 9 2021, 8:42 PM

Change 737783 had a related patch set uploaded (by Ottomata; author: Ottomata):

[labs/private@master] Add dummy ssh keypair for deploy_airflow keyholder agent

https://gerrit.wikimedia.org/r/737783

gerritbot added a comment.Nov 9 2021, 8:44 PM

Change 737783 merged by Ottomata:

[labs/private@master] Add dummy ssh keypair for deploy_airflow keyholder agent

https://gerrit.wikimedia.org/r/737783

Ottomata mentioned this in rLPRI51ddecf04c62: Add dummy ssh keypair for deploy_airflow keyholder agent.Nov 9 2021, 8:45 PM
gerritbot added a comment.Nov 10 2021, 7:29 PM

Change 737764 merged by Ottomata:

[operations/puppet@production] Add gitlab support for scap_source

https://gerrit.wikimedia.org/r/737764

gerritbot added a comment.Nov 12 2021, 3:42 PM

Change 737770 merged by Ottomata:

[operations/puppet@production] Declare airflow-dags scap for analytics-test instance

https://gerrit.wikimedia.org/r/737770

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2021, 4:10 PM
gerritbot added a comment.Nov 12 2021, 4:51 PM

Change 738436 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] scap::target - add $manage_ssh_key parameter

https://gerrit.wikimedia.org/r/738436

gerritbot added a project: Patch-For-Review.Nov 12 2021, 4:51 PM
gerritbot added a comment.Nov 12 2021, 4:57 PM

Change 738436 merged by Ottomata:

[operations/puppet@production] scap::target - add $manage_ssh_key parameter

https://gerrit.wikimedia.org/r/738436

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2021, 5:10 PM
Stashbot added a comment.Nov 12 2021, 5:15 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-12T17:15:17Z] <ottomata> restarting and arming keyholder on deploy1002 - T295380

gerritbot added a comment.Nov 12 2021, 5:25 PM

Change 738440 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Use airflow-dags/analytics on an-launcher

https://gerrit.wikimedia.org/r/738440

gerritbot added a project: Patch-For-Review.Nov 12 2021, 5:25 PM
gerritbot added a comment.Nov 12 2021, 5:33 PM

Change 738440 merged by Ottomata:

[operations/puppet@production] Use airflow-dags/analytics on an-launcher

https://gerrit.wikimedia.org/r/738440

Ottomata moved this task from Next Up to Done on the Data-Engineering-Kanban board.Nov 12 2021, 5:36 PM
Ottomata moved this task from Done to Ready to Deploy on the Data-Engineering-Kanban board.
Ottomata added a comment.Nov 12 2021, 6:58 PM

Alright we are ready to go!

Scap is being used for the analytics (an-launcher1002) and analytics-test (an-test-client1001) instances. I updated documentation:

We can switch the research and platform_eng instances to use airflow-dags whenever we are ready.

gerritbot added a comment.Nov 30 2021, 11:02 PM

Change 742813 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/airflow@debian] Airflow 2.2.2 with extra dependencies

https://gerrit.wikimedia.org/r/742813

odimitrijevic triaged this task as High priority.Dec 3 2021, 11:47 PM
odimitrijevic moved this task from Incoming to Transform on the Data-Engineering board.
Ottomata moved this task from Estimated to Done on the Data Pipelines board.Dec 13 2021, 4:41 PM
gerritbot added a comment.Dec 21 2021, 3:44 PM

Change 749201 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Use distinct deployment and config for analytics-test airflow instances

https://gerrit.wikimedia.org/r/749201

gerritbot added a comment.Dec 21 2021, 3:53 PM

Change 749201 merged by Ottomata:

[operations/puppet@production] Use distinct deployment and config for analytics-test airflow instances

https://gerrit.wikimedia.org/r/749201

gerritbot added a comment.Dec 21 2021, 4:01 PM

Change 749206 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Remove now unneeded analytics-test airflow scap target override

https://gerrit.wikimedia.org/r/749206

gerritbot added a comment.Dec 21 2021, 4:03 PM

Change 749206 merged by Ottomata:

[operations/puppet@production] Remove now unneeded analytics-test airflow scap target override

https://gerrit.wikimedia.org/r/749206

gerritbot added a comment.Dec 21 2021, 6:26 PM

Change 742813 merged by Ottomata:

[operations/debs/airflow@debian] Airflow 2.1.4 with extra dependencies

https://gerrit.wikimedia.org/r/742813

Ottomata moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.Feb 7 2022, 4:05 PM
Ottomata moved this task from Done to Ready to Deploy on the Data-Engineering-Kanban board.Feb 8 2022, 4:10 PM

TODO: set up research's deployment

fkaelin added a subscriber: fkaelin.Feb 8 2022, 4:11 PM
fkaelin added a subscriber: bmansurov.Feb 9 2022, 5:19 PM

Thanks @Ottomata!

cc @bmansurov

gerritbot added a comment.Feb 10 2022, 5:12 PM

Change 761670 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] airflow - Set up research instance and deployment of airflow-dags

https://gerrit.wikimedia.org/r/761670

Stashbot added a comment.Feb 10 2022, 6:54 PM

Mentioned in SAL (#wikimedia-analytics) [2022-02-10T18:54:54Z] <ottomata> setting up research airflow-dags scap deployment, recreating airflow database and starting from scractch (fab okayed this) - T295380

gerritbot added a comment.Feb 10 2022, 6:57 PM

Change 761670 merged by Ottomata:

[operations/puppet@production] airflow - Set up research instance and deployment of airflow-dags

https://gerrit.wikimedia.org/r/761670

Ottomata added a comment.Feb 10 2022, 7:05 PM

Ok @fkaelin @bmansurov, the research airflow instance is now using data-engineering/airflow-dags. Please put your configs and dags in that repository in the research/ directory.

Deployment instructions here:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow#research

Ottomata moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.Feb 15 2022, 7:47 PM
JArguello-WMF closed this task as Resolved.May 31 2022, 3:44 PM
Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL