Page MenuHomePhabricator
Create Task
Maniphest T295380

[Airflow] Set up scap deployment
Closed, ResolvedPublic
Actions

Description

Make it possible to deploy https://gitlab.wikimedia.org/data-engineering/airflow-dags with scap.
The deployment process should allow to specify a parameter containing the team name,
with which scap would choose the corresponding Airflow instance to deploy to,
and also point Airflow configuration (AIRFLOW_HOME, and maybe other properties) to the right place.

Details

SubjectRepoBranchLines +/-
airflow - Set up research instance and deployment of airflow-dagsoperations/puppetproduction+7 -13
Airflow 2.1.4 with extra dependenciesoperations/debs/airflowdebian+366 -274
Remove now unneeded analytics-test airflow scap target overrideoperations/puppetproduction+0 -10
Use distinct deployment and config for analytics-test airflow instancesoperations/puppetproduction+4 -3
Use airflow-dags/analytics on an-launcheroperations/puppetproduction+0 -19
scap::target - add $manage_ssh_key parameteroperations/puppetproduction+18 -0
Declare airflow-dags scap for analytics-test instanceoperations/puppetproduction+176 -16
Add gitlab support for scap_sourceoperations/puppetproduction+7 -3
Add dummy ssh keypair for deploy_airflow keyholder agentlabs/privatemaster+4 -0
Customize query in gerrit
TitleReferenceAuthorSource BranchDest Branch
Add research/ instance directoryrepos/data-engineering/airflow-dags!16ottoresearchmain
Add analytics/dags directory for testing scap deployment and configs of analytics-test instance.repos/data-engineering/airflow-dags!1ottootto-main-patch-13902main
Customize query in GitLab

Related Objects

Event Timeline

mforns created this task.Nov 9 2021, 4:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 9 2021, 4:38 PM
mforns moved this task from Backlog to Estimated on the Data Pipelines board.Nov 9 2021, 5:16 PM
gerritbot added a comment.Nov 9 2021, 6:52 PM

Change 737764 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add gitlab support for scap_source

https://gerrit.wikimedia.org/r/737764

gerritbot added a project: Patch-For-Review.Nov 9 2021, 6:52 PM
gerritbot added a comment.Nov 9 2021, 7:26 PM

Change 737770 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] [WIP] declare airflow/data_eng scap source and target for airflow analytics instance.

https://gerrit.wikimedia.org/r/737770

gerritbot added a comment.Nov 9 2021, 8:42 PM

Change 737783 had a related patch set uploaded (by Ottomata; author: Ottomata):

[labs/private@master] Add dummy ssh keypair for deploy_airflow keyholder agent

https://gerrit.wikimedia.org/r/737783

gerritbot added a comment.Nov 9 2021, 8:44 PM

Change 737783 merged by Ottomata:

[labs/private@master] Add dummy ssh keypair for deploy_airflow keyholder agent

https://gerrit.wikimedia.org/r/737783

Ottomata mentioned this in rLPRI51ddecf04c62: Add dummy ssh keypair for deploy_airflow keyholder agent.Nov 9 2021, 8:45 PM
gerritbot added a comment.Nov 10 2021, 7:29 PM

Change 737764 merged by Ottomata:

[operations/puppet@production] Add gitlab support for scap_source

https://gerrit.wikimedia.org/r/737764

gerritbot added a comment.Nov 12 2021, 3:42 PM

Change 737770 merged by Ottomata:

[operations/puppet@production] Declare airflow-dags scap for analytics-test instance

https://gerrit.wikimedia.org/r/737770

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2021, 4:10 PM
gerritbot added a comment.Nov 12 2021, 4:51 PM

Change 738436 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] scap::target - add $manage_ssh_key parameter

https://gerrit.wikimedia.org/r/738436

gerritbot added a project: Patch-For-Review.Nov 12 2021, 4:51 PM
gerritbot added a comment.Nov 12 2021, 4:57 PM

Change 738436 merged by Ottomata:

[operations/puppet@production] scap::target - add $manage_ssh_key parameter

https://gerrit.wikimedia.org/r/738436

Maintenance_bot removed a project: Patch-For-Review.Nov 12 2021, 5:10 PM
Stashbot added a comment.Nov 12 2021, 5:15 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-12T17:15:17Z] <ottomata> restarting and arming keyholder on deploy1002 - T295380

gerritbot added a comment.Nov 12 2021, 5:25 PM

Change 738440 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Use airflow-dags/analytics on an-launcher

https://gerrit.wikimedia.org/r/738440

gerritbot added a project: Patch-For-Review.Nov 12 2021, 5:25 PM
gerritbot added a comment.Nov 12 2021, 5:33 PM

Change 738440 merged by Ottomata:

[operations/puppet@production] Use airflow-dags/analytics on an-launcher

https://gerrit.wikimedia.org/r/738440

Ottomata moved this task from Next Up to Done on the Data-Engineering-Kanban board.Nov 12 2021, 5:36 PM
Ottomata moved this task from Done to Ready to Deploy on the Data-Engineering-Kanban board.
Ottomata added a comment.Nov 12 2021, 6:58 PM

Alright we are ready to go!

Scap is being used for the analytics (an-launcher1002) and analytics-test (an-test-client1001) instances. I updated documentation:

We can switch the research and platform_eng instances to use airflow-dags whenever we are ready.

gerritbot added a comment.Nov 30 2021, 11:02 PM

Change 742813 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/debs/airflow@debian] Airflow 2.2.2 with extra dependencies

https://gerrit.wikimedia.org/r/742813

odimitrijevic triaged this task as High priority.Dec 3 2021, 11:47 PM
odimitrijevic moved this task from Incoming (new tickets) to Transform on the Data-Engineering board.
Ottomata moved this task from Estimated to Done on the Data Pipelines board.Dec 13 2021, 4:41 PM
gerritbot added a comment.Dec 21 2021, 3:44 PM

Change 749201 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Use distinct deployment and config for analytics-test airflow instances

https://gerrit.wikimedia.org/r/749201

gerritbot added a comment.Dec 21 2021, 3:53 PM

Change 749201 merged by Ottomata:

[operations/puppet@production] Use distinct deployment and config for analytics-test airflow instances

https://gerrit.wikimedia.org/r/749201

gerritbot added a comment.Dec 21 2021, 4:01 PM

Change 749206 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Remove now unneeded analytics-test airflow scap target override

https://gerrit.wikimedia.org/r/749206

gerritbot added a comment.Dec 21 2021, 4:03 PM

Change 749206 merged by Ottomata:

[operations/puppet@production] Remove now unneeded analytics-test airflow scap target override

https://gerrit.wikimedia.org/r/749206

gerritbot added a comment.Dec 21 2021, 6:26 PM

Change 742813 merged by Ottomata:

[operations/debs/airflow@debian] Airflow 2.1.4 with extra dependencies

https://gerrit.wikimedia.org/r/742813

Ottomata moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.Feb 7 2022, 4:05 PM
Ottomata moved this task from Done to Ready to Deploy on the Data-Engineering-Kanban board.Feb 8 2022, 4:10 PM

TODO: set up research's deployment

fkaelin subscribed.Feb 8 2022, 4:11 PM
fkaelin added a subscriber: • bmansurov.Feb 9 2022, 5:19 PM

Thanks @Ottomata!

cc @bmansurov

gerritbot added a comment.Feb 10 2022, 5:12 PM

Change 761670 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] airflow - Set up research instance and deployment of airflow-dags

https://gerrit.wikimedia.org/r/761670

Stashbot added a comment.Feb 10 2022, 6:54 PM

Mentioned in SAL (#wikimedia-analytics) [2022-02-10T18:54:54Z] <ottomata> setting up research airflow-dags scap deployment, recreating airflow database and starting from scractch (fab okayed this) - T295380

gerritbot added a comment.Feb 10 2022, 6:57 PM

Change 761670 merged by Ottomata:

[operations/puppet@production] airflow - Set up research instance and deployment of airflow-dags

https://gerrit.wikimedia.org/r/761670

Ottomata added a comment.Feb 10 2022, 7:05 PM

Ok @fkaelin @bmansurov, the research airflow instance is now using data-engineering/airflow-dags. Please put your configs and dags in that repository in the research/ directory.

Deployment instructions here:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow#research

Ottomata moved this task from Ready to Deploy to Done on the Data-Engineering-Kanban board.Feb 15 2022, 7:47 PM
JArguello-WMF closed this task as Resolved.May 31 2022, 3:44 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits