Create airflow instances for Platform Engineering and Research
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Ottomata
	Jun 3 2021, 2:21 PM

Description

In today's Analytics Systems hangtime meeting, we talked with @fkaelin and @gmodena about work they want to do with Airflow. I told them that Airflow is basically ready for testing, and we could create instances for them now if they liked. They do like! They understand that we are still iterating and figuring it out for ourselves too. It will do us all good to be able to work out best practices together.

Let's create instances for them now. We haven't done this before, so we'll likely need to formalize this process. It will be something like:

Create new Ganeti VMs: an-airflow1002 (research), an-airflow1003 (platform eng)

Create new system users: analytics-research, analytics-platform-eng. Declare these system users in profile::analytics::cluster::users These users should also be added in admin data.yaml, but commented out until T231067 is complete (as other system users are). See analytics-search as an example.

Create new user groups analytics-research-users and analytics-platform-eng-users and include relevant users in members and system user in system_members. Members in these groups should have sudo privileges to their system user. Also include the system users in the analytics-privatedata-users group. See analytics-search-users as an example. We'll also need to make sure users in these groups can manage airflow services. See airflow-search-admins for an example. Q: should these be -admins groups instead of -users groups? Perhaps if they need sudo privs they should.

Create kerberos principals and keytabs for these system users @ their airflow VM hostname following https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos#Create_a_keytab_for_a_service

Create the airflow instances on the VMs following https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow#Creating_a_new_Airflow_Instance

Details

Subject	Repo	Branch	Lines +/-
Add analytics-research and analytics-platform-eng to analytics-privatedata-users	operations/puppet	production	+2 -1
Set up airflow@platform_eng instance on an-airflow1003	operations/puppet	production	+72 -18
Set up airflow-research instance on an-airflow1002	operations/puppet	production	+70 -4
Add dummy keytabs for analytics-research and analytics-platform-eng airflow	labs/private	master	+0 -0
Add system users and groups for for Airflow for Research and Platform Eng	operations/puppet	production	+72 -1
DRY profile::analytics::cluster::users	operations/puppet	production	+82 -89

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		odimitrijevic	T282033 Airflow collaborations
		Resolved		Ottomata	T284225 Create airflow instances for Platform Engineering and Research

Event Timeline

Ottomata created this task.Jun 3 2021, 2:21 PM

@Ottomata what is the plan for the related databases?

@elukey I think the thing to do for now is create them in analytics meta mariadb instance, and then refactor everything as part of T284150: Bring an-mariadb100[12] into service when we have new hardware.

We could consider hosting the mariadb instances on the ganeti VMs themselves. I think I'd rather have them on dedicated hardware in the same place, especially since replication setup is manual; I think it will be easier to manage that in one place.

In this case make sure to check the max-conns + innodb buffer for meta, I am pretty sure that there may be some tuning to do :)

Ottomata updated the task description. (Show Details)Jun 3 2021, 3:12 PM

odimitrijevic moved this task from Incoming to Smart Tools for Better Data on the Analytics board.Jun 3 2021, 4:39 PM

BPirkle moved this task from Inbox to Tracking/Watching on the Platform Engineering board.Jun 8 2021, 9:06 PM

@razzi, I think we can go ahead and create an-airflow1002 for platform eng. Could you make that happen? :) TY!

Ottomata added a project: User-razzi.Jun 11 2021, 6:43 PM

Sounds good @Ottomata, creating vms in https://phabricator.wikimedia.org/T284934

• razzi moved this task from Default to Ready for action on the User-razzi board.Jun 14 2021, 2:58 PM

• razzi mentioned this in T284934: Site: 2 VM request for an-airflow100{2,3}.Jun 21 2021, 6:43 PM

Oh right, there is more to do than just https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow#Creating_a_new_Airflow_Instance. I'll see if I can figure out some of the earlier system user steps now.

Change 701112 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] DRY profile::analytics::cluster::users

https://gerrit.wikimedia.org/r/701112

gerritbot added a project: Patch-For-Review.Jun 23 2021, 2:25 PM

Change 701112 merged by Ottomata:

[operations/puppet@production] DRY profile::analytics::cluster::users

https://gerrit.wikimedia.org/r/701112

Maintenance_bot removed a project: Patch-For-Review.Jun 23 2021, 3:10 PM

mforns moved this task from Smart Tools for Better Data to Airflow on the Analytics board.Jun 25 2021, 3:49 PM

• razzi moved this task from Ready for action to Default on the User-razzi board.Jul 12 2021, 8:19 PM

Hey yall, quick update: We haven't been working on this as the Gobblin migration has been taking a lot of my time (should be done soon), but also I'm hoping to wait until T278423 is 100% done (should be early next week). Once everything is on Buster, some of the system user account creation can be improved and made much easier from our side. We'll have to do that to make your airflow system users.

So, I hope to get to work on this again in the next couple of weeks.

• razzi removed a project: User-razzi.Jul 19 2021, 8:15 PM

Blocked by T287063: Refactor profile::analytics::cluster::users

Ottomata claimed this task.Jul 26 2021, 7:01 PM

Ottomata updated the task description. (Show Details)Jul 26 2021, 7:18 PM

Change 708159 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add system users and groups for for Airflow for Research and Platform Eng

https://gerrit.wikimedia.org/r/708159

gerritbot added a project: Patch-For-Review.Jul 26 2021, 7:21 PM

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Jul 27 2021, 7:20 PM

Change 708159 merged by Ottomata:

[operations/puppet@production] Add system users and groups for for Airflow for Research and Platform Eng

https://gerrit.wikimedia.org/r/708159

Created kerberos principals and keytabs:

[@krb1001:/home/otto] $ cat airflow-keytabs.list
an-airflow1002.eqiad.wmnet,create_princ,analytics-research
an-airflow1002.eqiad.wmnet,create_keytab,analytics-research
an-airflow1003.eqiad.wmnet,create_princ,analytics-platform-eng
an-airflow1003.eqiad.wmnet,create_keytab,analytics-platform-eng

[@krb1001:/home/otto] $ sudo generate_keytabs.py --realm WIKIMEDIA airflow-keytabs.list
analytics-research/an-airflow1002.eqiad.wmnet@WIKIMEDIA

Created airflow databases on an-coord1001:

CREATE DATABASE airflow_research;
CREATE USER 'airflow_research' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON airflow_research.* TO 'airflow_research';

CREATE DATABASE airflow_platform_eng;
CREATE USER 'airflow_platform_eng' IDENTIFIED BY 'xxxxxx';
GRANT ALL PRIVILEGES ON airflow_platform_eng.* TO 'airflow_platform_eng';

Maintenance_bot removed a project: Patch-For-Review.Jul 28 2021, 4:11 PM

Change 708583 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Install hadoop client on an-airflow1002

https://gerrit.wikimedia.org/r/708583

gerritbot added a project: Patch-For-Review.Jul 28 2021, 6:41 PM

Change 708590 had a related patch set uploaded (by Ottomata; author: Ottomata):

[labs/private@master] Add dummy keytabs for analytics-research and analytics-platform-eng airflow

https://gerrit.wikimedia.org/r/708590

Change 708590 merged by Ottomata:

[labs/private@master] Add dummy keytabs for analytics-research and analytics-platform-eng airflow

https://gerrit.wikimedia.org/r/708590

Ottomata mentioned this in rLPRI6201d34dcf89: Add dummy keytabs for analytics-research and analytics-platform-eng airflow.Jul 28 2021, 8:52 PM

Change 708583 merged by Ottomata:

[operations/puppet@production] Set up airflow-research instance on an-airflow1002

https://gerrit.wikimedia.org/r/708583

Maintenance_bot removed a project: Patch-For-Review.Jul 29 2021, 12:11 AM

Change 708609 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Set up airflow@platform_eng instance on an-airflow1003

https://gerrit.wikimedia.org/r/708609

gerritbot added a project: Patch-For-Review.Jul 29 2021, 12:20 AM

Change 708609 merged by Ottomata:

[operations/puppet@production] Set up airflow@platform_eng instance on an-airflow1003

https://gerrit.wikimedia.org/r/708609

Ok! @fkaelin @gmodena @Clarakosi I've set up airflow instances for you all! Consider them still a bit WIP and don't start to rely on them yet, but you can experiment and develop for sure!

We still need lots more documentation, but instructions for accessing your instances are here:

All the users in the platform-engineering posix group can log into the airflow1003.eqiad.wmnet instance.

@fkaelin, for now, I only added you in the new analytics-research-admins group. Let me know who else to add!

I need to follow up and make sure database backups and replication work properly, but I think I will do that as part of T284150: Bring an-mariadb100[12] into service once we get hardware.

Ottomata moved this task from In Progress to Done on the Analytics-Kanban board.Jul 29 2021, 2:51 PM

@Ottomata - FYI I spotted this on an-test-coord1001 this morning.

Warning: /Stage[main]/Profile::Airflow/Airflow::Instance[analytics-test]/File[/srv/airflow-analytics-test]: Could not back up file of type directory
Notice: /Stage[main]/Profile::Airflow/Airflow::Instance[analytics-test]/File[/srv/airflow-analytics-test]: Not removing directory; use 'force' to override
Notice: /Stage[main]/Profile::Airflow/Airflow::Instance[analytics-test]/File[/srv/airflow-analytics-test]/ensure: removed (corrective)

I guess it's just trying to do an ensure absent on a directory here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/airflow/manifests/instance.pp#191

We could fix it for now by manually removing the directory, but I thought you might like to know about it for future reference.
I think it's probably OK just to add a force => true to the file resource as well.

Oh thanks, cool, that's from when we moved this instance over to an-test-client for kerberos reasons. Hm yeah let's add force => true. doing.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/709044

Hey @Ottomata,

Many thanks for this! Just wanted to give an ack that login on the host worked.

https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow#platform-eng

Terrific!

All the users in the platform-engineering posix group can log into the airflow1003.eqiad.wmnet instance.

Minor thing; I guess the instance host is an-airflow1003.eqiad.wmnet (just for future reference).

OO yup.

Ottomata closed this task as Resolved.Sep 1 2021, 1:35 PM

Submitting a spark job from an airflow instance results in a hadoop/hdfs permission error AccessControlException: Permission denied: user=analytics-research, access=WRITE, inode="/user":hdfs:hadoop:drwxrwxr-x.

The example job doesn't read or write from hdfs, presumably it is spark attempting to copy temp files. Is this likely submit configuration issue, or does this user need additional permissions and/or a home directory to run spark/hadoop jobs?

I'm a little fuzzy here but I do know this is because there's no /user/analytics-research directory in hdfs, and that we can create such a thing, as we did for analytics-search for example. I'm not sure what the process is, so I made T290918

Change 721601 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add analytics-research and analytics-platform-eng to analytics-privatedata-users

https://gerrit.wikimedia.org/r/721601

Change 721601 merged by Ottomata: