Page MenuHomePhabricator
Create Task
Maniphest T238905

Allow analytics-search-users members to sudo as the airflow user
Closed, ResolvedPublic
Actions

Description

In T236180 the Search team is trying to bootstrap an Apache Airflow service (https://airflow.apache.org/).
In order to properly manage it, they'd need to be able to impersonate the airflow system user and perform some related sudo command (to restart/start/stop the service, etc..) on an-airflow1001.eqiad.wmnet (Ganeti VM in the Analytics VLAN).

Currently the proposal is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/552304/1/modules/admin/data/data.yaml

Caveat: the analytics-search-users group is deployed to all the analytics hosts. We (as Analytics) don't have any Airflow service yet, but we might in the future. I can see two paths forward:

  1. We review/merge this change now and then revise the permissions in the future if needed.
  2. We create a new group called airflow-search-users (or similar) that is only deployed to an-airflow1001.eqiad.wmnet, more tedious but probably a little bit more clean from the user perms point of view.

Let me know what you think about it :)

Details

SubjectRepoBranchLines +/-
Create airflow-search-admins admin groupoperations/puppetproduction+13 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

elukey created this task.Nov 22 2019, 9:17 AM
Restricted Application added a project: SRE. · View Herald TranscriptNov 22 2019, 9:17 AM
Ottomata added a comment.Nov 22 2019, 3:36 PM

Since this instance is maintained by the search team, I think re-using analytics-search-users makes sense to me. We can re-evaluate that when if/we run an official airflow.

elukey added a comment.Nov 22 2019, 6:02 PM
In T238905#5684836, @Ottomata wrote:

Since this instance is maintained by the search team, I think re-using analytics-search-users makes sense to me. We can re-evaluate that when if/we run an official airflow.

The only drawback is that analytics-search-users is not deployed in various places, so adding sudo perms to that group will automatically push them on various hosts. No harm, just wanted to bring it up :)

• crusnov triaged this task as Medium priority.Nov 22 2019, 8:09 PM
gerritbot added a comment.Nov 23 2019, 12:19 AM

Change 552613 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] analytics/admins: create admin group and for for airflor, apply on an-airflow1001

https://gerrit.wikimedia.org/r/552613

gerritbot added a project: Patch-For-Review.Nov 23 2019, 12:19 AM
Dzahn subscribed.Nov 23 2019, 12:25 AM

probably a little bit more clean from the user perms point of view.

Yes please, let's create a new admin group.

adding sudo perms to that group will automatically push them on various hosts

Yea, let's avoid that.

more tedious

Yea, but you need a role for that host anyways to apply the admin group or we start using ./hosts/ and so on. Let's just do it right away?

I made https://gerrit.wikimedia.org/r/c/operations/puppet/+/552613

That also adds the host to the analytics cluster in Icinga, installs the base packages and gives the people access.

And then you can skip:

re-evaluate that when if/we run an official airflow

Hope that is helpful.

gerritbot added a comment.Nov 25 2019, 6:31 PM

Change 552613 merged by Elukey:
[operations/puppet@production] Create airflow-search-admins admin group

https://gerrit.wikimedia.org/r/552613

Dzahn added a comment.Nov 26 2019, 12:37 AM

The group has been created and request was approved in SRE meeting.

It needs merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/552304 though. (which looks fine to me, but the units listed in the sudo privileges don't seem to exist on an-airflow1001 yet).

Dzahn added a comment.Nov 26 2019, 1:22 AM
[an-airflow1001:~] $ id ebernhardson
uid=3088(ebernhardson) gid=500(wikidev) groups=500(wikidev),816(airflow-search-admins)

[an-airflow1001:~] $ grep airflow /etc/group
airflow:x:1001:
airflow-search-admins:x:816:ebernhardson,dcausse,gehel,bearloga,chelsyx

[an-airflow1001:~] $ sudo cat /etc/sudoers.d/airflow-search-admins 
# This file is managed by Puppet!

%airflow-search-admins ALL = NOPASSWD: /usr/sbin/service airflow-webserver *
%airflow-search-admins ALL = NOPASSWD: /usr/sbin/service airflow-scheduler *
%airflow-search-admins ALL = NOPASSWD: /bin/systemctl start airflow-scheduler
%airflow-search-admins ALL = NOPASSWD: /bin/systemctl restart airflow-scheduler
%airflow-search-admins ALL = NOPASSWD: /bin/systemctl stop airflow-scheduler
%airflow-search-admins ALL = NOPASSWD: /bin/systemctl start airflow-webserver
%airflow-search-admins ALL = NOPASSWD: /bin/systemctl stop airflow-webserver
%airflow-search-admins ALL = NOPASSWD: /bin/systemctl restart airflow-webserver
%airflow-search-admins ALL = (airflow) NOPASSWD: /srv/deployment/search/airflow/venv/bin/airflow *
Dzahn closed this task as Resolved.Nov 26 2019, 1:23 AM
Dzahn removed a project: Patch-For-Review.

I merged the change by @EBernhardson which added the new group on an-airflow1001. Puppet has created the users and sudo privileges for them as above. Please comment if something is missing.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits