Requesting Creation of a new POSIX group and system user for the Analytics WMDE team.
Closed, ResolvedPublicRequest
Actions

Assigned To

Authored By

	Stevemunene
	Sep 6 2023, 11:44 AM

Description

Requestor provided information and prerequisites

As part of T342331 [EPIC] Set up a sustainable tech stack for Wikidata Analytics, WMDE team require an airflow instance to run various analytics jobs, which is being worked on here T340648 [Airflow] Setup Airflow instance for WMDE.
We need a system user to help manage the WMDE related analytics/data jobs and the admin tasks around it like start, stop, restart airflow services.
Thus we need to create a system user analytics-wmde same (uid/gid) across nodes (airflow, stat100x, hadoop worker nodes, etc..). Then and add the related system user to the analtyics-privatedata-users group for the user to carry out certain functions like submitting jobs to yarn and to run regular airflow services maintenance.

Next we shall add a group of users approved by WMDE Engineering Manager for

Managing WMDE related analytics/data jobs as Analytics WMDE Aiflow admins. Using the system user analytics-wmde
The same group of users shall used to deploy airflow dags.

Details

	Subject	Repo	Branch	Lines +/-
	admin: Create analytics-wmde system user and airflow admin group	operations/puppet	production	+41 -6

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		karapayneWMDE	T342331 [EPIC] Set up a sustainable tech stack for Wikidata Analytics
Resolved		Stevemunene	T340648 [Airflow] Setup Airflow instance for WMDE
Resolved		Stevemunene	T342424 eqiad: 1 VM request for WMDE Airflow
Resolved	Request	Stevemunene	T342546 Requesting access to analytics-wmde-users (no kerberos, with ssh) for karapayneWMDE
Resolved	Request	joanna_borun	T345726 Requesting Creation of a new POSIX group and system user for the Analytics WMDE team.

Event Timeline

Stevemunene created this task.Sep 6 2023, 11:44 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2023, 11:44 AM

Stevemunene renamed this task from Requesting access to RESOURCE for USER[S] to Requesting Creation of a new POSIX group and system user for the Analytics WMDE team..Sep 6 2023, 12:01 PM

Stevemunene updated the task description. (Show Details)

Stevemunene added a project: Data-Platform-SRE.

Stevemunene added parent tasks: T340648: [Airflow] Setup Airflow instance for WMDE, T342331: [EPIC] Set up a sustainable tech stack for Wikidata Analytics.

Change 949001 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] [WIP] admin: Create analytics-wmde system user and airflow admin group

https://gerrit.wikimedia.org/r/949001

gerritbot added a project: Patch-For-Review.Sep 6 2023, 1:45 PM

Hi @joanna_borun -- does this need Infrastructure Foundations approval?

Gehel moved this task from Incoming to In Progress on the Data-Platform-SRE board.Sep 13 2023, 8:49 AM

In T345726#9158579, @RLazarus wrote:

Hi @joanna_borun -- does this need Infrastructure Foundations approval?

I think you're right. We added a similar system group for another Airflow instance back in May (https://gerrit.wikimedia.org/r/c/operations/puppet/+/914788) and that required discussion in the I/F meeting, as I understand it.

There's a slight difference with this request, which is that analytics-wmde is already a pre-existing normal user account and group.
This analytics-wmde user and group only exists on stat1007 at the moment. See https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/statistics/manifests/wmde.pp
However this account no longer has any active jobs running on stat1007, so we are going to reuse the name of the user and group for running Airflow.

Gehel moved this task from In Progress to Blocked / Waiting on the Data-Platform-SRE board.Sep 15 2023, 8:21 AM

Hi @odimitrijevic ,
Requesting approval for adding the analytics-wmde user to analtyics-privatedata-users group for T340648.

approved

We discussed this in our Monday I/F meeting and approved it.

Change 949001 merged by CDanis:

[operations/puppet@production] admin: Create analytics-wmde system user and airflow admin group

https://gerrit.wikimedia.org/r/949001

Maintenance_bot removed a project: Patch-For-Review.Sep 26 2023, 2:32 PM

Will be live in half an hour.

Requesting Creation of a new POSIX group and system user for the Analytics WMDE team.Closed, ResolvedPublicRequestActions