Change 379004 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Allow admin module to ensure system user membership in managed groups

https://gerrit.wikimedia.org/r/379004

This is solely for T174110 or are we anticipating other use cases?

The high level implementation idea seems okay, but wouldn't it be cleaner to simply grant the Hadoop (or whatever) privilege a service needs in the puppet profile/role which sets up the service and creates the system user? It's the principle of least surprise and we can tailor the privileges to the minimal set the service needs. We might e.g. extend the scope of analytics-privatedata-users in the future in a way not necessary or risky for the services using it via a system group. Also if that service is ever removed, the privileges will also be removed in the context of the removal of the service without indirections via data.yaml (which are easy to miss).

Ι 'll echo Moritz on this one. It does look like adding system users to the admin module adds some complexity and does break the principle of least surprise as well as the KISS principle. I am wondering how much overhead it would be to create system users and groups in the puppetization of the service and provide the access required via that.

In T174465#3908880, @MoritzMuehlenhoff wrote:

This is solely for T174110 or are we anticipating other use cases?

I know that @GoranSMilovanovic will need to use this to finish puppetizing the Wikidata Concepts Monitor work. (He's currently doing what we're doing, which is running everything under his staff account.)

Analytics might need this too because if I remember right their puppetized ReportUpdater jobs run in some weird configuration to allow for Hadoop private data access. (Although Andrew or @mforns would need to confirm.)

@chelsyx @Tbayer and I will probably need to have automated-via-puppet Readers metrics calculation at some point in 2018.

Yeah, this is a very common request, and in the past we've told people we can't do it. Their only recourse then is to not puppetize/productionize things, and keep them running in their home dirs and user crons. Last summer when we moved people from e.g. stat1002 -> stat1005, this caused a lot of headaches, as the teams were relying on unpuppetized things for their reports.

The problem is that access to data in HDFS is granted via posix group membership on the NameNode. E.g. webrequest data is: drwxr-x--- 6 hdfs analytics-privatedata-users. In order to read this data, a user account must belong to the analytics-privatedata-users group on analytics1001.

For Analytics team productionization of jobs, we get around this by running crons as the hdfs user. This is not ideal on its own, as hdfs is the Hadoop super user, and has privileges akin to root inside of Hadoop. We can deal with this as the Analytics team, but we don't want to run jobs that other teams control as hdfs. And we don't want to puppetize/productionize jobs that other teams control as real human users, as those accounts come and go with employment and contracts.

So, we need a way to ensure that a system user and a human user have the ability to access the same restricted data via regular old posix permissions. If there's an idea out there other than putting system users and human users in the same group, I'm all ears!

Change 410763 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] WIP: Force jenkins-slave being member of docker

https://gerrit.wikimedia.org/r/410763

Change 410763 merged by Alexandros Kosiaris:
[operations/puppet@production] Force jenkins-slave being member of docker

https://gerrit.wikimedia.org/r/410763

@akosiaris @MoritzMuehlenhoff, I need to resurrect this task. We also need this in order for the druid user to access hdfs:analytics-privatedata-users owned data.

Any thoughts since my last comment?

Another bump for my friends @akosiaris and @MoritzMuehlenhoff :)

Change 379004 merged by Ottomata:
[operations/puppet@production] Allow admin module to ensure system user membership in managed groups

https://gerrit.wikimedia.org/r/379004

Change 438075 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Fix join call in groupmembers.pp

https://gerrit.wikimedia.org/r/438075

Change 438075 merged by Ottomata:
[operations/puppet@production] Fix default_member in groupmembers.pp

https://gerrit.wikimedia.org/r/438075

Change 438081 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Ensure system_members are in user groups

https://gerrit.wikimedia.org/r/438081

Change 438081 merged by Ottomata:
[operations/puppet@production] Ensure system_members are in user groups

https://gerrit.wikimedia.org/r/438081

Change 438083 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Ensure analytics cluster users are on notebook machines

https://gerrit.wikimedia.org/r/438083

Change 438083 merged by Ottomata:
[operations/puppet@production] Ensure analytics cluster users are on notebook machines

https://gerrit.wikimedia.org/r/438083

Change 438087 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Include analytics cluster users on stat1004

https://gerrit.wikimedia.org/r/438087

Change 438087 merged by Ottomata:
[operations/puppet@production] Include analytics cluster users on stat1004

https://gerrit.wikimedia.org/r/438087