I added "level 1", "level 2", "level 3" to the doc page at https://wikitech.wikimedia.org/w/index.php?title=Data_Platform/Data_access&diff=prev&oldid=2347634. Let's see if folks think that's a useful change. If not feel free to revert.

Adding the team responsible for access management and the SME in this topic.

Most of these have underlying technical reasons, I'll defer to the Data Platform SREs if they want to reorganise the access or not

Split analytics-privatedata-users into 3 groups in data.yaml that have more precise names. Then folks can say precisely what flavor of it they are applying for. (Hat tip Dzahn for the idea.)

I'd be strongly not in favor of this, since the actual setup is the end result of a very painful refactoring where multiple groups were used. The main problem at the time was that the group names diverged from their semantic after little time, and people were really confused about what group to be in. Let's keep a single POSIX group, and possibly improve the rest.

Perhaps the lowest level of analytics-privatedata-users can be converted to LDAP instead? That level is checked by Superset and Turnilo and does not involve logging into production servers, so it is a bit odd to me that it is a shell group and not LDAP.

This is needed because the uid that Superset/Turnilo see in their auth request will eventually be matched with the one on HDFS/Hadoop, that use POSIX groups to allow/disallow access to a specific datadir.

Is Kerberos needed? Could that be refactored to use shell groups instead?

Yep it is needed, it is the only security fence that we have to prevent anybody having access to PII data.

Perhaps there is an opportunity to reorganize things to be less confusing.

Could we get some clarity about what is confusing about the current workflow?

In T405517#11276521, @elukey wrote:

Could we get some clarity about what is confusing about the current workflow?

1. The division of https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Access_Levels into 5 levels that involve simultaneous access requests on up to 3 systems (LDAP, POSIX, Kerberos).

2. Additionally, I think analytics-privatedata-users may be the only POSIX request that involves sometimes submitting a blank SSH key, which is not the normal workflow for POSIX requests.

3. The lack of precise names for each of these 5 levels. I boldly added some level numbers in this edit that may help, although I notice that no one has adopted this system yet in Phabricator SRE-Access-Requests tickets.

Overall, this just seems like the most complicated kind of access to request, requiring lots of extra knowledge and steps for both the requestor and the SRE on clinic duty.

@Dzahn, would you agree that these are the main points of confusion? Did I miss anything?

Thanks all for raising this ticket and for your kind feedback so far. I totally agree that:

analytics-privatedata-users is confusing for both applicants, and the SREs that action the applications.

I hope that we can make some quick-win improvements of the docs and processes that will benefit all of these stakeholders.
However, as @MoritzMuehlenhoff and @elukey mentioned, the underlying reason for the granularity in the levels of access is that there is complexity in the underlying systems.

These systems are always evolving and we try to reduce complexity where we can, but not at the expense of security or privacy.
That's why I think that it is more likely that we will be able to get some improvements in the short-term around making things clearer, rather than implementing the systems-level changes that have been suggested.
There are several longer-term project in progress at the moment, which I hope will have a beneficial impact on the user experience of Data-Platform users. I can link to these projects separately.

I'll try to answer your specific points inline.

In T405517#11276576, @Novem_Linguae wrote:

In T405517#11276521, @elukey wrote:

Could we get some clarity about what is confusing about the current workflow?

1. The division of https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Access_Levels into 5 levels that involve simultaneous access requests on up to 3 systems (LDAP, POSIX, Kerberos).

I'd have to say that even 5 levels is just an arbitrary selection for this level of granularity, at the moment. There is such a wide range of tasks that people are trying to carry out, that it is difficult to pigeon-hole these requests easily.

My experience is that, in general, many users requesting access to data do not necessarily know in detail which systems they might need to use to access and manipulate that data.
They may not know if they need to use the Presto or Spark query engines, or Superset or Jupyter or Turnilo front-ends, or indeed whether their data is stored in Druid or Hive or MariaDB or Cassandra.

The way that the documentation is written at the moment is intended to help us (SRE, Data Platform SRE, WMF in general) to apply the principle of least privilege, but I do appreciate that it can be very confusing.
One thing that probably adds to the confusion is that we often suggest that the user submit their own access requests. These requests can be picked up either by SRE during clinic duty, or by Data-Platform-SRE ad-hoc, sometimes depending on which tags or subscribers the user adds when creating the ticket, as well as the perceived urgency of the request.

Maybe it would help if we had a more systematic approach of:

1. Having an established colleague submit an access request on behalf of the applicant, particularly in the case of new staff members.
2. Changing the wording of the template to raise the relative priority what the applicant wants to do - rather than what rights they believe that they need.
3. Perhaps shifting greater responsibility of checking that the correct access rights are proposed to the Data-Platform-SRE team.

2. Additionally, I think analytics-privatedata-users may be the only POSIX request that involves sometimes submitting a blank SSH key, which is not the normal workflow for POSIX requests.

This is true. It is a side-effect of the way that HDFS file system permissions work. They are not truly POSIX and do not rely on users having numeric uid or gid values, or an active shell account on any system.
So, for example, if a specific user needs access to a Superset dashboard that uses data stored on HDFS, they do not need a shell account on any system, but the HDFS namenodes need to know that user is a member of analytics-privatedata-users. Having a blank SSH key is currently the most efficient way for us to achieve this, but I agree that it is counterituitive.

3. The lack of precise names for each of these 5 levels. I boldly added some level numbers in this edit that may help, although I notice that no one has adopted this system yet in Phabricator SRE-Access-Requests tickets.

I do appreciate your edit and your intent, but maybe we could get more clarity by another means than just implementing numeric levels.

There is this section at the bottom of the page: https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#Accounts_explained - It begins with:

There are too many different accounts and passwords one has to deal with in order to access analytics systems. For now it's what we've got. Let's try to explain them all explicitly.

Perhaps we could move this section nearer the top and review the text to make it clearer what applicants might need.

Overall, this just seems like the most complicated kind of access to request, requiring lots of extra knowledge and steps for both the requestor and the SRE on clinic duty.

Agreed. We are working to reduce the complexity, but it is a multi-year effort.

@Dzahn, would you agree that these are the main points of confusion? Did I miss anything?

Yes, I agree and can generally confirm 2 things:

The majority of all access requests is for analytics related groups and

on nearly every request the requestor themselves are not sure what groups they really need to request.

Then the SRE on clinic duty tries to figure out which of the different levels it is. We end up guessing more often than not. Sometimes we err on the side of caution and start with the lowest level and later the ticket gets reopened to add more. In other cases we likely give out more access than actually needed.

The combination of:

a) asking the user themselves to create their request (rather than doing it as a service for them)

b) not usually having direct input from the people who designed the groups, specifically when no more group approval is needed

c) the managers not being familiar with what specifically they are approving

d) the access requests being handled by a different SRE each week

leads to a situation where nobody involved is really sure and tickets taking longer to resolve.

Here is a recent example: T408164

The link above is common example. The user asks for analytics-privatedata-users (or is told to ask for it as part of some onboarding docs).

But that is not sufficient to know what they actually need.

Even if they add information like "superset/turnilo" we still have at least 2 options as described on:

https://wikitech.wikimedia.org/wiki/Data_Platform/Data_access#What_access_should_I_request?

Now the SRE would have to send them back to the docs and the (usually brandnew) user would have to figure out whether they need private data or not and whether they need shell or not, before they ever had access. This tends to lead to more access than needed because "just in case".

Or alternatively the SRE processing the ticket has to start off-ticket communication to ask what they _really_ need and then try to match that with one of the levels.

Or finally, the SRE starts with the lowest type of access to follow the principle of least access needed.. but this often leads to tickets being reopened to upgrade access to higher levels and an overall longer turn around time.

Imho this could be improved by _not_ making the new user request it themselves. Instead the access could be requested by a manager or hiring manager. They should know best what access is actually needed. (Even if just to be able to give proper approvals). SRE could create templates or forms for that to make it simple leading to more consistency overall.

Thanks a lot for the feedback @Dzahn!

I checked T408164 and it seems to me that there were two things that caused delay:

• The user asking to be added to ariflow-specific admin groups.
• The user not knowing what data access to request for their needs (again related to Airflow).

As you mentioned these long tasks seem to be caused by a new WMF member asked to create the task from their team without a good and clear guidance of what to request. The analytics-privatedata-users group is described in the docs, but maybe we could improve the Phab template listed in https://wikitech.wikimedia.org/wiki/SRE/Production_access#Filing_the_request to specifically link/list Data Platform accesses? I am pretty sure a lot of new people get bounced to multiple links to read without having a clear summary of how things are linked together.

@BTullis probably it is also worth to expand our data platform access docs with a reference of what to ask for Airflow, and/or what not to ask (like it was pointed out in T408164#11306467.

Lemme know your thoughts :)

More examples:

T411679 - requestor actively says they don't know the level - request gets approved regardless - discussion on actual code review if shell access is needed or not and what other types of access are requested

T409854 - user is about to get kerberos - discussion starts - then does not get kerberos after all

It's like this on an ongoing basis.

It may be helpful to know that the Data-Engineering and Data-Platform-SRE teams are currently working on a project to migrate our Data Lake away from Hadoop and HDFS, to Kubernetes and Ceph/S3.
We are currently devising a security model for this future version of the in T428255: Define a comprehensive security model for the Data Platform S3/Ceph services.

When complete, this is expected to allow us to deprecate many of the POSIX groups in data.yaml and simplify access request procedures.

Access control procedures will build on the work of the Infrastructure-Foundations team and the IDM systems, so group membership will largely be backed by LDAP and the approvals should be a case of self-service requests via https://idm.wikimedia.org/permissions/

We're still in the early stages of implementation, so feel free to share your input on the draft proposal linked from that ticket, or through any other channel you feel may help.