I am Justin's manager and I approve these requests.

Approved.

@RLazarus sorry for the nuisance, but we would appreciate if this ticket could be expedited. 🙏

Following the procedure as defined at: https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests

I verified that the supplied SSH key has not been used in Cloud Services with:

btullis@ldap-maint1001:~$ cross-validate-accounts --username jmoore111 --uid 100902 --email jmoore@wikimedia.org --ssh-key "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGVgMtLkTSTQyFyTuFAhdTcopPCMcirC0W7z8RObDXPv" --real-name "Justin Moore" --kerberos
btullis@ldap-maint1001:~$ echo $?
0

I haven't yet verified the SSH key with an out-of-band check, but I'll prepare the patch to data.yaml and mark it with -1 until this step is done.

Change #1198504 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/puppet@production] Configure production shell access and posix groups for jmoore111

https://gerrit.wikimedia.org/r/1198504

I have created the patch to enable shell access and configure the requested groups: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198504 - but downvoted it and marked it as WIP until I can check the SSH key and complete the approvals process. As per: data.yaml, we require explicit approval from @mpopov for membership of the analytics-product-users group.

Membership of the analytics-privatedata-users group no longer requires explicit approval for WMF staff, based on the guidance here.
However, @Ahoelzl has already stated his approval on this ticket, anyway.

The analytics-search-users group has no approver field, so I think that this is fine.

@JMoore-WMF There has been a recent change in the procedure regarding membership of the wmf LDAP group.

Whilst I believe that all of the relevant approvals are in place, and therefore I could add you to that group now, the procedure states that I should direct you to these self-service instructions for requesting membership of the wmf group.

• https://wikitech.wikimedia.org/wiki/SRE/LDAP/Groups/Request_access#Using_the_Wikimedia_Identity_Management_System

You don't need to request access to the nda group, since wmf will grant you the rights that you need.
If you could let me know how you get on with that request, I'd be grateful.

@JMoore-WMF has supplied me with an SSH key via our authenticated WMF Slack org, but said this:

I just realized that the public key i included in the ticket was the wrong one- accidentally included the one for Wikimedia cloud SSH access
will update the ticket accordingly

I had previously run the cross-validation against this key as shown in T408164#11305495 so I am confused as to why this didn't get picked up.
It could be one of either:

1. The cross-validate script didn't work correctly.
2. The key isn't actually used in WMCS, although perhaps it is/was intended to be used there.
3. I executed the check incorrectly.

I have updated the patch with the new SSH key.

So now we are only waiting for approval from @mpopov for the analytics-product-users group membership.

In T408164#11305956, @BTullis wrote:

@JMoore-WMF has supplied me with an SSH key via our authenticated WMF Slack org, but said this:

I just realized that the public key i included in the ticket was the wrong one- accidentally included the one for Wikimedia cloud SSH access
will update the ticket accordingly

I had previously run the cross-validation against this key as shown in T408164#11305495 so I am confused as to why this didn't get picked up.
It could be one of either:

1. The cross-validate script didn't work correctly.
2. The key isn't actually used in WMCS, although perhaps it is/was intended to be used there.
3. I executed the check incorrectly.

I don't think the script works. I have been checking by querying for the user with ldapsearch -x 'uid=Jmoore111'. Usually this does include a cloud SSH key, in which case I check it's different. In this case there is no SSH key, so I believe there is no cloud SSH key configured.

Great! Thanks @Raine

I have also confirmed this with @JMoore-WMF
The previous public key has been uploaded to Gerrit, but had not been uploaded to https://idm.wikimedia.org/keymanagement/
I advise that the same key may be used for Gerrit, GitLab, and for Cloud Services via IDM.

The new key will only be used for production SSH access.

requested wmf access, log access, and airflow access through idm.wikimedia.org

@JMoore-WMF: Why do you need be added to analytics-product-users? And analytics-search-users for that matter, actually.

Those groups are for special permissions to manage Airflow instances & DAGs and datasets, which I don't think you'll need to do in your role. They don't grant you any more access to the data and dashboards than being in analytics-privatedata-users does.

I recommend de-scoping this access request to just membership in analytics-privatedata-users

In T408164#11305545, @BTullis wrote:

The analytics-search-users group has no approver field, so I think that this is fine.

That honestly seems like an oversight and the approver should probably be @Gehel.

Thanks @mpopov - I'm re-scoping to analytics-privatedata-users as suggested.

With the new group membership defined, we have all of the approvals required, so I'm marking https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198504 as ready for review.

Once that is done, we will be able to create the kerberos principal requested in T408165.

Is there another tool for me to understand data lineage and data flows?
This is why i requested those specific airflow accesses for teams I will be
working with- the search and data engineering teams

In T408164#11307183, @JMoore-WMF wrote:

Is there another tool for me to understand data lineage and data flows?
This is why i requested those specific airflow accesses for teams I will be
working with- the search and data engineering teams

You can actually check out the Airflow instances without that access. For example, I'm not in analytics-ml-users group but I can access https://airflow-ml.wikimedia.org/home and see all the DAGs there.

Can you try going to https://airflow-search.wikimedia.org/? (See https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Instances for all the URIs for web UI.)

(I don't know how intentional that is.)

Although that doesn't get you any more info than looking at the DAGs and the sensors they use, e.g. https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/main/dags/edit/edit_hourly_dag.py

But also, in FY24/25 the Data Engineering team implemented lineage in DataHub, for example here's edit_hourly's:

https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:hive,wmf.edit_hourly,PROD)/Lineage?is_lineage_mode=true&

i can't access datahub or superset. just checked. getting "Authentication Failure
Service access denied due to missing privileges."

Right, the patch Ben uploaded hasn't been merged yet.

In T408164#11308166, @JMoore-WMF wrote:

i can't access datahub or superset. just checked. getting "Authentication Failure
Service access denied due to missing privileges."

If you can't access Superset at all, then your membership in wmf probably isn't set up yet.

https://ldap.toolforge.org/group/wmf

I am not sure if wmf shows up to apply for in IDM or not. I hear it does, but as a counter example, it does not show up in mine.

i don't see myself in this list, but i did request wmf membership here https://idm.wikimedia.org/permissions/

Got it. My guess is your wmf membership application in IDM is still pending and that Superset will start working when it is approved. You can also try logging out and back in from Superset to see if that refreshes your LDAP permissions.

I just checked- and I still cannot access:
https://airflow-ml.wikimedia.org/home
https://datahub.wikimedia.org
https://superset.wikimedia.org

Am I missing something or has the patch still not been merged? If it hasn't been merged can it be ASAP- do i need to loop someone else in for another expedited approval? Please let me know.

FYI I now can see myself here: https://ldap.toolforge.org/group/wmf

Change #1198504 merged by Brouberol:

[operations/puppet@production] Configure production shell access and posix groups for jmoore111

https://gerrit.wikimedia.org/r/1198504

access confirmed to airflow, superset, and datahub. thanks!

hi- i'm unable to access https://superset.wikimedia.org/superset/dashboard/409/?native_filters_key=0nOyfZX2obcPi68hX8Ote4F6VfWIOWVp2R2P4RTS0w_nLFBLznvdH9R_fLWq7AB1 dashboard, and need my access expanded so i can view this and other relevant dashboards for data platform issues

I suspect Justin is seeing the same error as me:

Error: {'message': 'Permission denied: user=bearloga, access=EXECUTE, inode="/wmf/data/hdfs":analytics:analytics-admins:drwxr-x---\n\tat

And just needs to be added to analytics-admins POSIX group. Created new task for that: T422963