I believe that the approvals given in the linked ticket T325004: Requesting access to analytics-privatedata-users & analytics-product-users for Hxi-ctr are sufficient to permit me to create the Kerberos principal, so I'll go ahead and do it now.

btullis@krb1001:~$ sudo manage_principals.py create hxi-ctr --email_address=hxi-ctr@wikimedia.org
Principal successfully created. Make sure to update data.yaml in Puppet.
Successfully sent email to hxi-ctr@wikimedia.org

@HXi-WMF - could you please check your email now and follow the instructions in the email regarding kerberos? Many thanks.

@BTullis: I pinged Hua about this and apparently Google thought it was spam. Just a heads-up for future Kerberos principal requests.

@HXi-WMF: Can't remember if the email includes a link to https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos/UserGuide#Authenticate_via_Kerberos but adding it here just in case.

Hi! I put my shell username incorrectly in the original ticket — it is actually xihua

Would that affect my ability to login to Jupyter because I haven't been able to?

Would that affect my ability to login to Jupyter because I haven't been able to?

Yep. The original ticket has been re-opened and the username will need to be updated before you're able to log in.

Hi @HXi-WMF - I'm sorry to hear that you haven't been able to use Jupyter yet, let me see if I can help you.

You mentioned this:

I put my shell username incorrectly in the original ticket — it is actually xihua

I don't think that's quite right, as you can see from the following output. The shell account hxi-ctr exists, but xihua doesn't.

btullis@stat1007:~$ id hxi-ctr
uid=42819(hxi-ctr) gid=500(wikidev) groups=500(wikidev),107(render),731(analytics-privatedata-users),820(analytics-product-users)

btullis@stat1007:~$ id xihua
id: ‘xihua’: no such user

So the next questions is, are you able to get a shell session on any stat host?

What happens when you just type the following into a terminal, for example?

ssh stat1007.eqiad.wmnet

I get this result

The authenticity of host 'stat1007.eqiad.wmnet (<no hostip for proxy command>)' can't be established.
ED25519 key fingerprint is SHA256:8rvZNxbCHfNE8dxB8+CIjUZHsblCeOgBu9c2JylKs54.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'stat1007.eqiad.wmnet' (ED25519) to the list of known hosts.
Linux stat1007 4.19.0-20-amd64 #1 SMP Debian 4.19.235-1 (2022-03-17) x86_64
Debian GNU/Linux 10 (buster)

  _  __         _               _             _   _               _   
 | |/ /        | |             (_)           | | | |             | |  
 | ' / ___ _ __| |__   ___ _ __ _ _______  __| | | |__   ___  ___| |_ 
 |  < / _ \ '__| '_ \ / _ \ '__| |_  / _ \/ _` | | '_ \ / _ \/ __| __|
 | . \  __/ |  | |_) |  __/ |  | |/ /  __/ (_| | | | | | (_) \__ \ |_ 
 |_|\_\___|_|  |_.__/ \___|_|  |_/___\___|\__,_| |_| |_|\___/|___/\__|


This host is capable of Kerberos authentication in the WIKIMEDIA realm.

For more info: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos/UserGuide

Netbox Status: active
stat1007 is a Statistics & Analytics cluster explorer (private data access, no local compute) (statistics::explorer)
Bare Metal host on site eqiad and rack B3
The last Puppet run was at Thu Jan 19 17:21:47 UTC 2023 (1 minutes ago). 
Last Puppet commit: (71e50455d0) Eric Evans - admin: add jebe to ldap_only_users (wmf)
Debian GNU/Linux 10 auto-installed on Tue Oct 6 09:21:50 UTC 2020.

You do not have a valid Kerberos ticket in the credential cache, remember to kinit.

When I run

ssh -N stat1005.eqiad.wmnet -L 8880:127.0.0.1:8880

I'm able to connect but when I go to the Jupyter login, it says my username and Password are not correct even when I enter the same one I use for Wikitech.

Thanks for that explanation @HXi-WMF - yes it looks like we might have to rename your shell account. Apologies for the delay in getting you this acces.
I'm seeking guidance on the best course of action, so please bear with us. We should have updated instructions for you soon.

I've deleted this principal with the following command:

btullis@krb1001:~$ sudo manage_principals.py delete hxi-ctr
Principal successfully deleted. Since the principal seems to be related to a user, make sure to update the krb flag in Puppet's data.yaml.

I will wait until the account has been renamed as per T325004#8548633 and https://gerrit.wikimedia.org/r/881872, at which point I will recreate the kerberos principal using the correct name.

Hi! I can now connect using xihua as my shell username but I am still not able to sign into Juypter notebook when I go to localhost:8880

To clarify, I should be using my shell username xihua and my LDAP password which is the same password I use for Wikitech?

@BTullis

Hi! Sorry just wanted to check in on this again. I am able to ssh, but when I try to sign into Jupyter notebook with my shell username (xihua) and LDAP password (Wikitech password), it tells me username and password not valid. Could there possibly still be some issue? @Aklapper

when I try to sign into Jupyter notebook with my shell username (xihua) and LDAP password (Wikitech password), it tells me username and password not valid

@EChetty: Hua is providing the correct info but still can't login, which is a major blocker. Can someone from your team please help her with this? I think Ben's been busy putting out the fires related to the Superset upgrade.

Hello @HXi-WMF

I'm sorry that you're having such trouble with this part of the process. I will do my best to help you to get this sorted as soon as possible.

Firstly, I have reissued your Kerberos principal with the following command, using the correct shell account name.

btullis@krb1001:~$ sudo manage_principals.py get xihua
get_principal: Principal does not exist while retrieving "xihua@WIKIMEDIA".
btullis@krb1001:~$ sudo manage_principals.py create xihua --email_address=hxi-ctr@wikimedia.org
Principal successfully created. Make sure to update data.yaml in Puppet.
Successfully sent email to hxi-ctr@wikimedia.org

Please check your spam folder to find your temporary credentials and the instructions to change your Kerberos password it. Unfortunately, there is known issue with these emails being marked as spam, for which I apologise.

You'll need this Kerberos password regardless of the authentication problem that you mentioned, namely:

...when I try to sign into Jupyter notebook with my shell username (xihua) and LDAP password (Wikitech password), it tells me username and password not valid

I have found the log entries for your most recent login attempt today.

Feb 01 16:25:04 stat1005 jupyterhub-conda[20605]: [D 2023-02-01 16:25:04.417 JupyterHub ldapauthenticator:379] Attempting to bind xihua with uid=xihua,ou=people,dc=wikimedia,dc=org
Feb 01 16:25:04 stat1005 jupyterhub-conda[20605]: [D 2023-02-01 16:25:04.536 JupyterHub ldapauthenticator:392] Status of user bind xihua with uid=xihua,ou=people,dc=wikimedia,dc=org : True
Feb 01 16:25:04 stat1005 jupyterhub-conda[20605]: [D 2023-02-01 16:25:04.536 JupyterHub ldapauthenticator:431] username:xihua Using dn uid=xihua,ou=people,dc=wikimedia,dc=org
Feb 01 16:25:04 stat1005 jupyterhub-conda[20605]: [W 2023-02-01 16:25:04.542 JupyterHub ldapauthenticator:454] username:xihua User not in any of the allowed groups

Looking at the jupyterhub configuration in /etc/jupyterhub-conda/jupyterhub_config.py on each stat host, we can see that the allowed groups are as follows:

'allowed_ldap_groups': ["cn=nda,ou=groups,dc=wikimedia,dc=org", "cn=wmf,ou=groups,dc=wikimedia,dc=org"],
'allowed_posix_groups': ["analytics-privatedata-users", "analytics-admins", "gpu-users", "analytics-search-users", "analytics-wmde-users", "analytics-product-users", "deploy-ml-service"],

So the next step is checking for LDAP group membership and we can see that you have not been added to any LDAP groups.

btullis@seaborgium:~$ ldapsearch -x member=uid=xihua,ou=people,dc=wikimedia,dc=org dn
# extended LDIF
#
# LDAPv3
# base <dc=wikimedia,dc=org> (default) with scope subtree
# filter: member=uid=xihua,ou=people,dc=wikimedia,dc=org
# requesting: dn 
#

# search result
search: 2
result: 0 Success

# numResponses: 1

We can also confirm this by browsing here: https://ldap.toolforge.org/

That's reasonable because there is no request for LDAP group membership here: T325004: Requesting access to analytics-privatedata-users & analytics-product-users for Hxi-ctr

...but it's listed as a requirement here: https://wikitech.wikimedia.org/wiki/Analytics/Data_access#ssh_login_to_analytics_client_servers_(AKA_stat_boxes)_with_Hadoop,_Hive,_Presto_access

It's also reasonable that this wasn't assumed, because as this page says: https://wikitech.wikimedia.org/wiki/SRE/Clinic_Duty/Access_requests#WMF_Group

WMF staff and contractors with a wikimedia.org email address can be added to the "wmf" group on request (not everyone needs this access).

I'll go ahead and add you to the wmf LDAP group now, following the guidelines here.

Here is the LDAP request ticket: T328607: Grant Access to cn=wmf for xihua

I have now added you to the wmf group in LDAP, so please would you try again to access JupyterHub @HXi-WMF.
Here is confimation of that by means of an LDAP search.

btullis@seaborgium:~$ ldapsearch -x member=uid=xihua,ou=people,dc=wikimedia,dc=org dn
# extended LDIF
#
# LDAPv3
# base <dc=wikimedia,dc=org> (default) with scope subtree
# filter: member=uid=xihua,ou=people,dc=wikimedia,dc=org
# requesting: dn 
#

# wmf, groups, wikimedia.org
dn: cn=wmf,ou=groups,dc=wikimedia,dc=org

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

Once again, apologies for the delay in getting this sorted and I hope that this gets you past your blocker to getting data access and crafting visualizations.

Thank you so much @BTullis for looking into it and performing the necessary fixes. I also appreciate the detailed notes from your investigation.

Thank you so much! I'm able to login now. @BTullis Really appreciate it!