Change 997952 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Revert "admin: remove ssh key of Connie Chen"

https://gerrit.wikimedia.org/r/997952

Change 997952 merged by Muehlenhoff:

[operations/puppet@production] Revert "admin: remove ssh key of Connie Chen"

https://gerrit.wikimedia.org/r/997952

Change 997953 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Revert: admin: remove conniecc1 from groups, set to absent

https://gerrit.wikimedia.org/r/997953

Change 997953 merged by Muehlenhoff:

[operations/puppet@production] Revert: admin: remove conniecc1 from groups, set to absent

https://gerrit.wikimedia.org/r/997953

• The SSH key was reinstated, the changes roll out across the next 30 minutes.
• The POSIX groups were readded, the changes roll out across the next 30 minutes.
• The cn=wmf LDAP membership was re-added
• A new Kerberos principal was created (i.e. the passwords needs to be set on first login, but functionally it's identical to the previous Kerberos principal).

@cchen: Let us know if anything is missing.

Thanks so much @MoritzMuehlenhoff!!

@MoritzMuehlenhoff thank you for helping me restoring my access!

I am trying to log into Superset and Hue, but l cannot access them. I also reset the developer account's password a couple of times but still saw "Your password has expired".

Tagging DPE SRE in case this is specific to those tools.

@cchen: Can you please verify if you can ssh to the stat hosts and also use JupyterHub? And query with hive, etc.?

I ssh the stats machine and kinit, and got `Password incorrect while getting initial credentials. and I also tried JupyterHub, and it also showed "Invalid username or password".

@cchen When you ran kinit the first time after you logged in, did it ask you to change the password? Did you get a new temporary one by mail?

I see this: "The first time that kinit is executed it will ask to change the temporary password that you should have received via email " and Moritz said "passwords needs to be set on first login", so that's why I ask.

@Dzahn Oh, I see. I found the email and reran the kinit with the temporary password, it works now.

@cchen: How about Superset & Hue?

I still not able to access Superset & Hue, and i tried to reset my password again, still not working.

You look still to be blocked on wikitech https://wikitech.wikimedia.org/wiki/Special:Contributions/Conniecc1 - not sure if that's related but it should probably be undone

I've added a checklist based on the private task.

@MoritzMuehlenhoff (or another SRE): please update based on what already works

@cchen: if there's anything you think you should have that isn't covered by that (a lot of your access was granted by the 'WMF' ldap group) then please add it

@cchen I unblocked your wikitech account. I checked all services above which should work.

Can you try again accessing superset? (or resetting your password). thanks!

@Jelto I just reset wikitech account. Superset, hue and Jupyterhub access all work now. thank you!