Page MenuHomePhabricator

Requesting access to restricted production access and analytics-privatedata-users for Karen Brown
Closed, ResolvedPublicRequest

Description

Username: kbrown
Full name: Karen Brown
SSH Key: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDfO6BVcbHhtufdzA8/H8Jjmg3oqfx10UCHSdb2ZoYmFettMxwXzzcWxzOoPThk7ihKAZWUrweIbV/KZuKtkcGRB3bSN5NyHckbwnzbf18TBqoAMjnDZO2shctuMn0gMuVVyk76W2KXzsn+kP0mry+Nt/UJjb1ZTjignEi8CzREpQV1JNw8L8U3Jhd0U18VN99+R2Bo1LeSnWWIMFxeAEJt6EI3tBTUSjQ3rCkAS6piC7+usB94n3sSXZcc9y3YjIVhWhVtiZlT7lqOEisjZ0w/3u0Nu3FFJRF0VoYeOpOO/fO4ir1P4JyGcMC3SEk/Cu1jgeZI7NHmNzYEPQBv9huswNYgvqLY35/JG3B3dP/fqMcn64sObXN8MCpH03iUsLgJe/rg8Knv5FodAK5ASMSakbuNdb1TC2LHYrbswX4vxrYLdOON0L6LoeV5a58lbkt1pB5u/0A8RC3kaEMBw6e5UVe9a1YYZphpHW5P2bqZIjIIORuAk5n+XgnOyc5MG9YZHEA91Aqit26Dnrm3PDxACtqMcsu2gLoTcjB0c/iP3u4HsUnFlzSFkNjFw94YWeBaLJrScAPIPJog3uH88eeLjhB1KwCobkZeRw/DelfiH9oOv21FPnuFHbkXB0JxTxUOwPEeUDpx4u74rsauEtJKmg+BLqFUsxwPV2726g9LIw==

I'd like to request access for @Kbrown to what I believe will be the restricted group and analytics-privatedata-users (the same that I have). Trust and Safety has had a number of workflows requiring shell access and private analytics logs (hadoop). Many of our workflows (including these ones) have been increasing and the only two people on our team with access are myself and Joe Sutherland. This has caused some major bottleneck issues at times and we want to expand the available people within our team to include other members of the T&S Operations team which includes Karen.

Specifically some of the workflows she needs to be able to do (and I believe needs this access for):

  • Run maintenance scripts (mwmaint servers) to:
    • To remove 2FA for users who have lost their backup codes (after identity verification)
    • To add or reset user email addresses when locked out of their account (again after identity verification)
    • To permanently remove illegal images from the servers
  • Lookup private information such as user email addresses for legal or T&S investigations (such as urgent threats of harm or court orders).
  • Query webserver logs for private information such as IPs which have viewed certain pages (usually court orders)

Karen has already signed L3. @JanWMF is our people manager and I'll have him comment here in support. As always please let me know if any issues or questions.

SRE Clinic Duty Checklist for Access Requests

Most requirements are outlined on https://wikitech.wikimedia.org/wiki/Requesting_shell_access

This checklist should be used on all access requests to ensure that all steps are covered. This includes expansion to access. Please do not check off items on the list below unless you are in Ops and have confirmed the step.

  • - User has signed the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document.
  • - User has a valid NDA on file with WMF legal. (This can be checked by Operations via the NDA tracking sheet & is included in all WMF Staff/Contractor hiring.)
  • - User has provided the following: wikitech username, preferred shell username, email address, and full reasoning for access (including what commands and/or tasks they expect to perform.
  • - User has provided a public SSH key. This ssh key pair should only be used for WMF cluster access, and not share with any other service (this includes not sharing with WMCS access, no shared keys.)
  • - access request (or expansion) has sign off of WMF sponsor/manager (sponser for volunteers, manager for wmf staff)
  • - non-sudo requests: 3 business day wait must pass with no objections being noted on the task
  • - sudo requests: all sudo requests require explicit approval during the weekly operations team meeting. No sudo requests will be approved outside of those meetings without the direct override of the Director of Operations.
  • - Patchset for access request

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptAug 10 2018, 8:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
herron triaged this task as Normal priority.Aug 10 2018, 5:51 PM

Looping in @Nuria for review/approval of analytics-privatedata-users membership request

Dzahn claimed this task.Aug 13 2018, 5:42 PM

Note that "kbrown" is a username already taken in LDAP and it's KEVIN Brown, not Karen.

@Jalexander @Kbrown Could you please make a Wikitech/LDAP user (on https://wikitech.wikimedia.org) and let us know which one you picked?

Dzahn reassigned this task from Dzahn to Kbrown.Aug 13 2018, 11:18 PM
Dzahn added a subscriber: Dzahn.

Change 452583 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: add shell user for Karen Brown

https://gerrit.wikimedia.org/r/452583

Dzahn added a comment.EditedAug 13 2018, 11:30 PM

@JanWMF Do you approve of this request?

Sorry, i saw your approval above. You can ignore my question.

Thank you, Daniel, I appreciate you checking and working on this :)

Note that "kbrown" is a username already taken in LDAP and it's KEVIN Brown, not Karen.
@Jalexander @Kbrown Could you please make a Wikitech/LDAP user (on https://wikitech.wikimedia.org) and let us know which one you picked?

Thanks, she now has Kbrown (WMF) as wiki name with karen as shell (and I believe ldap)

This is a bit strange, i can't find a user "karen" nor any user with email kbrown_at_wikimedia.org in LDAP but i can see how you created it in wiki.

Looks like the account was created by a logged-in user instead of anonymously, no idea if that even works with the LDAP integration. What does Special:Preferences say for 'Instance shell account name'?

@Jalexander I was able to go directly to the wikitech wiki database and look in the user table and i see both of the users you created.. with their email addresses. but they are not in LDAP.. also searching by email. it's almost as if your method to create the user for somebody else only created a local user. will have to keep debugging this..

@Krenair this would confirm my suspicion. thank you. it looks like that might not work with LDAP integration.

on DB level i can see these differences:

For my own user the fields "user_real_name" and "user_password" have some values in them. (Dzahn and a string with "sha512" in it).

MariaDB [labswiki]> select user_real_name,user_password from user where user_email="dzahn@wikimedia.org";
+----------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| user_real_name | user_password                                                                                                                              |
+----------------+--------------------------------------------------------------------------------------------------------------------------------------------+
| Dzahn          |  ..:sha512... |
+----------------+--------------------------------------------------------------------------------------------------------------------------------------------+

For Karen's user these fields are empty:

MariaDB [labswiki]> select user_name,user_real_name,user_password from user where user_email="kbrown@wikimedia.org";
+--------------+----------------+---------------+
| user_name    | user_real_name | user_password |
+--------------+----------------+---------------+
| Kbrown (WMF) |                |               |
+--------------+----------------+---------------+

Did she log into wikitech and set a real password instead of the temporary one? That would populate user_password.

@Jalexander Can we just let her create a normal user (as anon) and not worry about the "(WMF)" part on wikitech? There aren't many users with (WMF) in here and i think that should solve the issue.

Ok, so let's first have the 2 users (also see T201667) confirm they set their intial password. Maybe that already changes things.

Dzahn changed the task status from Open to Stalled.Aug 14 2018, 11:41 PM

Did she log into wikitech and set a real password instead of the temporary one? That would populate user_password.

Yes-ish. I logged in and tried to set a real password, but got an error message (the exact content of which I have unfortunately forgotten, but I think it was something about the "authorization plugin"?), and now can't log in with either original temporary password OR the new password I tried to use.

Did she log into wikitech and set a real password instead of the temporary one? That would populate user_password.

Yes-ish. I logged in and tried to set a real password, but got an error message (the exact content of which I have unfortunately forgotten, but I think it was something about the "authorization plugin"?), and now can't log in with either original temporary password OR the new password I tried to use.

My memory from what you told me earlier was "the authentication plugin rejected your password change" or something like that?

@Jalexander Can we just let her create a normal user (as anon) and not worry about the "(WMF)" part on wikitech? There aren't many users with (WMF) in here and i think that should solve the issue.

:( I'd really rather not...I agree it would probably solve the issue but I don't want to basically write off any WMF accounts ever again on that site. Having the WMF is the best practice (and the preference from Legal) for all of our public wikis, Wikitech got an exception originally because of existing users and a desire not to have that fight at the time by legal but it still isn't a good thing especially since there is a lot more non-staff editing there now with toollabs/cloud services merged in. We'd also have to do it for more members of my team who would generally like to have the same username everywhere if they can since it's a lot easier to remember.

My guess is that it would also solve the issue if I created the account just with a password and have them change it (rather then click "send a new password by email" which is the normal way to do it for someone else and what I did in this case). My guess is that's where the bug happened.

(Granted I obviously can't do that now for them when the account is already created... though I guess I could try to rename it away)

RobH added a subscriber: RobH.Aug 20 2018, 8:33 PM

Please note that I'm now on clinic duty this week, so I need to confirm a few things. This task is currently blocked by @Kbrown logging into their wikitech account. Once they do so, we'll be able to check if the ldap user was created and can move from there.

Please note that I'm now on clinic duty this week, so I need to confirm a few things. This task is currently blocked by @Kbrown logging into their wikitech account. Once they do so, we'll be able to check if the ldap user was created and can move from there.

I think this is fixed now (and I'm able to find the her ldap user (karen) via ldapsearch) .

RobH added a comment.EditedAug 21 2018, 7:02 PM

Ok, there is a problem with this request in the way the prodution public ssh key was provided. It seems @Jalexander pasted it in the original request, that he made on behalf of @karen. Unfortunately, this is not an acceptable way for us to get @Kbrown's public ssh key (from a third party, @Jalexander.)

As such, we'll need @Kbrown to please comment on this task, pasting in your public SSH key. Please ensure this key is not used for anything but production ssh access (do not use the same key for cloud services or anywhere else.)

This is not meant as a personal attack on @Jalexander (whom I know, have worked with, and trust) but simply just enforcing what is acceptable security practice. Otherwise this can lead to opening production access up to social engineering attacks and the like.

RobH updated the task description. (Show Details)Aug 21 2018, 7:10 PM

Hi there. Hopefully I'm doing this right - please let me know if I've borked it:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDfO6BVcbHhtufdzA8/H8Jjmg3oqfx10UCHSdb2ZoYmFettMxwXzzcWxzOoPThk7ihKAZWUrweIbV/KZuKtkcGRB3bSN5NyHckbwnzbf18TBqoAMjnDZO2shctuMn0gMuVVyk76W2KXzsn+kP0mry+Nt/UJjb1ZTjignEi8CzREpQV1JNw8L8U3Jhd0U18VN99+R2Bo1LeSnWWIMFxeAEJt6EI3tBTUSjQ3rCkAS6piC7+usB94n3sSXZcc9y3YjIVhWhVtiZlT7lqOEisjZ0w/3u0Nu3FFJRF0VoYeOpOO/fO4ir1P4JyGcMC3SEk/Cu1jgeZI7NHmNzYEPQBv9huswNYgvqLY35/JG3B3dP/fqMcn64sObXN8MCpH03iUsLgJe/rg8Knv5FodAK5ASMSakbuNdb1TC2LHYrbswX4vxrYLdOON0L6LoeV5a58lbkt1pB5u/0A8RC3kaEMBw6e5UVe9a1YYZphpHW5P2bqZIjIIORuAk5n+XgnOyc5MG9YZHEA91Aqit26Dnrm3PDxACtqMcsu2gLoTcjB0c/iP3u4HsUnFlzSFkNjFw94YWeBaLJrScAPIPJog3uH88eeLjhB1KwCobkZeRw/DelfiH9oOv21FPnuFHbkXB0JxTxUOwPEeUDpx4u74rsauEtJKmg+BLqFUsxwPV2726g9LIw== kbrown@DESKTOP-VEAKNNV

Jalexander removed Kbrown as the assignee of this task.Aug 23 2018, 10:34 PM
Dzahn updated the task description. (Show Details)Aug 23 2018, 11:33 PM

@Kbrown That looks good, it matches what we had before.

Since i have the patchset to create the user from last week's clinic duty i amended my change at : https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/452583/

Since i now had the UID to match the LDAP user which i still needed.

I can confirm the provided SSH keys match and the existing groups James are "restricted" and "analytics-privatedata-user".

I went through the checkboxes on the ticket and all requirements are fulfilled except maybe the one about sudo because the restricted group does grant some sudo privileges.

Since creating a user and adding it to groups has to be done as 2 steps anyways i can go ahead with that before adding the groups.

Change 452583 merged by Dzahn:
[operations/puppet@production] admins: add shell user for Karen Brown

https://gerrit.wikimedia.org/r/452583

The new user has been created on bastion hosts but doesn't have additional groups yet. Those will have to be added in an additional changeset and might require approval in meeting on Monday due to the sudo privileges involved in them.

RobH added a comment.Aug 27 2018, 5:59 PM

This was approved in today's SRE team meeting.

Dzahn changed the task status from Stalled to Open.Aug 27 2018, 6:20 PM

Change 455616 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: add karen to restricted, analytics-privatedata-users

https://gerrit.wikimedia.org/r/455616

Change 455616 merged by Dzahn:
[operations/puppet@production] admins: add karen to restricted, analytics-privatedata-users

https://gerrit.wikimedia.org/r/455616

Dzahn closed this task as Resolved.Aug 27 2018, 6:42 PM
Dzahn claimed this task.

@Jalexander @Kbrown You have the same admin groups now. (Or at least in max 30 minutes after puppet ran everywhere). Workflows should work the same for both of you. Let us know if any issues.

Dzahn updated the task description. (Show Details)Aug 27 2018, 6:43 PM