Page MenuHomePhabricator

Requesting access to production shell groups for JAnstee
Closed, ResolvedPublicRequest

Description

Requestor provided information and prerequisites

This section is to be completed by the individual requesting access.

  • Wikitech username: JAnstee
  • Preferred shell username: janstee
  • Email address: janstee@wikimedia.org
  • Ssh public key (must be dedicated key for wmf production):
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHANABlTMFTI2RSUFDnz8yV+Ew74/zho1rXS7HHF+9/3 janstee@Jaimes-MBP
  • Requested group membership:
wikidev, analytics-privatedata-users, and researchers (and wmf or nda groups, if not already)
  • Reason for access: Data access for the Global Data & Insights team
  • Name of approving party (hiring manager for WMF staff): Sumeet Bodington
  • Requestor -- Please Acknowledge that you have read and signed the L3 Wikimedia Server Access Responsibilities document
  • Requestor -- Please coordinate obtaining a comment of approval on this task from the approving party. Note: Our new Director, Sumeet Boddington, begins Monday Oct. 26th, currently the team is being overseen by our department chief, Janeen Uzzell.

SRE Clinic Duty Confirmation Checklist for Access Requests

This checklist should be used on all access requests to ensure that all steps are covered, including expansion to existing access. Please double check the step has been completed before checking it off.

This section is to be confirmed and completed by a member of the SRE team.

  • - User has signed the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document.
  • - User has a valid NDA on file with WMF legal. (This can be checked by Operations via the NDA tracking sheet & is included in all WMF Staff/Contractor hiring.)
  • - User has provided the following: wikitech username, preferred shell username, email address, and full reasoning for access (including what commands and/or tasks they expect to perform)
  • - User has provided a public SSH key. This ssh key pair should only be used for WMF cluster access, and not share with any other service (this includes not sharing with WMCS access, no shared keys.)
  • - access request (or expansion) has sign off of WMF sponsor/manager (sponser for volunteers, manager for wmf staff)
  • - non-staff requests: 3 business day wait must pass with no objections being noted on the task
  • - Patchset for access request

For additional details regarding access request requirements, please see https://wikitech.wikimedia.org/wiki/Requesting_shell_access

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Marostegui triaged this task as Medium priority.Oct 23 2020, 5:38 AM
Marostegui updated the task description. (Show Details)
Marostegui added a subscriber: elukey.

Confirmed janstee@wikimedia.org via ldap corp as staff.
@JAnstee_WMF we'd need your manager to sign this off.
Thanks!

We are working to get our Director onboarded to phabricator and will hopefully be able to add to the card soon for approval!

jijiki changed the task status from Open to Stalled.Nov 10 2020, 4:50 PM
jijiki changed the task status from Stalled to Open.Nov 10 2020, 4:56 PM
herron added a subscriber: herron.

I'll transition this to closed for the time being due to inactivity. When ready to proceed please add a comment of manager approval and re-open the task. Thanks in advance!

JAnstee_WMF added a subscriber: Sbodington.

Reopened and added my Director, Sumeet (Sbodington), for approval

Aklapper changed the task status from Open to Stalled.EditedNov 21 2020, 7:04 AM

@Sbodington: The previous "Approved" comment here initially looked like drive-by vandalism to me.
It was made by a self-created account with zero previous on-wiki activity or Phab activity, thus it is impossible to verify who that account actually belongs to.
If the account @Sbodington belongs to WMF staff, then please follow the onboarding guide and use an official WMF account so staff membership could be verified. Thanks a lot.

To add to what @Aklapper said, when we try to verify managers on corp LDAP servers for janstee and nbdubane it tells us that is Dana McCurdy.

Searching on https://wikimediafoundation.org/role/staff-contractors/ I could find both of you in "Global Data & Insights". So it looks like @Sbodington started recently after they were already hired but is the manager now. Is that correct?

Dzahn changed the task status from Stalled to Open.Nov 23 2020, 11:35 PM
Dzahn claimed this task.

Change 643533 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admin: upgrade janstee from ldap_only to shell, add to researchers

https://gerrit.wikimedia.org/r/643533

https://wikimediafoundation.org/profile/sumeet-bodington/

shows how Dumisani (T266791) and @JAnstee_WMF are both in Sumeet's team.

moving forward based on that

Change 643533 merged by Dzahn:
[operations/puppet@production] admin: upgrade janstee from ldap_only to shell, add to privatedata

https://gerrit.wikimedia.org/r/643533

Change 643558 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admin: add janstee to analytics-privatedata-users

https://gerrit.wikimedia.org/r/643558

Change 643558 merged by Dzahn:
[operations/puppet@production] admin: add janstee to analytics-privatedata-users

https://gerrit.wikimedia.org/r/643558

Dzahn closed this task as Resolved.EditedNov 25 2020, 9:09 PM
Dzahn reassigned this task from Dzahn to JAnstee_WMF.

Hi @JAnstee_WMF

your shell account has been created. You have been upgraded from "ldap_only" to SSH/shell user.

I ran puppet (our configuration management system) on 2 hosts, bast1002 and stat1006 to confirm and it created your user.

It is "janstee" as requested. On all other relevant hosts it will be automatically created within the next 30 min.

You can now setup your access (https://wikitech.wikimedia.org/wiki/Production_access#Setting_up_your_access) and try it out.

Let us know if you run into problems.


P.S. The comment from T266791#6649893 also applies here. We added you to analytics-privatedata-users. The researchers group is deprecated and the wikidev group is default for all.

I talked to @elukey about that and possibly updating docs/template used for these tickets to reflect that. If it turns out you also need kerberos that will be added in an extra step.

[stat1006:~] $ id janstee
uid=3720(janstee) gid=500(wikidev) groups=500(wikidev),107(render),731(analytics-privatedata-users)

Excellent, @Dzahn -- thank you - I will reach out if I need further support!

SRE.

Having trouble gaining access despite approved production access.

Verbose output:
Last login: Wed Jun 2 16:14:12 on ttys000
janstee@wmf2682 ~ % ssh -N stat6 -L 8880:127.0.0.1:8880 -v
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/janstee/.ssh/config
debug1: /Users/janstee/.ssh/config line 1: Applying options for *
debug1: /Users/janstee/.ssh/config line 5: Applying options for *
debug1: /Users/janstee/.ssh/config line 37: Applying options for stat6
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Executing proxy command: exec ssh -a -W stat1006.ulsfo.wmnet:22 janstee@bast4003.wikimedia.org
debug1: identity file /Users/janstee/.ssh/id_ed25519 type 3
debug1: identity file /Users/janstee/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
Enter passphrase for key '/Users/janstee/.ssh/id_ed25519':

Apparently the stored keychain passphrase is not working.

@JAnstee_WMF do you recall what password you used when creating the ssh key? It may be different from what you have saved, have you tried others? We cannot help a lot in this case, if you don't succeed in unlocking the password we could let you create another one and change the current settings in puppet for your production access (to use the new key). Let us know :)

debug1: Executing proxy command: exec ssh -a -W stat1006.ulsfo.wmnet:22 janstee@bast4003.wikimedia.org

"stat1006.ulsfo.wmnet" in there caught my eye. There is no such host in ulsfo. It is in eqiad. You'll want that to be stat1006.eqiad.wmnet. Take a look for that in your ssh config, where does it add the ulsfo.wmnet part.

Thank you both -- @elukey I thought I did, but nothing worked. I am hoping the fix needed that @Dzahn spotted will work, I will try that and report back.

@elukey the correction to equid host did not resolve the problem and terminal continues to ask for the passphrase. Should I just delete the existing ssh keys from my folder and create new ones now, or could there be something else causing interference?

Would you mind pasting the contents of /Users/janstee/.ssh/config, Jaime?

One more thing to try is, try to SSH just to the bastion host directly, and let's forget about jumping over to stat1006 for a moment. Just to confirm if it's the key and you can connect at least to the bastion. Once we get that to work we can look at the config for making the "jump via bastion to a host behind it" again.

So that would be just a straight "ssh janstee@bast4003.wikimedia.org" with your key loaded. without any proxy/bastion config, and let's also skip the " -L 8880:127.0.0.1:8880" part for now.

I'll take a look at the logs.

440601 Jun  2 23:05:38 bast4003 sshd[6968]: Accepted key ED25519 SHA256:plaVmNDA1Ug/00RQCUV2WfIKRDNwP7GLq9NouyMKMJM found at /etc/ssh/userkeys/janstee:1
440602 Jun  2 23:05:38 bast4003 sshd[6968]: Postponed publickey for janstee from ...  port 60707 ssh2 [preauth]

Hm... so the bastion host did see the right key, that would be:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHANABlTMFTI2RSUFDnz8yV+Ew74/zho1rXS7HHF+9/3 janstee@Jaimes-MBP

@JAnstee_WMF When it asks you for " Enter passphrase for key '/Users/janstee/.ssh/id_ed25519':" I assume you also have "/Users/janstee/.ssh/id_ed25519.pub" with the public part of it? If you look at that, is it the same thing that I pasted above? Is that the key with the comment "janstee@Jaimes-MBP"?

Because I am wondering how the bastion can already see the right key if you don't get past loading the private key, so I am suspecting it might be a different one.

@Dzahn Here is my config file paste:

Host *

ForwardAgent no
IdentitiesOnly yes

Host *

AddKeysToAgent yes
UseKeychain yes
  1. From https://wikitech.wikimedia.org/wiki/Bastion
  2. Virginia: bast1003.wikimedia.org
  3. Texas: bast2002.wikimedia.org
  4. Netherlands: bast3005.wikimedia.org
  5. California: bast4003.wikimedia.org
  6. Singapore: bast5002.wikimedia.org

Host !bast4003.wikimedia.org

User janstee
ProxyCommand ssh -a -W %h:%p janstee@bast4003.wikimedia.org
IdentityFile ~/.ssh/id_ed25519

Host analytics*

User janstee
ProxyCommand ssh -a -W %h:%p janstee@bast4003.wikimedia.org
IdentityFile ~/.ssh/id_ed25519

Host *.eqiad.wmnet
User janstee
IdentityFile ~/.ssh/id_ed25519
ProxyCommand ssh -a -W %h:%p janstee@bast4003.wikimedia.org

Host stat5

HostName stat1005.eqiad.wmnet
User janstee
IdentityFile ~/.ssh/id_ed25519
ProxyCommand ssh -a -W %h:%p janstee@bast4003.wikimedia.org

Host stat6

HostName stat1006.eqiad.wmnet
User janstee
IdentityFile ~/.ssh/id_ed25519
ProxyCommand ssh -a -W %h:%p janstee@bast4003.wikimedia.org

Host stat7

HostName stat1007.eqiad.wmnet
User janstee
IdentityFile ~/.ssh/id_ed25519
ProxyCommand ssh -a -W %h:%p janstee@bast4003.wikimedia.org

When I try just ssh janstee@bast4003.wikimedia.org. it asks for passphrase

As for :

I assume you also have "/Users/janstee/.ssh/id_ed25519.pub" with the public part of it? If you look at that, is it the same thing that I pasted above?

Yes

Is that the key with the comment "janstee@Jaimes-MBP"?

Yes

Ok, thank you. Hmm.. Let's try this:

Comment out or temp. remove these lines:

> Host * 
>   AddKeysToAgent yes
>   UseKeychain yes

and then

ssh -i /Users/janstee/.ssh/id_ed25519 janstee@bast4003.wikimedia.org

This way we are excluding possible issues with the key agent / key chain locally and just tell the ssh client directly on the command line which key to use for that host.

This should ask you for the passphrase to the key and if you have that it should connect you and if you don't have it and don't have that passphrase anymore, then yes, it would be back to what you said earlier, make a fresh keypair and store that new passphrase in a secure place, like a password store.

If it turns out you need to make new keys, just paste the public part here and ask us to update the repository.

I created new keys, please use the following to update the repository:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKIp5RxtQOU35h+P/B+MgpSarZJnr73c8aIMBGEaZnT9 janstee@wmf2682

Change 698071 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] admin: replace SSH key for janstee

https://gerrit.wikimedia.org/r/698071

@JAnstee_WMF Alright, I made a patch to replace your key and uploaded it to code review.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/698071/

@colewhite Could you maybe verify this (and treat it like other access requests) as part of clinic duty?

Change 698071 merged by Cwhite:

[operations/puppet@production] admin: replace SSH key for janstee

https://gerrit.wikimedia.org/r/698071

Key has been updated. Please let us know if this action resolved the problem.

@Dzahn While I have been able to access through Jupyter, I haven't been able to get the kerberos login. When I type in kinit it just hangs - Do I need to request a new kerberos password since I had to reset my keys?

@JAnstee_WMF no you shouldn't. Where are you typing 'kinit'? Into an ssh terminal or into a Jupyter shell terminal? Also, how are you accessing Jupyter?

@Ottomata Entered kinit in terminal. Accessed Jupyter hub via localhost:8880

Entered kinit in terminal

Which terminal? In your browser in Jupyter or via ssh?

Might be hard to troubleshoot this async, wanna ping me on IRC in #wikimedia-analytics or in Slack?

Accessed Jupyter hub via localhost:8880

Which stat box?

I deduced from your question that one terminal location was right and the other one wrong and have now been able to authenticate in jupyter - thanks everyone!

Tentatively closing since this sounds like issues are resolved.

Feel free to reopen it if there is anything else missing.