Thanks so much @ArielGlenn ! @toddleroux , @RyanSteinberg @Afandian please follow the steps above!

Note that the three checklists are for whatever sres are working on these tasks; as we see different users doing the steps we will double-check them and then mark them off. No need for you to do so :-)

Are collaborators going to be able to log in to officewiki?
bastiononly hasn't existed for years.

I don't seem to have access to Office Wiki and I don't see an option to create an account. Should I share my public SSH key here or wait for Office Wiki access?

It's fine to share public keys here right on the ticket since they are public and will be added to public repos either way.

public key:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBN1rS7OObcft7lDa9+H45kLfkdGHwlJ6rL2Fm2IPsMB

preferred shell login:
ryanmax

public key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsJPyatQmAgubnM6ChTohZdEYTOfVJjzpsOtiVrBcwTOVBwEl3qcORlMEF0MMk+BdMfiMd12jmfxGWuOhzJAZ8iPDE9Bk9z0/ttb6NJj09CfO5YlfOsgi8gAXthZ7gAX7EulUVW1jKHksbG3tS7zIxmx5+GzHWJ6SGapKsUrf2LC0RwQkiNyha2535vZlvmbMRhaICeN9F/2GCSoEbgdvf2+4Ln3hO/me5EcKvxDfXCcCYklHxNS9ikfK84RpXd8EfIv2tmf1LYWw38DUVv6io3ehuar7c/twBTmodiKqkkywLHNY0u6KFAnKEsBuQ989CiepxIJjb+NYd+KuZvMBh jwass@herring.local

preferred shell login:
jwass or afandian

SRE CLINIC DUTY NOTE: I've gone ahead and checked & all 3 users HAVE THE NDA ON FILE. So I'll check off that step on the checklist. The expiry date on the NDA (and thus shell access) is 2019-09-15 with Dario Taraborelli (@DarTar) as the notification of expiry contact in the admin module.

We still need each user to do the following:

• sign the l3 document (right now only @RyanSteinberg has done so)
• provide a public ssh key that is ONLY USED FOR WMF SHELL ACCESS (and not cloud services either). We still need this from @toddleroux.
• ensure user has a wikitech login, as a wikitech login is required for shell access (its how we generate the shell user id)

Once we have all of that above, we can prepare a couple of patches to push all three users live.

wikitech info for @RyanSteinberg

Username: RyanSteinberg
Instance shell account name: ryanmax

• We are still waiting on almost all details from Todd.
• L3 is signed by both Ryan and Joe.
• Need clarification on wikitech acount by Joe (he should confirm he owns the second one he proposes as shell login or create a new one).
• Waiting for detailed full reasoning/scope of access requested, as that is the first thing asked for Director of Operations to provide access. If not done on time for Monday Ops meeting, it will be at least delayed by another week.

Sorry for the confusion, I'm struggling to understand how many different types of accounts I need and how they link up. The Phabricator account I am logging in with is the only one I have access to.

I have now created a 'wikitech' account with shell account 'afandian2'.

Thank you for helping with this!

I will reach out to Todd re outstanding items

If possible, can we move forward at least on Ryan Steinberg's access? Miriam, Ryan and I will be meeting up in person this week at WikiCite to discuss this project.

Regarding purpose we are working on: https://meta.wikimedia.org/wiki/Research:Investigating_Wikipedia%27s_role_as_a_gateway_to_medical_content
The purpose of this project is to understand how links in medical pages are used.

In terms of full scope of access, I'm going to defer to @DarTar and @Miriam.

Thank you!

Sorry for the confusion, @Afandian. You need to register an account on https://wikitech.wikimedia.org and you have to tell me the name of it. This account is unrelated to this access, but it is a prerequisite to avoid conflicts. Could you tell me the name of the account on wikitech you have or registered, 'afandian2' is not a registered Username at the moment.

Your Phabricator account tells me you have 'Afandian' as a Wikitech/LDAP account, but that was created in 2015. There is also the recently created 'Joe Wass' account. Please register or log in to one and confirm you have control of it (no matter the name). Linking it to your Phabricator is recommended, but not required.

If possible, can we move forward at least on Ryan Steinberg's access?

I will schedule it for approval, pending your meeting and ok of @DarTar and @Miriam as sponsors.

@jcrespo The account name is "Joe Wass".

Thanks, @Afandian - this is unrelated to this request, but let me suggest to link it to your Phabricator account as the LDAP linked account at https://phabricator.wikimedia.org/settings/user/Afandian/page/external/ for your convenience (the other account you have linked can be confusing).

In T209298#4773520, @jcrespo wrote:

If possible, can we move forward at least on Ryan Steinberg's access?

I will schedule it for approval, pending your meeting and ok of @DarTar and @Miriam as sponsors.

Yes please, all of the above access request are approved on my end and by Legal, have MOUs and NDAs filed, and are similar to other formal collaborations we do routinely in Research. If we can fast-track Ryan while the other two accounts are pending, that'd be fantastic. If there are specific questions or concerns on the scope of this collaboration, please contact me. Thanks.

Regarding "a reference for the signed MOU/NDA", the process we've followed so far is to use the tracker maintained by @RStallman-legalteam called "NDA and MOU: Volunteer accounts with Server and LDAP-level access", which is the authoritative source of information.

yes, no problem- I brought these up for Ops approval and some people pointed out that your signoff was enough (in addition to the rest of the process).

I can prepare the patches and deploy them on Thursday 29 Nov (3 day wait) for both afandian and RyanSteinberg.

Change 476039 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] admin: Add Ryan Steinberg and Joe Wass access to production cluster

https://gerrit.wikimedia.org/r/476039

Change 476039 merged by Jcrespo:
[operations/puppet@production] admin: Add Ryan Steinberg and Joe Wass access to production cluster

https://gerrit.wikimedia.org/r/476039

Change 476487 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] Revert "Revert "admin: Add Ryan Steinberg and Joe Wass access to production cluster""

https://gerrit.wikimedia.org/r/476487

Change 476487 merged by Jcrespo:
[operations/puppet@production] Revert "Revert "admin: Add Ryan Steinberg and Joe Wass access to production cluster""

https://gerrit.wikimedia.org/r/476487

It appears I supplied a key in the wrong format. I believe this is preventing me from signing in. Would you please replace with for my afandian2 account:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHt287L9YmVILW5tSEruECUqJ+Ad0Ja+Q3Dl8Pnncbxc jwass@herring.local

Apologies for this.

In T209298#4799902, @Afandian wrote:

It appears I supplied a key in the wrong format. I believe this is preventing me from signing in. Would you please replace with for my afandian2 account:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHt287L9YmVILW5tSEruECUqJ+Ad0Ja+Q3Dl8Pnncbxc jwass@herring.local

Apologies for this.

@Afandian Before making any changes, could you please let us know what makes you believe that they key format is wrong? What kind of error are you getting?
It would be great if you could ping the clinic duty person (topic on #wikimedia-operations on IRC), to discuss this.

@toddleroux please provide the missing information, we would really like to have this task resolved:)

@jijiki - sorry for the delay on my end (no excuses).

1. L3 is now signed.
2. my wikitech username is toddleroux
3. my preferred shell username is toddleroux
4. my email address is todd.c.leroux@gmail.com
5. my public SSH key is as follows

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDX2JxzqK7TeBZ2unDfbfyg90oHxAhipobW+Fzg0xwg1BV7xo3VYJXuKRhPhuP1pp6Ju/Z1zfuCCnZ9QhGzU27Y6zigM1Ze7fCAGU2jUdaNJJ1cK5V8N/Zvi6ywOSz3s/jHAHYyDD9EkfVDny53xeI0tq4xS9fWZePyZy8Ga1XxEKx7kngileN89ExYZ7aw/tTvKjfn9KM+frXXI3RfGO7usmlEmBgrC0DQcrP6i/bHvfP8HDjd3Kj29VS2N8FdlZJQq3zO5q1zr9r2rXxf7Ohs+G+qqyGJ3TTxX2ISBuN6UNHpaBusUIhMY+/b1FwT682mkEx0ma9wY5pcod66oCstUEliT9dpekAFxcboh8gNJ4GK4QMLWsfk68HfZJCXEIACTJ2x+ZviMQUWgXQq0K7Gcp0Ypzm+kBjz3cPEIdP0bUK+7T2jDRAiZAk+rfQ7h4ryHwuKY9Ir39xtZHRgs8UNXAh4hF6rEvXtHwlhNkdDy93od4f63Fuf4UjCtNwZFLjvnUTZIajllnzWzjwOizY4Z7Bj/byh7hSPORhuGnqXgW4vz0JWDKMBH2ad5t6jQaE3WkFu01n19SQDickaZKBfUE7dNgQrKfFeVfYiu3rnmk9E8nkGrQOH/WcfzaDGgF6GIdy6SXy1QpbH39wEHJruVX3RQimSjCo+p5hkSdT8NQ== todd.c.leroux@gmail.com

Change 478717 had a related patch set uploaded (by Mathew.onipe; owner: Mathew.onipe):
[operations/puppet@production] admins: add user toddleroux

https://gerrit.wikimedia.org/r/478717

Change 478717 merged by Dzahn:
[operations/puppet@production] admins: add user toddleroux

https://gerrit.wikimedia.org/r/478717

Change 478802 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: add toddleroux to analytics-privatedata-users

https://gerrit.wikimedia.org/r/478802

Change 478802 merged by Dzahn:
[operations/puppet@production] admins: add toddleroux to analytics-privatedata-users

https://gerrit.wikimedia.org/r/478802

@toddleroux Your user has been created now. For example i ran puppet and i see it on the host stat1007.eqiad.wmnet. Within 30 minutes puppet will create it on all other hosts with the analytics-privatedata-user role. Your access is now the same as for afandian2 and ryanmax, so you can copy config from them if needed. I think this resolves the ticket. Let us know if any remaining issues.

@Afandian Is your access working meanwhile? If not, please add details which host you are trying to connect to and what error you are getting. Also a timeframe when you tried would be great, then we can check logs.

@Dzahn thank you so much!

@jijiki Sorry this is getting drawn out!

I am unable to log in using the instructions at https://wikitech.wikimedia.org/wiki/Production_shell_access#Setting_up_your_SSH_config

My config file says:

Host bast1002.wikimedia.org
    # Direct connection for the bastion host
    ProxyCommand none
    ControlMaster auto

Host *.wikimedia.org *.wmnet !gerrit.wikimedia.org !git-ssh.wikimedia.org
    User afandian2
    # Everything else goes via bastion acting as a proxy
    ProxyCommand ssh -a -W %h:%p bast1002.wikimedia.org
    # Do not offer other identities loaded in ssh-agent
    IdentitiesOnly yes
    IdentityFile ~/.ssh/wiki

When I try to connect I am prompted for a password:

herring:~ jwass$ ssh -vvv  afandian2@stat1007.eqiad.wmnet
OpenSSH_7.8p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/jwass/.ssh/config
debug1: /Users/jwass/.ssh/config line 19: Applying options for *.wmnet
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: /etc/ssh/ssh_config line 52: Applying options for *
debug1: Executing proxy command: exec ssh -a -W stat1007.eqiad.wmnet:22 bast1002.wikimedia.org
debug1: identity file /Users/jwass/.ssh/wiki type 0
debug1: identity file /Users/jwass/.ssh/wiki-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
Password:

The same happens for bast1002.wikimedia.org

This leads me to believe that there is a problem with the acceptance of the key. Because I created the key in a different way to those given in the instructions, I thought that the key format was causing this problem.

I don't use IRC, but can join the channel if it's necessary to solve this problem.

Thanks for your help!

@Afandian Hi, i can help you with this. First of all let me confim i see in the logs on bast1002 some failed logins from you, and they all say "failed public key". On stat1007 i see no attempts in the log. So this is a key issue at the bastion server already, yes. To simplify let's first just focus on a direct login on the bastion host and make sure that works, then we can get back to stat1007.

In T209298#4821089, @Afandian wrote:

Because I created the key in a different way to those given in the instructions, I thought that the key format was causing this problem.

Given the symptoms we see this seems indeed to be the case. How did you create it? Want to upload a new one? I see "RSA SHA256" btw

@Afandian Could you make a new one with either "ssh-keygen -t ed25519" or "ssh-keygen -t rsa -b 4096 -o" per https://wikitech.wikimedia.org/wiki/Production_shell_access#Technical_details ?

This was created, in error, using the standard arguments for ssh-keygen, i.e. RSA 2048, SHA256. My mistake.
Per https://phabricator.wikimedia.org/T209298#4799902 , my new public key is:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHt287L9YmVILW5tSEruECUqJ+Ad0Ja+Q3Dl8Pnncbxc jwass@herring.local

Thanks!

Change 479527 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] admins: replace/fix ssh key for afandian (rsa2048 -> ed25519)

https://gerrit.wikimedia.org/r/479527

Change 479527 merged by Dzahn:
[operations/puppet@production] admins: replace/fix ssh key for afandian (rsa2048 -> ed25519)

https://gerrit.wikimedia.org/r/479527

@Afandian I replaced the key and also noticed in https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/479527/1/modules/admin/data/data.yaml that there was an additional typo on our side before, see there was no "ssh-rsa" at the beginning of the key string. That was an additional problem, so it wasn't just you. I merged and ran puppet on both bast1002 and stat1007, i saw puppet replace the key. Please try again now.

Thanks for all your help @Dzahn @jijiki @RyanSteinberg . I can now log in, all working as expected.

(Yes, those previous failed attempts were me trying various configurations.)

@toddleroux, @Afandian, @RyanSteinberg if everything if alright, we can mark this as resolved @Dzahn

I'm having trouble logging in using my public key.

toddleroux@toddleroux-UX310UA:~/.ssh$ ssh -vvv stat1007.eqiad.wmnet
OpenSSH_7.2p2 Ubuntu-4ubuntu2.6, OpenSSL 1.0.2g 1 Mar 2016
debug1: Reading configuration data /home/toddleroux/.ssh/config
debug1: /home/toddleroux/.ssh/config line 6: Applying options for *.wmnet
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Executing proxy command: exec ssh -a -W stat1007.eqiad.wmnet:22 bast1002.wikimedia.org
debug1: permanently_drop_suid: 1000
debug1: identity file /home/toddleroux/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/toddleroux/.ssh/id_rsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
Password:

I believe I am suffering from the same typo that impacted Joe @Afandian. I believe there is a typo in my public key entry:
https://gerrit.wikimedia.org/g/operations/puppet/+/refs/changes/27/479527/1/modules/admin/data/data.yaml#3144

If someone could assist it would be greatly appreciated. Thanks.

• Todd

Change 492932 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] fixing user toddleroux ssh key entry

https://gerrit.wikimedia.org/r/492932

Change 492932 merged by RobH:
[operations/puppet@production] fixing user toddleroux ssh key entry

https://gerrit.wikimedia.org/r/492932