Page MenuHomePhabricator

Requesting access to labweb1001 and labweb1002 for jhernandez
Closed, ResolvedPublicRequest

Description

Requestor provided information and prerequisites

This section is to be completed by the individual requesting access.

  • Wikitech username: Jhernandez
  • Preferred shell username: jhernandez
  • Email address: jhernandez@wikimedia.org
  • Ssh public key (must be dedicated key for wmf production): ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFay7brSap8ALMkuqdnOQ+nQJefZUCGzNS5dhHzHeNzq joakin@DESKTOP-THEJR4U
  • Requested group membership: deployment
  • Reason for access: I need access to labweb1001/labweb1002 to run the Cloud Services Annual Survey/2020 right now, and for future work with WMCS
  • Name of approving party (hiring manager for WMF staff): @Bmueller
  • Requestor -- Please Acknowledge that you have read and signed the L3 Wikimedia Server Access Responsibilities document: Yes
  • Requestor -- Please coordinate obtaining a comment of approval on this task from the approving party.

SRE Clinic Duty Confirmation Checklist for Access Requests

This checklist should be used on all access requests to ensure that all steps are covered, including expansion to existing access. Please double check the step has been completed before checking it off.

This section is to be confirmed and completed by a member of the SRE team.

  • - User has signed the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document.
  • - User has a valid NDA on file with WMF legal. (This can be checked by Operations via the NDA tracking sheet & is included in all WMF Staff/Contractor hiring.)
  • - User has provided the following: wikitech username, preferred shell username, email address, and full reasoning for access (including what commands and/or tasks they expect to perform)
  • - User has provided a public SSH key. This ssh key pair should only be used for WMF cluster access, and not share with any other service (this includes not sharing with WMCS access, no shared keys.)
  • - access request (or expansion) has sign off of WMF sponsor/manager (sponser for volunteers, manager for wmf staff)
  • - non-staff requests: 3 business day wait must pass with no objections being noted on the task
  • - Patchset for access request

For additional details regarding access request requirements, please see https://wikitech.wikimedia.org/wiki/Requesting_shell_access

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 656375 had a related patch set uploaded (by Ayounsi; owner: Ayounsi):
[operations/puppet@production] Add jhernandez to deployment

https://gerrit.wikimedia.org/r/656375

Change 656375 merged by Ayounsi:
[operations/puppet@production] Add jhernandez to deployment

https://gerrit.wikimedia.org/r/656375

You're all set, give it 30min for Puppet to run. Let me know if any issues.

ayounsi claimed this task.

Thank you @ayounsi!

It works, although I have an issue with my config. If I connect with ssh -v -i .ssh/id_wiki_prod -l jhernandez bast3004.wikimedia.org specifying the identity in the cli args I can connect correctly. If I do ssh bast3004.wikimedia.org it doesn't seem to respect my IdentityFile config. Do you know what could be happening? Here is my config and logs from trying to connect:

.ssh/config
# Turn this on for Match to work.
CanonicalizeHostname yes

# Defaults for all WMF hosts.
Match host=*.wikimedia.org,*.wmnet
    ForwardAgent no
    IdentitiesOnly yes
    KbdInteractiveAuthentication no
    PasswordAuthentication no
    User jhernandez

# Configure the initial connection to the bastion host, with the one
# HostName closest to you
Host bast
    HostName bast3004.wikimedia.org
    IdentityFile ~/.ssh/id_wiki_prod

# Proxy all connections to internal servers through the bastion host
Host *.wmnet *.wikimedia.org !gerrit.wikimedia.org !bast*.wikimedia.org
    ProxyJump bast
    IdentityFile ~/.ssh/id_wiki_prod

Host gerrit.wikimedia.org
    Port 29418
    IdentityFile ~/.ssh/id_rsa_wiki
ls -l .ssh/
total 40K
drwx------  2 joakin joakin 4.0K Jan 15 19:30 .
drwxr-xr-x 11 joakin joakin 4.0K Jan 15 19:30 ..
-rw-------  1 joakin joakin 1.2K Jan 15 19:11 config
-rw-------  1 joakin joakin 3.2K Nov 27  2018 id_rsa
-rw-r--r--  1 joakin joakin  746 Nov 27  2018 id_rsa.pub
-rw-------  1 joakin joakin 1.7K Dec  2  2014 id_rsa_wiki
-rw-r--r--  1 joakin joakin  402 Dec  2  2014 id_rsa_wiki.pub
-rw-------  1 joakin joakin  419 Jan 12 20:56 id_wiki_prod
-rw-r--r--  1 joakin joakin  104 Jan 12 20:56 id_wiki_prod.pub
-rw-r--r--  1 joakin joakin 2.0K Jan 15 19:16 known_hosts
ssh -vvv bast3004.wikimedia.org
~ → ssh -vvv bast3004.wikimedia.org
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /home/joakin/.ssh/config
debug2: checking match for 'host=*.wikimedia.org,*.wmnet' host bast3004.wikimedia.org originally bast3004.wikimedia.org
debug3: /home/joakin/.ssh/config line 5: matched 'host "bast3004.wikimedia.org"'
debug2: match found
debug1: /home/joakin/.ssh/config line 19: Skipping Host block because of negated match for bast*.wikimedia.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_addr: could not resolve name bast3004.wikimedia.org as address: Name or service not known
debug3: resolve_canonicalize: not canonicalizing hostname "bast3004.wikimedia.org" (max dots 1)
debug1: hostname canonicalisation enabled, will re-parse configuration
debug1: re-parsing configuration
debug1: Reading configuration data /home/joakin/.ssh/config
debug2: checking match for 'host=*.wikimedia.org,*.wmnet' host bast3004.wikimedia.org originally bast3004.wikimedia.org
debug3: /home/joakin/.ssh/config line 5: matched 'host "bast3004.wikimedia.org"'
debug2: match found
debug1: /home/joakin/.ssh/config line 19: Skipping Host block because of negated match for bast*.wikimedia.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "bast3004.wikimedia.org" port 22
debug2: ssh_connect_direct
debug1: Connecting to bast3004.wikimedia.org [91.198.174.60] port 22.
debug1: Connection established.
debug1: identity file /home/joakin/.ssh/id_rsa type 0
debug1: identity file /home/joakin/.ssh/id_rsa-cert type -1
debug1: identity file /home/joakin/.ssh/id_dsa type -1
debug1: identity file /home/joakin/.ssh/id_dsa-cert type -1
debug1: identity file /home/joakin/.ssh/id_ecdsa type -1
debug1: identity file /home/joakin/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/joakin/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/joakin/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/joakin/.ssh/id_ed25519 type -1
debug1: identity file /home/joakin/.ssh/id_ed25519-cert type -1
debug1: identity file /home/joakin/.ssh/id_ed25519_sk type -1
debug1: identity file /home/joakin/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/joakin/.ssh/id_xmss type -1
debug1: identity file /home/joakin/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u7 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to bast3004.wikimedia.org:22 as 'jhernandez'
debug3: hostkeys_foreach: reading file "/home/joakin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/joakin/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from bast3004.wikimedia.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:vp48DKQHJ0x/cjPZ7MsXe+ChoMZdkVLS0m0xwg832dw
debug3: hostkeys_foreach: reading file "/home/joakin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/joakin/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from bast3004.wikimedia.org
debug3: hostkeys_foreach: reading file "/home/joakin/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/joakin/.ssh/known_hosts:5
debug3: load_hostkeys: loaded 1 keys from 91.198.174.60
debug1: Host 'bast3004.wikimedia.org' is known and matches the ECDSA host key.
debug1: Found key in /home/joakin/.ssh/known_hosts:4
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/joakin/.ssh/id_rsa RSA SHA256:5cUuZbx97m6QUPGCPrEm17JlDVRM5KQ4hsNJnfvKo8o
debug1: Will attempt key: /home/joakin/.ssh/id_dsa
debug1: Will attempt key: /home/joakin/.ssh/id_ecdsa
debug1: Will attempt key: /home/joakin/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/joakin/.ssh/id_ed25519
debug1: Will attempt key: /home/joakin/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/joakin/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: start over, passed a different list publickey,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/joakin/.ssh/id_rsa RSA SHA256:5cUuZbx97m6QUPGCPrEm17JlDVRM5KQ4hsNJnfvKo8o
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /home/joakin/.ssh/id_dsa
debug3: no such identity: /home/joakin/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/joakin/.ssh/id_ecdsa
debug3: no such identity: /home/joakin/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/joakin/.ssh/id_ecdsa_sk
debug3: no such identity: /home/joakin/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/joakin/.ssh/id_ed25519
debug3: no such identity: /home/joakin/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/joakin/.ssh/id_ed25519_sk
debug3: no such identity: /home/joakin/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/joakin/.ssh/id_xmss
debug3: no such identity: /home/joakin/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred:
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 60
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1
Password:

👆 for some reason it is trying the default file names and not trying my specified IdentityFile... Any pointers would be appreciated, I've been looking for answers for a couple of hours now without success 🤦‍♂️

@Jhernandez you're excluding bast* for the config block that specifies the correct key to use:

Host *.wmnet *.wikimedia.org !gerrit.wikimedia.org !bast*.wikimedia.org
    ProxyJump bast
    IdentityFile ~/.ssh/id_wiki_prod

I'd guess that is because you don't want to jump to a bastion via a bastion

@Majavah the config is basically straight from https://wikitech.wikimedia.org/wiki/Production_access#Setting_up_your_SSH_config

That block AFAICT is to avoid proxy jumping if you are connecting to a bastion or gerrit. For the bastion, there is the host block that specifies the key too:

Host bast
    HostName bast3004.wikimedia.org
    IdentityFile ~/.ssh/id_wiki_prod

@Jhernandez yes, but that block does not apply to the host bast3004.wikimedia.org, it only applies to bast

@Majavah right on the money! I thought Host bast would be matching bast on the input host but apparently not (no asterisks I guess).

I've added a explicit section for the bastions and works like a charm now:

# Configure connection to the bastion hosts
Host bast*.wikimedia.org
    IdentityFile ~/.ssh/id_wiki_prod

I'll update the wikitech docs about this 👍 Thanks for running through this with me! 🙌