Page MenuHomePhabricator

ppenloglou sharing wmcs and production ssh key
Closed, ResolvedPublic

Description

We had an alert today that @ppenloglou is using the same ssh key for production and wmcs. As result we took away the production rights as this is a security risk. (see https://wikitech.wikimedia.org/wiki/SRE/Production_access#Generating_your_SSH_key: "Reminder: the key you use for production access must be different from the key you use for Cloud VPS, so do not paste it into the Openstack field under Special:Preferences on this wiki")

Details

Event Timeline

@ppenloglou please let us know if you need help submitting a new SSH key for the production environment. Otherwise we will close this task

Vgutierrez triaged this task as Medium priority.Sep 4 2023, 10:06 AM
Vgutierrez moved this task from Untriaged to Awaiting User Input on the SRE-Access-Requests board.

Dear @Ladsgroup,

Thanks for letting me know about the misuse of my ssh key. Could you guide in the right direction for the following?
Currently, I would like to be a able to:

Did this occur because the SSH key for GitLab and people.wikimedia is the same? I could keep the same one for GitLab as is now and create a new one for people.wikimedia.

Thanks in advance for your suggestions and guidance!

Kind regards,
Panos

@ppenloglou that's right. as stated in https://wikitech.wikimedia.org/wiki/People.wikimedia.org people.wm.o is part of the production environment and the SSH key can't be shared with other environments.

Please create a new one for the production environment.

Thank you @Vgutierrez for your reply, now it makes sense.

I've created a new SSH key locally saved as "id_ed25519_wmprod.pub" so I can tell them apart.

And it is:
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILPROgSF5rtGxEB6vZzaDZXYj0zJE88NbG1jEwe+Sf6t ppenloglou@wikimedia.org

Should I add this to my User Page alongside the current one, or replace the old one?

Thanks!
Panos

the key needs to be uploaded to the puppet repo, you could use this CR as an example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/949839 or I could craft a new one for you.

Could you kindly give me a hand with this @Vgutierrez whenever you have a spare moment?

Change 954879 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] admin: Add new SSH key for ppenloglou

https://gerrit.wikimedia.org/r/954879

Vgutierrez changed the task status from Open to Stalled.Sep 5 2023, 7:07 AM

Change 954879 merged by Vgutierrez:

[operations/puppet@production] admin: Add new SSH key for ppenloglou

https://gerrit.wikimedia.org/r/954879

Vgutierrez claimed this task.

your new key should be deployed in the next ~30 minutes. Please do not upload it to gitlab/wikitech to prevent this from happening again.

Thanks!

As per the debugging instructions here, I'm unfortunately stuck and can't seem to connect via SSH.

My "instance shell account name" in Wikitech preferences is

image.png (447×34 px, 5 KB)

which should become my production shell username if I'm not mistaken.

This is my current 'config' file contents

# Turn CanonicalizeHostname on for Match to work below.
CanonicalizeHostname yes

# Defaults for all Wikimedia Foundation hosts.
Match host=*.wikimedia.org,*.wmnet
    ForwardAgent no
    IdentitiesOnly yes
    KbdInteractiveAuthentication no
    PasswordAuthentication no
    User ppenloglou

# Configure the initial connection to the bastion host, with the one
# HostName closest to you.
Host bast
    HostName bast6002.wikimedia.org
    IdentityFile ~/.ssh/id_ed25519_wmprod
    # In theory this User line shouldn't be necessary due to the Match above,
    # but in practice it seems to be.  In any case, it doesn't hurt.
    User ppenloglou

# Proxy all connections to internal servers through the bastion host.
Host *.wmnet *.wikimedia.org !gerrit.wikimedia.org !bast*.wikimedia.org !gitlab.wikimedia.org
    ProxyJump bast
    IdentityFile ~/.ssh/id_ed25519_wmprod

# Configure direct connection to the bastion hosts.
Host bast*.wikimedia.org
    IdentityFile ~/.ssh/id_ed25519_wmprod

Host gerrit.wikimedia.org
    Port 29418
    IdentityFile ~/.ssh/cloud.key

I've used bast6002 since I'm located in Greece and it's the closest physically.
Inside my '.ssh' directory I have these:

id_ed25519
id_ed25519.pub
id_ed25519_wmprod
id_ed25519_wmprod.pub

I've reserved "id_ed25519" for my Gitlab usage and in this request I've shared the public key of "id_ed25519_wmprod.pub"

To connect, I run the following command:

ssh people1004.eqiad.wmnet

If I run with the verbose option, I get the following output:

ppenloglou@wmf3105 .ssh % ssh -v people1004.eqiad.wmnet
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/ppenloglou/.ssh/config
debug1: /Users/ppenloglou/.ssh/config line 1: Applying options for people1004.eqiad.wmnet
debug1: /Users/ppenloglou/.ssh/config line 25: Applying options for *.wmnet
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: hostname canonicalisation enabled, will re-parse configuration
debug1: re-parsing configuration
debug1: Reading configuration data /Users/ppenloglou/.ssh/config
debug1: /Users/ppenloglou/.ssh/config line 1: Applying options for people1004.eqiad.wmnet
debug1: /Users/ppenloglou/.ssh/config line 25: Applying options for *.wmnet
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -v -W '[%h]:%p' bast
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Executing proxy command: exec ssh -v -W '[people1004.eqiad.wmnet]:22' bast
debug1: identity file /Users/ppenloglou/.ssh/id_ed25519_wmprod type 3
debug1: identity file /Users/ppenloglou/.ssh/id_ed25519_wmprod-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/ppenloglou/.ssh/config
debug1: /Users/ppenloglou/.ssh/config line 17: Applying options for bast
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to bast6002.wikimedia.org port 22.
ssh: Could not resolve hostname bast6002.wikimedia.org: nodename nor servname provided, or not known
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Could you guide me in the right direction?
Thanks a ton for all of your help so far everyone!

Kind regards,
Panos

Oh my god, it worked immediately. Thank you so much!!