Page MenuHomePhabricator

500 Internal Server Error when trying to access ssh keys on toolsadmin
Open, LowPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

When I visit https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ I am getting 500 error. This does not happen for others in the Wikimania hackathon.

What happens?:

image.png (485×1 px, 75 KB)

Request ID = b48093a2e328422b80acee0e496c7ba0

What should have happened instead?:

Software version (on Special:Version page; skip for WMF-hosted wikis like Wikipedia):

Other information (browser name/version, screenshots, etc.):

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
taavi subscribed.
Traceback (most recent call last):
  File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapper_view
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/app/striker/profile/views.py", line 85, in ssh_keys
    pkey = utils.parse_ssh_key(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/srv/app/striker/profile/utils.py", line 46, in parse_ssh_key
    key = SSHPublicKey(pubkey, strict_mode=True, skip_option_parsing=True)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/sshpubkeys/keys.py", line 135, in __init__
    self.parse(keydata)
  File "/opt/lib/poetry/striker-2uZo5AhP-py3.11/lib/python3.11/site-packages/sshpubkeys/keys.py", line 470, in parse
    if key_type is not None and key_type != unpacked_key_type.decode():
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte

Would deleting the stored ssh key on the Toolforge side solve this?

Does https://idm.wikimedia.org/keymanagement/ recognize and/or allow you to manage the existing key on your account?

I think that there's a typo in the key that was uploaded, the one that's in ldap starts like:

ssh-rsa AAAAB4NzaC1yc2EAAAADAQABAAABAQCaBSmjYcMzZ9...

When it should be:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCaBSmjYcMzZ9...

Note the B3 instead of B4. If you can delete and update on idm (the url @taavi shared) would be the best yep, as those are stored outside the toolforge system.

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpzVhg5BZCq4y+Y+N0VLq5xNoiv37E04WFoC8eyvXdAsCBx7p7h6tpUMFNxu/guWBXo7hTNLo2hp51PVT39R/MAxPgGCmdynhgolxvXGLD/UDWYy9rsqQHmqF76KTapPJHbmShwniyEp42k08ZVn/7Ro7XpBAF6uQqgoW7soeZ16ZZfdVwLYPWSjepMvEpWHkdcmejDpc+P2gL43czG8YEDAhyiCq/adMA+5PxmuD5bDz6A1Ubb9hs/wLk0OeFvz9mgkmnH1jc34fRElUOeuvoEmE3PaFZmJC5DeHGeoZreYCm3OkizSJE/kW3E6it4E8eTCVNHZqM/oQuIkxxXsZ0Af51jyPWxhAi9KNvuSsdbUH65TDsr8MuHVNyZeKeG/iIwe3JxHDdQ9M6sDfJ5oiKN3is4cxVfht/b17PZraW2p5njDa6Yucig+bN1GAmq1KeLp+qcJPueVF96Q1n16vd7W51WYOwxmJCC1TPo7LtrJNQyc3JtFrllHCRe3+w9E0= thetis@thetis

I'm not confident if there's a typo, but this is the SSH key for my current laptop.

When I try to use https://idm.wikimedia.org/keymanagement/create/ I got this error

SSH key already exists
Please consult our documentation if you need help generating your SSH keys, available at: https://wikitech.wikimedia.org/wiki/Generate_an_SSH_Key
Valid SSH key types are ssh-rsa, ssh-ecdsa, ssh-ed25519, sk-ssh-ed25519@openssh.com, sk-ssh-ecdsa@openssh.com, ecdsa-sha2-nistp256

I'm not sure if someone else added my key, or I added it somehow. But my correct key shows up on https://idm.wikimedia.org/keymanagement/ now. I don't think showed there before.

https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ still 500s out though.

I am still not able to ssh onto toolforge though

$ ssh soni@login.toolforge.org
soni@login.toolforge.org: Permission denied (publickey).
$ ssh soni@dev.toolforge.org
soni@dev.toolforge.org: Permission denied (publickey).

Ldap is still giving the old one that starts with AAAAB4NzaC1yc2EAAAADAQABAAABAQCaBSmjYcMzZ9... (https://ldap.toolforge.org/user/soni), maybe @MoritzMuehlenhoff or @SLyngshede-WMF can help with the idm/ldap part?

I'm not sure if someone else added my key, or I added it somehow. But my correct key shows up on https://idm.wikimedia.org/keymanagement/ now. I don't think showed there before.

The new key shows only on IDM, https://ldap.toolforge.org/user/soni still lists my old key.

On IDM, I am not able to suspend/deactivate my keys. I hit 500 error even if I try to deactivate my new key there - https://idm.wikimedia.org/keymanagement/deactivate/592085/

500 Internal Server Error - Your request caused an error on the server.

Mentioned in SAL (#wikimedia-cloud) [2025-08-07T09:52:10Z] <taavi> remove ssh key from uid=soni LDAP user T401318

I removed the problematic key from LDAP, pasted here for backup/troubleshooting purposes:

sshPublicKey: ssh-rsa AAAAB4NzaC1yc2EAAAADAQABAAABAQCaBSmjYcMzZ9kEO84BLGYiWpElxujf9Pgvs7cD0ImBHd2akgX4ejYoiyf5uNEYGnEZ8aJEbmNrMmBNqhD9Z553qmBvFSpxEqqZM/nIh7523pfpD9yDd2QhihL84+H4oByQ7CESo3JkbaAx1jsPXvx68eTVXJkM9iY8AlHjBMpdeC4aKIWdwSyWhRl3s1+rWMGu7+bwDlliJJhT2dXUbBQIZv0kOEV9ogEzP6LrYBbi4nUA8vaRBPQpOKp5SaEMOJqUZQxscp+PkmOLyQ6UhJpRQ0ThupvEv/CxT1pg2Z6vLgqdrfXkgod18qu34ClOkOiH8Iw5LzD5NXe5ilNObxC/ the.original.soni@gmail.com

You should now be able to add a new key via toolsadmin.

It works now!

Added my key through https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/ and now I can ssh. Thank you!

I suspect there's some illegally formatted character or some problem with UTF-8 in the old key? Not sure.

@Soni I suspect that you are correct, ssh-keygen even claims that it's not a valid public key.

That makes sense.

I think it probably should not just 500 out, but I don't know enough about that error handling.

It really shouldn't, I'll look into fixing that :-)