Page MenuHomePhabricator

Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints
Open, LowPublic

Description

Under Instance Configuration, Wikimedia GitLab currently reports the following SSH host key fingerprints:

AlgorithmMD5SHA256
ECDSA8a:29:0a:54:bb:b0:06:21:f6:1a:7e:f7:b0:d6:7d:dcSHA256:i+fWYVTGTn8XXsbrYDOVUHCZZCk6rRyCG4rJk5nxxh8
ED2551970:d1:0d:b6:1d:bb:9d:d3:a1:12:10:f0:c3:3c:7f:c8SHA256:lKmdy0Xvi4AXeEWEbg3sM5Zc8GRSqqbhnP7uti8xKBs
RSAf4:48:50:da:5e:cd:eb:5e:0f:a6:1b:77:c6:9b:34:cbSHA256:u3BILUTd6NsM9/ItWulh3uzqcKh3SXzpkC+oNKmbG4k

But when I connect to GitLab, SSH reports different host key fingerprints:

The authenticity of host 'gitlab.wikimedia.org (2620:0:861:1:208:80:154:14)' can't be established.
ECDSA key fingerprint is SHA256:V1Q7vVIt/XR2yuue3kUa27/+cSwrCm8wLDfBROh8HMs.

ED25519 key fingerprint is SHA256:eGFVaGXuc0AkXE4n+SrBbp1Mkul4RAvZPWbVUrZ7sM0.

RSA key fingerprint is SHA256:hwHUSYaoPvrQGmphy+sCLbfGqiTcSdJ41FmRKIF+wK0.


gitlab-wikimedia-ssh-fingerprints.png (633×809 px, 151 KB)

Event Timeline

Lucas_Werkmeister_WMDE renamed this task from Self-reported GitLab SSH host keys don’t appear to match actual host keys to Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints.Dec 2 2021, 3:49 PM
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
Lucas_Werkmeister_WMDE updated the task description. (Show Details)
brennen added subscribers: Dzahn, brennen.

We're using two separate sshd instances on gitlab1001 - one for administrative access via gitlab1001.wikimedia.org and one for access to GitLab repos themselves at gitlab.wikimedia.org. I imagine GitLab is getting host key info either from /etc/ssh or the default sshd itself. I don't think this is likely to really be a problem, apart from misreporting the keys in the UI, but probably a good thing to be aware of / document.

@Dzahn might be able to comment further.

Brennen got that right, there are 2 separate SSH daemons on each gitlab server. One is the standard one we have on every single server, which we use to manage servers. These listen on _server_ names (the ones including numbers, gitlab1001 and gitlab2001, identify specific hardware) and are generally for shell admin access.

Then there is the additional SSH daemon that is for gitlab users, this one listens on a _service_ name, gitlab.wikimedia.org and gitlab-replica.wikimedia.org. Which of the 2 data centers is currently the main one and which is the replica could change in the future. Currently though gitlab1001 holds the gitlab.wikimedia.org name.

Both servers have 2 public IPs, this is how both SSHDs can listen on port 22 but on different DNS names.

So what I did first just now is create the missing wikitech server pages that are created from a template on Wikitech.

I created these:

https://wikitech.wikimedia.org/wiki/Gitlab1001

https://wikitech.wikimedia.org/wiki/Gitlab2001

from that template you automatically get links to pages for the fingerprints. which i pasted:

https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab1001.wikimedia.org
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab2001.wikimedia.org

then I protected the pages so that only wiki admins can change them.

So shell admins can look at that and compare what they are getting.. when they connect to specific server hardware, 1001 or 2001 currently.

All that being said, I don't feel it fully answers the question because what you get when you connect to the service IP, gitlab.wikimedia.org and what you see in the web UI under gitlab.wikimedia.org should arguably be the same.

What I can see is that the server fingerprints I pasted match what is shown in the web UI. I am tempted to call this a minor bug or improvement request for gitlab itself. In a setup like ours where we have multiple IPs on the machine it should let admins specifiy which one is the one that it should use to display fingerprints to the users.

I am tempted to call this a minor bug or improvement request for gitlab itself. In a setup like ours where we have multiple IPs on the machine it should let admins specifiy which one is the one that it should use to display fingerprints to the users.

Agreed. A quick search didn't find anything in existing settings, but I might have missed something, or it might be relatively straightforward to implement and get a patch merged upstream. I can do a bit of digging.

Thanks for the docs!

Cool! Yea, so our case is we have 2 IP addresses on the same network interface (or actually 4 if you count v4 and v6). But they are bound to the same NIC.

gitlab (understandably) picks the first one it sees but we want it to use the second one.

I am tempted to call this a minor bug or improvement request for gitlab itself. In a setup like ours where we have multiple IPs on the machine it should let admins specifiy which one is the one that it should use to display fingerprints to the users.

Agreed. A quick search didn't find anything in existing settings, but I might have missed something, or it might be relatively straightforward to implement and get a patch merged upstream. I can do a bit of digging.

Thanks for the docs!

I also couldn't find a setting where SSH host keys are stored. It seems GitLab looks just in /etc/ssh/ (see https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/instance_configuration.rb#L7)

So we could try to add a configuration parameter for that and get it merged upstream, swap the folder of our SSH daemons or publish the SSH fingerprints somewhere else. I would favor option 1 or 3

I also couldn't find a setting where SSH host keys are stored. It seems GitLab looks just in /etc/ssh/ (see https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/instance_configuration.rb#L7)
So we could try to add a configuration parameter for that and get it merged upstream, swap the folder of our SSH daemons or publish the SSH fingerprints somewhere else. I would favor option 1 or 3

I would also vote for 3 (short-term) and 1 mid-term) but not worry about 2.

I added a section to https://wikitech.wikimedia.org/wiki/GitLab to explain the fingerprint issue, linked to the new server pages I created, which link to their fingerprint pages. And linked back to this ticket.

I made this new page that shows all fingerprints in a central place:

https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab.wikimedia.org

That page is protected so only wiki admins can edit.

Linked to it from the existing gitlab and gitlab server pages and added comments.

Adding a screenshot in this ticket for completeness.

The part we haven't talked about yet is that also for the service IP there is one key per data center. I am not sure if that is just fine or we want to use the same key on both servers and copy it around.

f it has different sets of keys for the same hostnames that's going to trigger errors for users when it gets switched over, right? Using the same fingerprints for the *service* regardless of host seems necessary.

Yea, well.. unless you argue "if we switch over to another data center we will schedule it and tell users anyways and pointing out it's now a new host key is just a formality and all keys are already listed in public". But yea, I tend to agree, we probably want to sync that over. (unless you say it should change when in reality it also is a new backend behind it and that's more correct to tell the end user about the change instead of trying to hide it)