Brennen got that right, there are 2 separate SSH daemons on each gitlab server. One is the standard one we have on every single server, which we use to manage servers. These listen on _server_ names (the ones including numbers, gitlab1001 and gitlab2001, identify specific hardware) and are generally for shell admin access.

Then there is the additional SSH daemon that is for gitlab users, this one listens on a _service_ name, gitlab.wikimedia.org and gitlab-replica.wikimedia.org. Which of the 2 data centers is currently the main one and which is the replica could change in the future. Currently though gitlab1001 holds the gitlab.wikimedia.org name.

Both servers have 2 public IPs, this is how both SSHDs can listen on port 22 but on different DNS names.

So what I did first just now is create the missing wikitech server pages that are created from a template on Wikitech.

I created these:

https://wikitech.wikimedia.org/wiki/Gitlab1001

https://wikitech.wikimedia.org/wiki/Gitlab2001

from that template you automatically get links to pages for the fingerprints. which i pasted:

https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab1001.wikimedia.org
https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab2001.wikimedia.org

then I protected the pages so that only wiki admins can change them.

So shell admins can look at that and compare what they are getting.. when they connect to specific server hardware, 1001 or 2001 currently.

All that being said, I don't feel it fully answers the question because what you get when you connect to the service IP, gitlab.wikimedia.org and what you see in the web UI under gitlab.wikimedia.org should arguably be the same.

What I can see is that the server fingerprints I pasted match what is shown in the web UI. I am tempted to call this a minor bug or improvement request for gitlab itself. In a setup like ours where we have multiple IPs on the machine it should let admins specifiy which one is the one that it should use to display fingerprints to the users.

I am tempted to call this a minor bug or improvement request for gitlab itself. In a setup like ours where we have multiple IPs on the machine it should let admins specifiy which one is the one that it should use to display fingerprints to the users.

Agreed. A quick search didn't find anything in existing settings, but I might have missed something, or it might be relatively straightforward to implement and get a patch merged upstream. I can do a bit of digging.

Thanks for the docs!

Cool! Yea, so our case is we have 2 IP addresses on the same network interface (or actually 4 if you count v4 and v6). But they are bound to the same NIC.

gitlab (understandably) picks the first one it sees but we want it to use the second one.

In T296944#7561721, @brennen wrote:

I am tempted to call this a minor bug or improvement request for gitlab itself. In a setup like ours where we have multiple IPs on the machine it should let admins specifiy which one is the one that it should use to display fingerprints to the users.

Agreed. A quick search didn't find anything in existing settings, but I might have missed something, or it might be relatively straightforward to implement and get a patch merged upstream. I can do a bit of digging.

Thanks for the docs!

I also couldn't find a setting where SSH host keys are stored. It seems GitLab looks just in /etc/ssh/ (see https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/instance_configuration.rb#L7)

So we could try to add a configuration parameter for that and get it merged upstream, swap the folder of our SSH daemons or publish the SSH fingerprints somewhere else. I would favor option 1 or 3

In T296944#7583225, @Jelto wrote:

I also couldn't find a setting where SSH host keys are stored. It seems GitLab looks just in /etc/ssh/ (see https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/instance_configuration.rb#L7)
So we could try to add a configuration parameter for that and get it merged upstream, swap the folder of our SSH daemons or publish the SSH fingerprints somewhere else. I would favor option 1 or 3

I would also vote for 3 (short-term) and 1 mid-term) but not worry about 2.

I added a section to https://wikitech.wikimedia.org/wiki/GitLab to explain the fingerprint issue, linked to the new server pages I created, which link to their fingerprint pages. And linked back to this ticket.

I made this new page that shows all fingerprints in a central place:

https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab.wikimedia.org

That page is protected so only wiki admins can edit.

Linked to it from the existing gitlab and gitlab server pages and added comments.

Adding a screenshot in this ticket for completeness.

The part we haven't talked about yet is that also for the service IP there is one key per data center. I am not sure if that is just fine or we want to use the same key on both servers and copy it around.

f it has different sets of keys for the same hostnames that's going to trigger errors for users when it gets switched over, right? Using the same fingerprints for the *service* regardless of host seems necessary.

Yea, well.. unless you argue "if we switch over to another data center we will schedule it and tell users anyways and pointing out it's now a new host key is just a formality and all keys are already listed in public". But yea, I tend to agree, we probably want to sync that over. (unless you say it should change when in reality it also is a new backend behind it and that's more correct to tell the end user about the change instead of trying to hide it)

In T296944#7586043, @Dzahn wrote:

The part we haven't talked about yet is that also for the service IP there is one key per data center. I am not sure if that is just fine or we want to use the same key on both servers and copy it around.

I checked the fingerprint of GitLab SSH host keys on gitlab1004 (production) and gitlab2002 (replica) and the are identical. During the migration we copied also the hostkeys to prevent any errors when doing failovers.

So we should update https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab.wikimedia.org and only mention the keys once (the eqiad ones are correct). We should also remove gitlab1001 and gitlab2001 there as they are decommissioned. It seems I don't have permissions to edit the SSH key section and first have to find out how to get permissions :)

I'd say if we updated the wikitech page we can close this issue, as there is nothing else we can do on our side. We could create a upstream issue for that.

In T296944#8486844, @Jelto wrote:

So we should update https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab.wikimedia.org and only mention the keys once (the eqiad ones are correct). We should also remove gitlab1001 and gitlab2001 there as they are decommissioned.

I made this edit to reflect your comments:

https://wikitech.wikimedia.org/w/index.php?title=Help%3ASSH_Fingerprints%2Fgitlab.wikimedia.org&diff=2042454&oldid=2036387

It seems I don't have permissions to edit the SSH key section and first have to find out how to get permissions :)

Fixed by making you an "interface administrator".

20:42, 3 January 2023 Dzahn talk contribs block changed group membership for Jelto from (none) to interface administrator (Jelto needs to be able to edit SSH fingerprint pages, he is a server admin)

https://wikitech.wikimedia.org/w/index.php?title=Special:Log&logid=958501

I'd say if we updated the wikitech page we can close this issue, as there is nothing else we can do on our side. We could create a upstream issue for that.

Confirmed fingerprint I get connecting to gitlab.wikimedia.org matches.

Will close this as suggsted.