Host key of gitlab.wikimedia.org does not appear in auto configured wmf known hosts. It was added to https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/gitlab.wikimedia.org in T296944. This is a good first step but requires manual lookup by users.
To make the user experience easier and allow other machines to clone over ssh we should add gitlab.wikimedia.org to wmf known hosts, similar to gerrit.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | eoghan | T333840 Move gitlab ssh host keys to private puppet | |||
| Resolved | Jelto | T337107 gitlab.wikimedia.org ssh host key should appear in wmf-known-host |
Change 921506 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] gitlab: use sshkey for git-ssh public keys
Change 923341 had a related patch set uploaded (by Jbond; author: jbond):
[operations/puppet@production] puppetmaster: frontend
Change 923341 abandoned by Jbond:
[operations/puppet@production] puppetmaster: frontend
Reason:
dont think this is needed
Change 921506 merged by Jelto:
[operations/puppet@production] gitlab: use sshkey for git-ssh public keys
Change 960034 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] gitlab: use one sshkey for gitlab and remove suffix
Change 960034 merged by Jelto:
[operations/puppet@production] gitlab: use one sshkey for gitlab and remove suffix
The public key for gitlab.wikimedia.org and the replicas is configured in wmf-known-hosts now. So the key can be installed using wmf-update-known-hosts-prodcution or from config-master:
curl -s https://config-master.wikimedia.org/known_hosts | grep gitlab.wikimedia.org gitlab.wikimedia.org,208.80.153.8,2620:0:860:1:208:80:153:8 ecdsa ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIeZ3jAw1WCGn28jQOPbA+oP88UfJRqjD7rH9P3r0sGXv3GX0P3gqpTpsCeNk+xhhyOBMr8lN7vHNk5ZUmLFxSI= root@gitlab1001
The puppet code does a dns lookup of the actual configured GitLab domain. So this should be updated automatically after a switchover.
We added the key for ecdsa only, because otherwise there are puppet resource naming conflicts. This can be fixed with puppet 7. A TODO was added to the puppet code.
I'm closing this task.
I'm reopening the task because there are issues with malformed known_hosts file which prevents users from pushing to GitLab.
The gitlab service urls have a duplicate key algorithm field (as showed above as well):
gitlab.wikimedia.org,208.80.153.8,2620:0:860:1:208:80:153:8 ecdsa ecdsa-sha2-nistp256 ...
This is because the SSH key from private puppets starts with ecdsa-sha2-nistp256 ... and the @@sshkey also appends a key type in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/gitlab/manifests/ssh.pp#65.
I'll work on a puppet patch to make sure the duplicate algorithm field is removed.
Change 1013004 had a related patch set uploaded (by Jelto; author: Jelto):
[operations/puppet@production] gitlab: remove duplicate key type from gitlab known_hosts
There was a question in the -gitlab IRC channel if the host key had changed with a reference to this ticket.
Seems like the reason. Until that gets merged I pasted the 3 different public keys from the private repo in IRC. Maybe there would be value in moving the public part to the public repo. But if we are limiting to one type now maybe that won't be an issue anymore.
Change #1013004 merged by Jelto:
[operations/puppet@production] gitlab: remove duplicate key type from gitlab known_hosts
This should be fixed now. wmf known hosts contains gitlab.wikimedia.org with the ecdsa-sha2-nistp256 algorithm only:
curl -s https://config-master.wikimedia.org/known_hosts | grep gitlab.wikimedia.org gitlab.wikimedia.org,208.80.153.8,2620:0:860:1:208:80:153:8 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIeZ3jAw1WCGn28jQOPbA+oP88UfJRqjD7rH9P3r0sGXv3GX0P3gqpTpsCeNk+xhhyOBMr8lN7vHNk5ZUmLFxSI= root@gitlab1001
compare it to T337107#9238378