Change 890434 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/puppet@production] gitlab: allow rsync between replicas

https://gerrit.wikimedia.org/r/890434

Change 890434 merged by Jelto:

[operations/puppet@production] gitlab: allow rsync between replicas

https://gerrit.wikimedia.org/r/890434

Change 890779 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] Change the active gitlab replica host to be the eqiad instance

https://gerrit.wikimedia.org/r/890779

Change 890785 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/dns@master] Update DNS to switch gitlab-replica

https://gerrit.wikimedia.org/r/890785

Mentioned in SAL (#wikimedia-operations) [2023-02-22T11:13:02Z] <eoghan@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on gitlab2002.wikimedia.org with reason: Running failover to gitlab1003 - T329930

Mentioned in SAL (#wikimedia-operations) [2023-02-22T11:13:18Z] <eoghan@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on gitlab2002.wikimedia.org with reason: Running failover to gitlab1003 - T329930

Change 890779 merged by EoghanGaffney:

[operations/puppet@production] Change the active gitlab replica host to be the eqiad instance

https://gerrit.wikimedia.org/r/890779

Change 890785 merged by EoghanGaffney:

[operations/dns@master] Update DNS to switch gitlab-replica

https://gerrit.wikimedia.org/r/890785

After switching gitlab2002 and gitlab1003 I get a SSH key warning, although the hosts use the same host key:

$ git push
Warning: the ECDSA host key for 'gitlab-replica.wikimedia.org' differs from the key for the IP address '2620:0:861:1:208:80:154:15'
Offending key for IP in /home/jelto-wmf/.ssh/known_hosts.d/wmf-cloud:63
Matching host key in /home/jelto-wmf/.ssh/known_hosts.d/wmf-cloud:82
Exiting, you have requested strict checking.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I'm not sure how we handled that last time. I think we announced the warning will appear after the switchover.

I'm not 100% if we can fix this short term for the switchover on Monday. Long term we could try to use a static IP and map it to the current production instance. Maybe we can add the keys to wmf-known-hosts? I'll do some more research.

In T329930#8637475, @Jelto wrote:

I'm not sure how we handled that last time. I think we announced the warning will appear after the switchover.

I think T296944#8496308 is related here.

In T329930#8637828, @Dzahn wrote:

I think T296944#8496308 is related here.

Thanks for the link! We made sure we use the same SSH hostkey on all GitLab instances. So the reported host keys should not change and match the ones in the wikitech page.

I've done some more troubleshooting and it seems to be an issue related to my local SSH config/known hosts. In preparation for this switchover we made sure all GitLab instances use the same host key. gitlab1003 was using a different pair of host keys and I fixed that. However I already connected to that host before and accepted the host key. So the key changed due to the preparation work from my point of view.

I removed the old host key from my known hosts file and repeated the test by editing my local hosts file. I don't see the error about different key for address anymore.

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/17

update settings files for 2023 switchover eqiad -> codfw

Ah, cool, this is a nice explanation and outcome. Thanks!

brennen merged https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/17

update settings files for 2023 switchover eqiad -> codfw