sudo gnt-cluster verify fails in the ulsfo and eqsin cluster afters the update to Bullseye in the "Verifying node status" step where it's connecting in between nodes via SSH in a loop. Apart from that all cluster operations are working fine.
I obtained the exact command from the node-daemon.log:
```
jmm@ganeti5003:~$ sudo ssh -oEscapeChar=none -oHashKnownHosts=no -oGlobalKnownHostsFile=/var/lib/ganeti/known_hosts -oUserKnownHostsFile=/dev/null -oCheckHostIp=no -oConnectTimeout=10 -o\HostKeyAlias=ganeti01.svc.eqsin.wmnet -oPort=22 -oBatchMode=yes -oStrictHostKeyChecking=yes -4 root@ganeti5002.eqsin.wmnet
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:SvgqUu5xa8VQDLoLxIyOmrLn8MeUT9pxnen80BQKSfY.
Please contact your system administrator.
Add correct host key in /dev/null to get rid of this message.
Offending RSA key in /var/lib/ganeti/known_hosts:1
remove with:
ssh-keygen -f "/var/lib/ganeti/known_hosts" -R "ganeti01.svc.eqsin.wmnet"
ECDSA host key for ganeti01.svc.eqsin.wmnet has changed and you have requested strict checking.
Host key verification failed.
```
Surprisingly this doesn't happen on ganeti/drmrs (which was installed with Bullseye), ganeti/esams (which was upgraded from Buster to Bullseye), ganeti/test (which was upgraded from Buster to Bullseye). The same command on ganeti-test works fine:
```
jmm@ganeti-test2002:~$ sudo ssh -oEscapeChar=none -oHashKnownHosts=no -oGlobalKnownHostsFile=/var/lib/ganeti/known_hosts -oUserKnownHostsFile=/dev/null -oCheckHostIp=no -oConnectTimeout=10 -oHostKeyAlias=ganeti-test01.svc.codfw.wmnet -oPort=22 -oBatchMode=yes -oStrictHostKeyChecking=yes -4 root@ganeti-test2001.codfw.wmnet
```
When digging around I also found https://github.com/ganeti/ganeti/issues/1608 which seems to be the exact issue.
This will be fixed by managing /var/lib/ganeti/known_hosts with the fingerprints generated from Puppet, the following clusters are migrated:
[] eqiad
[] codfw
[x] codfw-test
[] esams01
[] esams02
[] ulsfo
[] eqsin
[] drmrs01
[] drmrs02
[] magru01
[] magru02