sudo gnt-cluster verify fails in the ulsfo and eqsin cluster afters the update to Bullseye in the "Verifying node status" step where it's connecting in between nodes via SSH in a loop. Apart from that all cluster operations are working fine.
I obtained the exact command from the node-daemon.log:
jmm@ganeti5003:~$ sudo ssh -oEscapeChar=none -oHashKnownHosts=no -oGlobalKnownHostsFile=/var/lib/ganeti/known_hosts -oUserKnownHostsFile=/dev/null -oCheckHostIp=no -oConnectTimeout=10 -o\HostKeyAlias=ganeti01.svc.eqsin.wmnet -oPort=22 -oBatchMode=yes -oStrictHostKeyChecking=yes -4 root@ganeti5002.eqsin.wmnet @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is SHA256:SvgqUu5xa8VQDLoLxIyOmrLn8MeUT9pxnen80BQKSfY. Please contact your system administrator. Add correct host key in /dev/null to get rid of this message. Offending RSA key in /var/lib/ganeti/known_hosts:1 remove with: ssh-keygen -f "/var/lib/ganeti/known_hosts" -R "ganeti01.svc.eqsin.wmnet" ECDSA host key for ganeti01.svc.eqsin.wmnet has changed and you have requested strict checking. Host key verification failed.
Surprisingly this doesn't happen on ganeti/drmrs (which was installed with Bullseye), ganeti/esams (which was upgraded from Buster to Bullseye), ganeti/test (which was upgraded from Buster to Bullseye). The same command on ganeti-test works fine:
jmm@ganeti-test2002:~$ sudo ssh -oEscapeChar=none -oHashKnownHosts=no -oGlobalKnownHostsFile=/var/lib/ganeti/known_hosts -oUserKnownHostsFile=/dev/null -oCheckHostIp=no -oConnectTimeout=10 -oHostKeyAlias=ganeti-test01.svc.codfw.wmnet -oPort=22 -oBatchMode=yes -oStrictHostKeyChecking=yes -4 root@ganeti-test2001.codfw.wmnet
When digging around I also found https://github.com/ganeti/ganeti/issues/1608 which seems to be the exact issue.
This will be fixed by managing /var/lib/ganeti/known_hosts with the fingerprints generated from Puppet, the following clusters are migrated:
- eqiad
- codfw
- codfw-test
- esams01
- esams02
- ulsfo
- eqsin
- drmrs01
- drmrs02
- magru01
- magru02