We've added a few more persistent databases in codfw1dev; volumes in cloudinfra-codfw1dev are backed up on cloudbackup1002-dev so today I set out to add more disk space to that (ganeti) host.

First, I added an additional drive, with

sudo gnt-instance modify --disk add:size=60g  cloudbackup1002-dev.eqiad.wmnet

That took about 40 minutes, but exited gracefully. After that I need to reboot the host via ganeti, because

WARNING!! After adding a device you have to reboot the VM (from ganeti level, not just reboot within the VM) and it might happen that your device names change and you get no networking.

I rebooted that with the cookbook, and now the the cookbook is looping forever waiting for a reboot. This issue is also documented:

If the VM does not come back from reboot, login via console and root password from pwstore and replace the device name with the "next" one in /etc/network/interfaces. For example replace all occurrences of "ens5" with "ens6" and reboot the machine one more time. also see phab:T272555. If you want to be sure about the "next" device name run ifconfig -a to double check.

So I tried to get a console on the host, but that does not work at all:

andrew@ganeti1048:~$ sudo gnt-instance console cloudbackup1002-dev.eqiad.wmnet
No ED25519 host key is known for ganeti01.svc.eqiad.wmnet and you have requested strict checking.
Host key verification failed.
Failure: command execution error:
Connection to console of instance cloudbackup1002-dev.eqiad.wmnet failed, please check cluster configuration

That may be related to T309724 (that task is still marked as open although the subject line says it's about Bullseye)