Turns out John already made a patch for this back in 2022: https://gerrit.wikimedia.org/r/c/operations/puppet/+/790657

We ran into this before with the Buster image, so probably we simply need to carry the previous fix forward: https://phabricator.wikimedia.org/T289694

Great work, really useful and well done!

The Ganeti version we currently run lacks support for chained certs in rapid. This was implemented in https://github.com/ganeti/ganeti/pull/1625 which I have backported to an updated ganeti 3.0.2-1~deb11u1+wmf1 package and with that RAPI access over the PKI certs is working fine.

I'm taking this one, for coordinationd and partly implementing myself.

There is progress, the last change only happened on October 26. This is a long standing task with low priority which is being worked on when time allows, as such I'm reassigning it myself again.

@Jhancock.wm Please note that these need to have virtualization enabled during provisioning, they will be used as virtualisation servers.

In T349402#9340355, @MoritzMuehlenhoff wrote:

crm2001.codfw.wmnet has been created and configured to allow logins by fr-tech SREs. Let me know if you run into any issues, I'm resolving the task for now.

I also created https://gerrit.wikimedia.org/r/c/operations/puppet/+/975242/ to give you sudo rights for root on the VM(s), but that needs approval in the SRE IF meeting happening next Monday.

Please also enable virtualisation for these in the BIOS, they will serve as virt servers.

crm2001.codfw.wmnet has been created and configured to allow logins by fr-tech SREs. Let me know if you run into any issues, I'm resolving the task for now.

When you reimage to Bookworm, please make sure to directly reimage them into the Puppet 7 environment (by passing the -p7 parameter to the reimage cookbook). The alert* hosts are currently on Buster for which we can't use Puppet 7, that's why the current alert* hosts haven't been migrated to Puppet 7 yet.

As part of the Puppet migration we already switched all Buster clients (where version of GNUTLS had problems with the new cert) towards OpenSSL, more details in the task Southparkfan linked.

I'll also open a separate task to eventually also move Bullseye and Bookworm hosts to OpenSSL, the less we use GNUTLS the better.

In T351181#9333641, @MoritzMuehlenhoff wrote:

In T351181#9333629, @jbond wrote:

Well i have updated apt1001 to 8.2102.0-2~deb10u1 and i still see the problem so that would suggest its not an issue with rsyslog :/. perhaps a different option would be to pressure T347565, however i fear we may hit the same issue

It's probably rather an issue in GNUTLS? rsyslog 8.2102 has support for OpenSSL (via the rsyslog-openssl package), we could maybe try that?

In T351181#9333629, @jbond wrote:

Well i have updated apt1001 to 8.2102.0-2~deb10u1 and i still see the problem so that would suggest its not an issue with rsyslog :/. perhaps a different option would be to pressure T347565, however i fear we may hit the same issue

@Dwisehaupt I think we have all data now except the hostname, see my earlier comment. crm1001 or something else?