This turned out to be also reproducible on classic Ganeti and after some painful debugging given the very early failure (approx 1.5 seconds after Linux starts) I got to the root cause of it:

Since the Trixie 13.4 point release some updated components of the initrd/netboot/kernel environment grew in size and on VMs with 1.5G RAM (as the VMs we had been attempting to create on routed Ganeti) the part of the very system bootup with initiates userspace ttys after the initial Linux boot completed (rootskel in the Debian installer) segfaulted to due a lack of available memory.

Effectively this means that our minimum VM RAM size is now 2GB of RAM. I'll make a change to sre.ganeti.makevm to require at least 2G for all new VMs. I also audited the current VMs and there's various a few more with less than 2G. To avoid surprises on future reimages I'll bump all of these to 2G:

• acmechief2002.codfw.wmnet
• aux-k8s-etcd1003.eqiad.wmnet
• aux-k8s-etcd2003.codfw.wmnet
• aux-k8s-etcd2004.codfw.wmnet
• aux-k8s-etcd2005.codfw.wmnet
• backupmon1001.eqiad.wmnet
• bast4006.wikimedia.org
• bast5004.wikimedia.org
• bast6003.wikimedia.org
• cloudcumin1001.eqiad.wmnet
• cloudcumin2001.codfw.wmnet
• config-master1001.eqiad.wmnet
• config-master2001.codfw.wmnet
• dborch1001.wikimedia.org
• planet1003.eqiad.wmnet
• planet2003.codfw.wmnet
• puppetboard1003.eqiad.wmnet
• puppetboard2003.codfw.wmnet
• testvm2004.codfw.wmnet
• testvm2005.codfw.wmnet
• testvm2006.codfw.wmnet
• testvm2007.codfw.wmnet
• testvm2008.wikimedia.org

Change #1273743 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/cookbooks@master] sre.ganeti.makevm: Bump memory requirements for VMs to 2G

https://gerrit.wikimedia.org/r/1273743

Change #1273743 merged by Muehlenhoff:

[operations/cookbooks@master] sre.ganeti.makevm: Bump memory requirements for VMs to 2G

https://gerrit.wikimedia.org/r/1273743

On a trixie system I had an issue in which the host decided to kill big processes rather than becoming slow (something related to stall detection, not due to VM/swapping), as the debian default assumed more of a k8s-style management. Modifying a kernel parameter worked for me, in case that is relevant.

Ignore this, I hadn't read all the comments.

In any case, backupmon1001.eqiad.wmnet is a very very tiny instance (an apache with just 1 user- me). No problem to up its memory, just making sure you are aware that that may be a waste of resources for that.

Maybe I should migrate that to k8s?

Mentioned in SAL (#wikimedia-operations) [2026-04-17T20:13:21Z] <mutante> planet1003, planet2003 - rebooting on ganeti level for T422596

I did the planet machines:

[ganeti1046:~] $ sudo gnt-instance modify -B memory=2G planet1003.eqiad.wmnet
..
[ganeti1046:~] $ sudo gnt-instance reboot planet1003.eqiad.wmnet

..

[ganeti2047:~] $ sudo gnt-instance modify -B memory=2G planet2003.codfw.wmnet
..
[ganeti2047:~] $ sudo gnt-instance reboot planet2003.codfw.wmnet

and verified they now have 2GB RAM with 1.5GB RAM before.

VM cloudcumin2001.codfw.wmnet rebooted by filippo@cumin1003 with reason: None

VM cloudcumin1001.eqiad.wmnet rebooted by filippo@cumin1003 with reason: None

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm2004.codfw.wmnet

• testvm2004.codfw.wmnet (WARN)
  • Host not found on Icinga, unable to downtime it
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster codfw_test to Netbox

• COMMON_STEPS (FAIL)
  • Failed to run the sre.dns.netbox cookbook, run it manually

ERROR: some step on some host failed, check the bolded items above

VM testvm2005.codfw.wmnet rebooted by jmm@cumin2002 with reason: None

VM testvm2007.codfw.wmnet rebooted by jmm@cumin2002 with reason: None

VM testvm2008.wikimedia.org rebooted by jmm@cumin2002 with reason: None

cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: testvm2006.codfw.wmnet

• testvm2006.codfw.wmnet (WARN)
  • Host not found on Icinga, unable to downtime it
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster codfw02 to Netbox
  • Removed from DebMonitor
  • Removed from Puppet server and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster codfw02 to Netbox

VM puppetboard2003.codfw.wmnet rebooted by jmm@cumin2002 with reason: None

VM puppetboard1003.eqiad.wmnet rebooted by jmm@cumin2002 with reason: None

In T422596#11833600, @jcrespo wrote:

In any case, backupmon1001.eqiad.wmnet is a very very tiny instance (an apache with just 1 user- me). No problem to up its memory, just making sure you are aware that that may be a waste of resources for that.

All good, we have sufficient memory on the Ganeti servers. I've bumped the VM to 2G and will sync up with you when to restart it.

VM backupmon1001.eqiad.wmnet rebooted by jmm@cumin2002 with reason: None

VM aux-k8s-etcd1003.eqiad.wmnet rebooted by jmm@cumin2002 with reason: None

VM aux-k8s-etcd1003.eqiad.wmnet rebooted by jmm@cumin2002 with reason: None

VM aux-k8s-etcd1003.eqiad.wmnet rebooted by jmm@cumin2002 with reason: None

VM aux-k8s-etcd2003.codfw.wmnet rebooted by jmm@cumin2002 with reason: bump memory

VM aux-k8s-etcd2004.codfw.wmnet rebooted by jmm@cumin2002 with reason: bump memory

VM aux-k8s-etcd2005.codfw.wmnet rebooted by jmm@cumin2002 with reason: bump memory

VM config-master2001.codfw.wmnet rebooted by jmm@cumin2002 with reason: bump memory

VM config-master1001.eqiad.wmnet rebooted by jmm@cumin2002 with reason: bump memory

VM bast4006.wikimedia.org rebooted by jmm@cumin2002 with reason: None

VM bast6003.wikimedia.org rebooted by jmm@cumin2002 with reason: None

VM acmechief2002.codfw.wmnet rebooted by jmm@cumin2002 with reason: None