Reading https://engineering.skroutz.gr/blog/entropy-changes-in-debian/ it became clear that VMs on buster KVM hosts might suffer from lack of entropy on boot, making boot times unnecessarily long.
The article is a pretty great read and while it goes through great lengths to explain everything, we can just get one simple thing out of all of it for now and that's that there is an extremely easy fix, namely passing the RDRAND CPU feature flag to the VMs. All of our hosts are new enough to support it already so there should be no problems.