Change 638146 had a related patch set uploaded (by Ahmon Dancy; owner: Ahmon Dancy):
[operations/puppet@production] openstack: Enable support for nested VMs

https://gerrit.wikimedia.org/r/638146

For the context, we once had CI jobs running the Android Emulator and they had access to /dev/kvm. That eventually disappeared at some point.

The relevant OpenStack doc seems to be: https://docs.openstack.org/nova/train/admin/configuration/hypervisor-kvm.html#nested-guest-support . If I have processed it properly:

On the host set kernel option options kvm_intel nested=1.

And for OpenStack:

  # Define custom CPU restriction to the lowest
  # common subset of features across all hypervisors.
  # Until we decom cloudvirt1001-1009, this is Ivy Bridge.
  cpu_mode=custom
  cpu_model=IvyBridge-IBRS
+ cpu_model_extra_flags = vmx,pcid

As a workaround if needed and worst case scenario, you can try running bare/non-hardware backed qemu (really slow though :S)

In T267433#6616589, @dcaro wrote:

As a workaround if needed and worst case scenario, you can try running bare/non-hardware backed qemu (really slow though :S)

I tested this approach and the performance was totally unacceptable.

update: I'm still waiting to have codfw1dev working properly so I can test things there. Right now our puppetmasters there are messed up (probably my doing) and @jbond is looking at untangling the cert mess there.

Thanks for the update @Andrew.

I still need to merge and implement this but it should be straightforward -- ping me if you don't hear back by the end of the week.

Change 638146 merged by Andrew Bogott:
[operations/puppet@production] openstack: Enable support for nested VMs

https://gerrit.wikimedia.org/r/638146

This will only be activated as hypervisors are rebooted. The current set of hypervisors that have been rebooted since the patch was merged are:

• cloudvirt1025
• cloudvirt1026

@dancy, the easiest workaround for this is for you to create your new VM(s) and then check to see what host they land on. If it's not any of the above, just ping me and I'll move them.

In T267433#6670257, @Andrew wrote:

This will only be activated as hypervisors are rebooted. The current set of hypervisors that have been rebooted since the patch was merged are:

• cloudvirt1025
• cloudvirt1026

@dancy, the easiest workaround for this is for you to create your new VM(s) and then check to see what host they land on. If it's not any of the above, just ping me and I'll move them.

I started
https://horizon.wikimedia.org/project/instances/ac4b2b46-ed0a-40fa-97d0-184330e9c82d/
It is currently on cloudvirt1023

ok! I moved it to cloudvirt1025; you might want to reboot it before you start your testing. Please let me know how/if it works!

@Andrew It's working! This is great. Thank you very much for getting this change in!