We need to be careful with huge QCOW2 files because moving them around will be really painful.

This will not be a problem once we have networked block storage, as the NFS servers would be just acting as app servers. In a way, networked block storage is a blocker for this task.

There is also a question about network throughput with the hypervisor's NIC being used to retrieve data from the distributed storage node and send it out to the NFS client. A dedicated backend network could alleviate these issues.

               A                    B
        +--------------+    +---------------+
        |              |    |               |
+-------v-----+    +---+----v-----+    +----+---------+
|             |    |              |    |              |
|  Ceph Node  |    |  NFS Server  |    |  NFS Client  |
|             |    |              |    |              |
+-------+-----+    +---^----+-----+    +----^---------+
        |              |    |               |
        +--------------+    +---------------+
               C                    D

NFS Server: A,B,C,D are flowing through a single 10GbE NIC in this diagram, potentially offering 25% in each direction. That's for a single VM, the hypervisor is likely to be running others.

It seems this ticket should be closed in light of the Ceph goal, right?

Closed or just listed as blocked.

If listed as blocked, it probably would want to be totally rewritten. Probably closed lol.

My thought process when writing this was not about using virtualization to turn our pets into cattle. It was about network isolation benefits of putting the NFS servers into the 172.16 network. I agree that evacuating enormous QCOW2 files to another cloudvirt would be functionally impossible.

Converting the base hardware to a cloudvirt and using virtualization for network isolation is the same thing that we are in the process of doing right now for the ToolsDB, OpenStreetMaps, and WikiLabels databases. Maybe we should get more experience with those instances however before we rush into repeating the pattern for other services.

I would like to give a reminder that we don't need to convert the hardware to a 'cloudvirt' server to have it available in the openstack instance network.
We could just hook an additional NIC to the 172.16 subnet/VLAN and then reserve that address in neutron for that concrete NIC. This was mentioned already somewhere by some of you, and just refreshing the idea here.

Quick and dirty diagram:

In T216422#4993392, @aborrero wrote:

I would like to give a reminder that we don't need to convert the hardware to a 'cloudvirt' server to have it available in the openstack instance network.
We could just hook an additional NIC to the 172.16 subnet/VLAN and then reserve that address in neutron for that concrete NIC. This was mentioned already somewhere by some of you, and just refreshing the idea here.

Great point @aborrero. One question to consider if we took this approach: would having a storage server (NFS or otherwise) attached to both the production 10.x network and the cloud tenant 172.16.x network actually provide any isolation or protection to the prod network from possible attacks originating from the cloud tenant network?

That's an interesting thought. Without digging into the security implications here, this would simply change where the management of the rules lies. I'm not sure it actually would be a gain for isolation. I think it would be a gain for convenience in a way that is perhaps not actually good.

If we consider the worst case, which is an attacker gaining control on the storage server proc, in the cloudvirt case there is still a hypervisor layer between the attacker and production network.
We could do a similar thing and isolate the storage server daemon/proc in the shared network case by using namespaces. That namespace layer would be the only thing between the compromised process and production network.

Kind of typical security tradeoff.

Ceph is here! T261132
Cinder will be piloted soon and this should be able to move forward when capacity is available.

Things like this are also related to https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/notes/NAT_loophole/NFS

We now have cinder. We still need additional space to do this, and we also need to be sure we actually want to do it. There are some complications in it.

It doesn't seem like we want to because the performance characteristics and network flows will not really be good. This would likely need to be scrapped in favor of doing something like hardware management in openstack.