ok, I'm digging in the nova-network code, and I see this:

1. NOTE(vish): if we are not multi_host pass to the network host
2. NOTE(tr3buchet): but if we are, host came from instance.host

That suggests that multi_host /only/ works in the use case where nova-network runs on each compute node. So multi_host with two external nodes is probably a non-starter.

OK, new proposal, which I've sent to the openstack list for criticism:

1. Install nova-network on all compute nodes
2. nova-network delete (current network setting)
3. nova-network create (same network settings as in step 2 except with multi_host on)
4. restart nova-network on all compute nodes

My supposition is that after doing that, old instances will continue to use my old, standalone network node (as specified in their cached dhcp settings) until they are restarted at which point they'll switch over to using their local compute host as a router thanks to the multi_host setting.

The only horrifying thing about this plan is that at this point the network settings on labnet1001 will remain cached but not specified anyplace. So a simple 'service nova-network-restart' on labnet1001 will break everything. That means that labnet1001 would have to remain unpuppetized and hands-off while we restart the virt nodes :(

ok, I've learned a bit more about transitioning between multiple networks.

Barring additional specs, when creating a new instance, nova queries the list of networks and associates the first one with the new instance. Jynus thinks he can hack the back-end database to specify which entry is 'first'. So we ought to be able to create a second network and then switch new instances over to it at our leisure. This will leave the old network defined in the meantime.

Currently intra-project network traffic isn't blocked by security groups & Co. Will this still be the case for traffic between an "old" and a "new" instance of the same project, i. e. instances on different "networks"?

That's a good question -- I'm not sure. They might wind up isolated from one another.

Updated, tentative plan:

0) switch to arp on the labs network (rather than explicitly routing everything to labnet1001 as now)

1. Disable puppet on labnet1001 so it doesn't restart nova-network and freak out
2. Install nova-network on all compute nodes
3. change network record in place to settings that support a multi_host network
4. restart nova-network on all compute nodes
5. wait for dhcp leases to expire on VMs, at which point they should be handed a new router ip using the new multi_host model
6. (eventually) remove nova-network from labnet1001