A few things are confusing in that https://bugs.launchpad.net/neutron/+bug/1365476 seems to have been targeted for a fix in Liberty and we are at Mitaka, but the symptoms are so very close to the original bug reports. My thinking is there is probably another variation of the same issue at play. The good news is VXLAN isn't required for the initial transition from nova-network but it muddies the waters in trying to define a common HA story for the VPC gateways.

My thinkning is:

• test w/o l2pop
• think on a different gateway HA mechanism (maybe similar to our current hacks until Newton or fix)
• ?

In labtestvirt2003.codfw.wmnet I can see the vxlan traffic by sniffing in eth0:

aborrero@labtestvirt2003:~$ sudo tcpdump -i eth0 udp port 8472
[...]
10:18:15.878886 IP labtestvirt2003.codfw.wmnet.35918 > labtestneutron2002.codfw.wmnet.8472: OTV, flags [I] (0x08), overlay 0, instance 2
ARP, Request who-has 172.16.130.1 tell 172.16.130.15, length 28
10:18:15.879545 IP labtestneutron2001.codfw.wmnet.59104 > labtestvirt2003.codfw.wmnet.8472: OTV, flags [I] (0x08), overlay 0, instance 2
ARP, Reply 172.16.130.1 is-at fa:16:3e:3f:d3:9a (oui Unknown), length 28
[...]

However, sniffing in vxlan-2 doesn't show the ARP reply packets:

aborrero@labtestvirt2003:~ $ sudo tcpdump -i vxlan-2
[...]
10:18:15.878856 ARP, Request who-has 172.16.130.1 tell 172.16.130.15, length 28
10:18:16.016930 ARP, Request who-has 172.16.130.1 tell 172.16.130.13, length 28
[...]

This could mean there is some mechanism preventing the replies from entering the vxlan-2 interface even though the packets arrived at the server via eth0.

I was able to get the setup working by disabling l2population, doing the following steps:

• hosts are: labtestneutron2001, labtestneutron2002, labtestcontrol2003, labtestvirt2002, labtestmetal2001.
• disable puppet on all of them
• edit the file /etc/neutron/plugins/ml2/ml2_config.ini
• in the [ml2] section, delete l2population in the mechanism_drivers= directive. It ends like mechanism_drivers = linuxbridge
• restart services: sudo systemctl restart neutron-l3-agent.service neutron-linuxbridge-agent.service neutron-metada-agent.service (some hosts may not have all the services)

After these steps, the traffic using the vxlan-2 interface cross in both direction, and communication is therefore effective for VMs in the 172.16.130.0/24range (for example, VM at 172.16.130.13).

This indicates that, as @chasemp suggested, there is some weird bug in the l2population mechanism (or other misconfiguration in our side).

BTW this makes a floatingip assigned to an instance work again.

BTW if you assign a floating IP to an instance, you can SSH directly to it with this ssh config:

## labtestn
Host 172.16.*
    User root
    ProxyCommand ssh -W %h:%p labtestcontrol2003.wikimedia.org
    IdentityFile ~/.ssh/my_wmf_cloud_strong_key

Nice, a note that we decided to disable l2pop for now

Change 436319 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: neutron: disable l2population

https://gerrit.wikimedia.org/r/436319

Change 436319 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: neutron: disable l2population

https://gerrit.wikimedia.org/r/436319