Allow routing between eqiad and eqiad1 regions
Open, NormalPublic

Description

We can almost live without this, but routing between the different regions will make a lot of things (e.g. bastions and proxies) a lot easier to deal with.

Andrew created this task.Thu, Aug 23, 3:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Aug 23, 3:36 PM
chasemp triaged this task as Normal priority.Thu, Aug 23, 3:42 PM
chasemp edited subscribers, added: chasemp; removed: Aklapper, Andrew.

We talked today about this problem and the most practical solution for this migration from bastions through toolforge seems to be allowing connectivity between the old instance network and the new one at the neutron virtual router layer. This would mean main instances can connect to 172.x instances through the neutron router and vice versa for eqiad1 instances. There seem to be 4 steps involved here:

  1. Add the cloud-instance1-eqiad network to the cloudnet trunks
  2. Add a LAN interface for cloud-instance1-eqiad to the virtual neutron router that exists now
  3. Add a route via DHCP option 121 (I think?) that tells cloud-instance1-eqiad instances to go through the neutron virtual router IP on the cloud-instance1-eqiad network.
  4. Make changes to relevant security groups

Which project is this about? Tag welcome so it can be seen in that basket :)

@ayounsi could you handle this trunking updates?
We would like to add VLAN 1102 to both cloudnet1003.eqiad.wmnet and cloudnet1004.eqiad.wmnet in their eth1 NIC.

(Please, confirm v1102 is [labs|cloud]-instances1-b-eqiad with addressing 10.68.16.0/21).

@ayounsi could you handle this trunking updates?
We would like to add VLAN 1102 to both cloudnet1003.eqiad.wmnet and cloudnet1004.eqiad.wmnet in their eth1 NIC.

(Please, confirm v1102 is [labs|cloud]-instances1-b-eqiad with addressing 10.68.16.0/21).

Hold on, I would like to test this setup first in labtest/labtestn, so please do this instead:

Add VLAN 2102 to both labtestneutron2001.codfw.wmnet and labtestneutron2002.codfw.wmnet in their eth1 NIC.
Other VLANs in this trunk are: 2105 and 2120, so we will end up with 3.

(Please confirm v2102 is labs-instances1-b-codfw with addressing 10.196.16.0/24).

ayounsi added a comment.EditedMon, Aug 27, 5:59 PM

(Please confirm v2102 is labs-instances1-b-codfw with addressing 10.196.16.0/24).

From DNS, it's 10.196.16.0/21 but close enough :) vlan name is correct.

I can easily trunk that new vlan, let me know when to do it.

[edit interfaces ge-5/0/42 unit 0 family ethernet-switching vlan]
-       members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw ];
+       members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw cloud-instances1-b-codfw ];
[edit interfaces ge-8/0/11 unit 0 family ethernet-switching vlan]
-       members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw ];
+       members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw cloud-instances1-b-codfw ];

Please, go ahead. I understand adding a new trunk won't affect any traffic in the machine.

Mentioned in SAL (#wikimedia-operations) [2018-08-27T18:12:00Z] <XioNoX> trunking cloud-instances1-b-codfw to labtestneutron2002:eth1 and labtestneutron2001:eth1 - T202636

Change 455615 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: labtestn: add network link with labtest deployment network

https://gerrit.wikimedia.org/r/455615

Change 455615 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: labtestn: add network link with labtest deployment network

https://gerrit.wikimedia.org/r/455615

Trying stuff (in labtest/labtestn). I used the name compat to refer to all this sutff.

Register the nova-network subnet into neutron:

oot@labtestcontrol2003:~# neutron subnet-create --no-gateway --name compat-subnet --ip-version 4 --disable-dhcp --allocation-pool start=10.196.16.3,end=10.196.16.3 flatcloudinstancesb 10.196.16.0/24
Created a new subnet:
+-------------------+------------------------------------------------+
| Field             | Value                                          |
+-------------------+------------------------------------------------+
| allocation_pools  | {"start": "10.196.16.3", "end": "10.196.16.3"} |
| cidr              | 10.196.16.0/24                                 |
| created_at        | 2018-08-28T12:21:52                            |
| description       |                                                |
| dns_nameservers   |                                                |
| enable_dhcp       | False                                          |
| gateway_ip        |                                                |
| host_routes       |                                                |
| id                | 27f55952-7698-4253-8e40-72d17761ecf9           |
| ip_version        | 4                                              |
| ipv6_address_mode |                                                |
| ipv6_ra_mode      |                                                |
| name              | compat-subnet                                  |
| network_id        | 05a5494a-184f-4d5c-9e98-77ae61c56daa           |
| subnetpool_id     |                                                |
| tenant_id         | admin                                          |
| updated_at        | 2018-08-28T12:21:52                            |
+-------------------+------------------------------------------------+

How network objects look:

root@labtestcontrol2003:~# neutron net-list
+--------------------------------------+-------------------------+-------------------------------------------------------+
| id                                   | name                    | subnets                                               |
+--------------------------------------+-------------------------+-------------------------------------------------------+
| d967e056-efc3-46f2-b75b-c906bb5322dc | HA network tenant admin | 651250de-53ca-4487-97ce-e6f65dc4b8ec 169.254.192.0/18 |
| 05a5494a-184f-4d5c-9e98-77ae61c56daa | flatcloudinstancesb     | 27f55952-7698-4253-8e40-72d17761ecf9 10.196.16.0/24   |
|                                      |                         | 7adfcebe-b3d0-4315-92fe-e8365cc80668 172.16.128.0/24  |
| 57017d7c-3817-429a-8aa3-b028de82cdcc | flattransportb          | 5f646219-ce2c-4eb8-8a40-14848c4aab22 10.192.22.0/24   |
|                                      |                         | 9dd8c6f6-9b58-4a14-a920-72b201c6b325 172.16.129.0/24  |
+--------------------------------------+-------------------------+-------------------------------------------------------+

Create our compat port:

root@labtestcontrol2003:~# neutron port-create --name compat-port --fixed-ip subnet_id=27f55952-7698-4253-8e40-72d17761ecf9,ip_address=10.196.16.3 flatcloudinstancesb
Created a new port:
+-----------------------+------------------------------------------------------------------------------------+
| Field                 | Value                                                                              |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                               |
| allowed_address_pairs |                                                                                    |
| binding:host_id       |                                                                                    |
| binding:profile       | {}                                                                                 |
| binding:vif_details   | {}                                                                                 |
| binding:vif_type      | unbound                                                                            |
| binding:vnic_type     | normal                                                                             |
| created_at            | 2018-08-28T12:29:43                                                                |
| description           |                                                                                    |
| device_id             |                                                                                    |
| device_owner          |                                                                                    |
| dns_name              |                                                                                    |
| extra_dhcp_opts       |                                                                                    |
| fixed_ips             | {"subnet_id": "27f55952-7698-4253-8e40-72d17761ecf9", "ip_address": "10.196.16.3"} |
| id                    | aed2634f-efc1-46c7-ac32-f80781451e9a                                               |
| mac_address           | fa:16:3e:dd:22:39                                                                  |
| name                  | compat-port                                                                        |
| network_id            | 05a5494a-184f-4d5c-9e98-77ae61c56daa                                               |
| port_security_enabled | True                                                                               |
| security_groups       | 6167edf8-87a0-457a-935d-7fd7fb6e22b2                                               |
| status                | DOWN                                                                               |
| tenant_id             | admin                                                                              |
| updated_at            | 2018-08-28T12:29:43                                                                |
+-----------------------+------------------------------------------------------------------------------------+

Associate port with virtual router:

root@labtestcontrol2003:~# neutron router-interface-add cloudinstances2b-gw port=compat-port
Added interface aed2634f-efc1-46c7-ac32-f80781451e9a to router cloudinstances2b-gw.

The compat IP address now is present in the virtual router:

aborrero@labtestneutron2002:~ $ sudo ip netns exec qrouter-5712e22e-134a-40d3-a75a-1c9b441717ad ip a | grep 10.196.16.3
    inet 10.196.16.3/24 scope global qr-aed2634f-ef

Routing in the virtual router:

aborrero@labtestneutron2002:~ $ sudo ip netns exec qrouter-5712e22e-134a-40d3-a75a-1c9b441717ad ip r
default via 10.192.22.1 dev qg-666fcda7-04 
10.192.22.0/24 dev qg-666fcda7-04  proto kernel  scope link  src 10.192.22.4 
10.196.16.0/24 dev qr-aed2634f-ef  proto kernel  scope link  src 10.196.16.3  <----
169.254.0.0/24 dev ha-e7fcaf0d-ec  proto kernel  scope link  src 169.254.0.1 
169.254.192.0/18 dev ha-e7fcaf0d-ec  proto kernel  scope link  src 169.254.192.1 
172.16.128.0/24 dev qr-21e10025-d4  proto kernel  scope link  src 172.16.128.1 
172.16.129.0/24 dev qg-666fcda7-04  scope link

Not sure if all this setup works. Will test it now.

My comments from IRC:

you added another subnet
you want to remove teh subnet from that network
add a new network
add the new subnet to it
have the new network tie in at the linuxbridge layer like you have
you want the virtual router to add a new subinterface for a new network
that plugs into the correct subinterface on the host
keep in mind those host subinterfaces are setup in puppet not by neutron

root@labtestcontrol2003:~# source /root/novaenv.sh
root@labtestcontrol2003:~# openstack network list
+--------------------------------------+-------------------------+----------------------------------------------------------------------------+
| ID                                   | Name                    | Subnets                                                                    |
+--------------------------------------+-------------------------+----------------------------------------------------------------------------+
| 57017d7c-3817-429a-8aa3-b028de82cdcc | flattransportb          | 5f646219-ce2c-4eb8-8a40-14848c4aab22, 9dd8c6f6-9b58-4a14-a920-72b201c6b325 |
| d967e056-efc3-46f2-b75b-c906bb5322dc | HA network tenant admin | 651250de-53ca-4487-97ce-e6f65dc4b8ec                                       |
| 05a5494a-184f-4d5c-9e98-77ae61c56daa | flatcloudinstancesb     | 27f55952-7698-4253-8e40-72d17761ecf9, 7adfcebe-b3d0-4315-92fe-e8365cc80668 |
+--------------------------------------+-------------------------+----------------------------------------------------------------------------+
root@labtestcontrol2003:~# openstack network show 05a5494a-184f-4d5c-9e98-77ae61c56daa | grep cloud
| name                      | flatcloudinstancesb                                                        |
| provider:physical_network | cloudinstances2b                                                           |
root@labtestcontrol2003:~#

Change 455873 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: labtestn: fix bridge mapping

https://gerrit.wikimedia.org/r/455873

Change 455873 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: labtestn: fix bridge mapping

https://gerrit.wikimedia.org/r/455873

From an instance in labtest (nova-network) I can ping the virtual router in labtestn (neutron) using this setup:

root@labtestcontrol2003:~# neutron net-create compat-net --router:external=true --provider:network_type=flat --provider:physical_network=br-compat --shared
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-08-28T17:15:35                  |
| description               |                                      |
| id                        | a5e6b90d-57a0-4906-8abe-c270680d7250 |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| is_default                | False                                |
| mtu                       | 1500                                 |
| name                      | compat-net                           |
| port_security_enabled     | True                                 |
| provider:network_type     | flat                                 |
| provider:physical_network | br-compat                            |
| provider:segmentation_id  |                                      |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| tenant_id                 | admin                                |
| updated_at                | 2018-08-28T17:15:35                  |
+---------------------------+--------------------------------------+
root@labtestcontrol2003:~# neutron subnet-create --gateway 10.196.16.1 --name compat-subnet --ip-version 4 --disable-dhcp --allocation-pool start=10.196.16.3,end=10.196.16.3 compat-net 10.196.16.0/24
Created a new subnet:
+-------------------+------------------------------------------------+
| Field             | Value                                          |
+-------------------+------------------------------------------------+
| allocation_pools  | {"start": "10.196.16.3", "end": "10.196.16.3"} |
| cidr              | 10.196.16.0/24                                 |
| created_at        | 2018-08-28T17:16:51                            |
| description       |                                                |
| dns_nameservers   |                                                |
| enable_dhcp       | False                                          |
| gateway_ip        | 10.196.16.1                                    |
| host_routes       |                                                |
| id                | 5f5fc653-f98f-4034-b5a0-51d4ecfa34ca           |
| ip_version        | 4                                              |
| ipv6_address_mode |                                                |
| ipv6_ra_mode      |                                                |
| name              | compat-subnet                                  |
| network_id        | a5e6b90d-57a0-4906-8abe-c270680d7250           |
| subnetpool_id     |                                                |
| tenant_id         | admin                                          |
| updated_at        | 2018-08-28T17:16:51                            |
+-------------------+------------------------------------------------+
root@labtestcontrol2003:~# neutron port-create --name compat-port --fixed-ip subnet_id=compat-subnet,ip_address=10.196.16.3 compat-net
Created a new port:
+-----------------------+------------------------------------------------------------------------------------+
| Field                 | Value                                                                              |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                               |
| allowed_address_pairs |                                                                                    |
| binding:host_id       |                                                                                    |
| binding:profile       | {}                                                                                 |
| binding:vif_details   | {}                                                                                 |
| binding:vif_type      | unbound                                                                            |
| binding:vnic_type     | normal                                                                             |
| created_at            | 2018-08-28T17:18:29                                                                |
| description           |                                                                                    |
| device_id             |                                                                                    |
| device_owner          |                                                                                    |
| dns_name              |                                                                                    |
| extra_dhcp_opts       |                                                                                    |
| fixed_ips             | {"subnet_id": "5f5fc653-f98f-4034-b5a0-51d4ecfa34ca", "ip_address": "10.196.16.3"} |
| id                    | 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa                                               |
| mac_address           | fa:16:3e:82:43:f8                                                                  |
| name                  | compat-port                                                                        |
| network_id            | a5e6b90d-57a0-4906-8abe-c270680d7250                                               |
| port_security_enabled | True                                                                               |
| security_groups       | 6167edf8-87a0-457a-935d-7fd7fb6e22b2                                               |
| status                | DOWN                                                                               |
| tenant_id             | admin                                                                              |
| updated_at            | 2018-08-28T17:18:29                                                                |
+-----------------------+------------------------------------------------------------------------------------+
root@labtestcontrol2003:~# neutron router-interface-add cloudinstances2b-gw port=compat-port
Added interface 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa to router cloudinstances2b-gw.
root@labtestcontrol2003:~# neutron port-show 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa
+-----------------------+------------------------------------------------------------------------------------+
| Field                 | Value                                                                              |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                               |
| allowed_address_pairs |                                                                                    |
| binding:host_id       | labtestneutron2001                                                                 |
| binding:profile       | {}                                                                                 |
| binding:vif_details   | {"port_filter": true}                                                              |
| binding:vif_type      | bridge                                                                             |
| binding:vnic_type     | normal                                                                             |
| created_at            | 2018-08-28T17:18:29                                                                |
| description           |                                                                                    |
| device_id             | 5712e22e-134a-40d3-a75a-1c9b441717ad                                               |
| device_owner          | network:router_interface                                                           |
| dns_name              |                                                                                    |
| extra_dhcp_opts       |                                                                                    |
| fixed_ips             | {"subnet_id": "5f5fc653-f98f-4034-b5a0-51d4ecfa34ca", "ip_address": "10.196.16.3"} |
| id                    | 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa                                               |
| mac_address           | fa:16:3e:82:43:f8                                                                  |
| name                  | compat-port                                                                        |
| network_id            | a5e6b90d-57a0-4906-8abe-c270680d7250                                               |
| port_security_enabled | True                                                                               |
| security_groups       | 6167edf8-87a0-457a-935d-7fd7fb6e22b2                                               |
| status                | ACTIVE                                                                             |
| tenant_id             | admin                                                                              |
| updated_at            | 2018-08-28T17:19:14                                                                |
+-----------------------+------------------------------------------------------------------------------------+
ayounsi removed a subscriber: ayounsi.Tue, Aug 28, 5:33 PM

Change 456127 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: nova-network: allow extra dnsmasq/dhcp option

https://gerrit.wikimedia.org/r/456127

Live testing the above patch, I see clients getting the new static route:

root@aborrero-test4:~# ip r
default via 10.196.16.1 dev eth0 
10.196.16.0/24 dev eth0  proto kernel  scope link  src 10.196.16.46 
172.16.128.0/24 via 10.196.16.3 dev eth0 

root@aborrero-test4:~# traceroute 172.16.128.1
traceroute to 172.16.128.1 (172.16.128.1), 30 hops max, 60 byte packets
 1  172.16.128.1 (172.16.128.1)  1.447 ms  1.390 ms  1.376 ms

And a packet nova-network --> neutron uses the right route:

root@aborrero-test4:~# traceroute 172.16.128.197
traceroute to 172.16.128.197 (172.16.128.197), 30 hops max, 60 byte packets
 1  10.196.16.3 (10.196.16.3)  0.518 ms  0.469 ms  0.442 ms
 2  172.16.128.197 (172.16.128.197)  0.785 ms  0.763 ms  0.738 ms
aborrero added a comment.EditedThu, Aug 30, 1:00 PM

And a packet neutron --> nova-network uses the right route:

root@aborrero-test2:~# ping 10.196.16.46 -c1
PING 10.196.16.46 (10.196.16.46) 56(84) bytes of data.
64 bytes from 10.196.16.46: icmp_seq=1 ttl=63 time=0.739 ms

--- 10.196.16.46 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.739/0.739/0.739/0.000 ms

root@aborrero-test2:~# traceroute  -I 10.196.16.46
traceroute to 10.196.16.46 (10.196.16.46), 30 hops max, 60 byte packets
 1  172.16.128.1 (172.16.128.1)  0.519 ms  0.515 ms  0.515 ms
 2  10.196.16.46 (10.196.16.46)  1.440 ms  1.437 ms  1.435 ms

So we can think on moving this setup to eqiad1/main

Change 456127 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: nova-network: allow extra dnsmasq/dhcp option

https://gerrit.wikimedia.org/r/456127

Mentioned in SAL (#wikimedia-cloud) [2018-08-30T13:07:56Z] <arturo> T202636 internal network routing now exists in labtest/labtestn for VM to communicate with each other

Note: I would like to explore neutron firewalling options to limit which traffic is able to transit the virtual router (probably only bastion traffic)

Note: I would like to explore neutron firewalling options to limit which traffic is able to transit the virtual router (probably only bastion traffic)

So we'd be blocking all other kinds of traffic between instances sitting on nova-network vs. neutron? Doesn't that mostly defeat the purpose of this?

Andrew added a comment.Tue, Sep 4, 3:12 PM

Note: I would like to explore neutron firewalling options to limit which traffic is able to transit the virtual router (probably only bastion traffic)

I'm not sure this is a good idea (or at least worth doing). Individual projects already have default restrictive firewall rules... it will be baffling to users if the UI-displayed rules aren't the whole story and there's an additional behind-the-scenes filter. Also I'm sure there are a lot of inter-project communication corner cases; having clear routing between regions should at least make those corner cases fixable by users.

Ok, I will withdraw any special firewalling and will proceed to migrating this setup to main/eqiad1 this week.

@ayounsi could you handle this trunking updates?
We would like to add VLAN 1102 to both cloudnet1003.eqiad.wmnet and cloudnet1004.eqiad.wmnet in their eth1 NIC.

(Please, confirm v1102 is [labs|cloud]-instances1-b-eqiad with addressing 10.68.16.0/21).

@ayounsi could we move forward with this?

Change 459573 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: eqiad1: add compat network link with main deployment network

https://gerrit.wikimedia.org/r/459573

(Please, confirm v1102 is [labs|cloud]-instances1-b-eqiad with addressing 10.68.16.0/21).

Confirmed.

ayounsi@asw2-b-eqiad# show | compare 
[edit interfaces interface-range cloud-net-trunk unit 0 family ethernet-switching vlan]
-       members [ cloud-instances2-b-eqiad cloud-instance-transport1-b-eqiad ];
+       members [ cloud-instances2-b-eqiad cloud-instance-transport1-b-eqiad cloud-instances1-b-eqiad ];

To be pushed.

Mentioned in SAL (#wikimedia-operations) [2018-09-10T17:42:47Z] <XioNoX> trunk cloud-instances1-b-eqiad to cloudnet1003/4:eth1 - T202636

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:22:40Z] <arturo> added myself as project admin while investigating where 10.68.16.2 is assigned and why (T202636)

I would like to use the address 10.68.16.2 for the virtual router, but that seems assigned.

Will use 10.68.23.253 instead.

root@labcontrol1001:~# nova fixed-ip-get 10.68.23.253
+--------------+---------------+----------+------+----------+
| address      | cidr          | hostname | host | reserved |
+--------------+---------------+----------+------+----------+
| 10.68.23.253 | 10.68.16.0/21 | -        | -    | False    |
+--------------+---------------+----------+------+----------+
root@labcontrol1001:~# nova fixed-ip-reserve 10.68.23.253
root@labcontrol1001:~# nova fixed-ip-get 10.68.23.253
+--------------+---------------+----------+------+----------+
| address      | cidr          | hostname | host | reserved |
+--------------+---------------+----------+------+----------+
| 10.68.23.253 | 10.68.16.0/21 | -        | -    | True     |
+--------------+---------------+----------+------+----------+

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:33:42Z] <arturo> T202636 manually reserve 10.68.23.253 (in nova-network)

Change 459573 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: eqiad1: add compat network link with main deployment network

https://gerrit.wikimedia.org/r/459573

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:36:54Z] <arturo> T202636 creating br-compat bridge in eqiad1 for the compat network

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:53:28Z] <arturo> T202636 creating all the compat-network configuration in neutron

This is what I did:

root@cloudcontrol1003:~# neutron net-create compat-net --router:external=true --provider:network_type=flat --provider:physical_network=br-compat --shared
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| availability_zone_hints   |                                      |
| availability_zones        |                                      |
| created_at                | 2018-09-11T10:41:59                  |
| description               |                                      |
| id                        | f6905bee-8c2c-4b3c-babe-17273694b4da |
| ipv4_address_scope        |                                      |
| ipv6_address_scope        |                                      |
| is_default                | False                                |
| mtu                       | 1500                                 |
| name                      | compat-net                           |
| port_security_enabled     | True                                 |
| provider:network_type     | flat                                 |
| provider:physical_network | br-compat                            |
| provider:segmentation_id  |                                      |
| router:external           | True                                 |
| shared                    | True                                 |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tags                      |                                      |
| tenant_id                 | admin                                |
| updated_at                | 2018-09-11T10:41:59                  |
+---------------------------+--------------------------------------+
root@cloudcontrol1003:~# neutron subnet-create --gateway 10.68.16.1 --name compat-subnet --ip-version 4 --disable-dhcp --allocation-pool start=10.68.23.253,end=10.68.23.253 compat-net 10.68.16.0/21
Created a new subnet:
+-------------------+--------------------------------------------------+
| Field             | Value                                            |
+-------------------+--------------------------------------------------+
| allocation_pools  | {"start": "10.68.23.253", "end": "10.68.23.253"} |
| cidr              | 10.68.16.0/21                                    |
| created_at        | 2018-09-11T10:43:50                              |
| description       |                                                  |
| dns_nameservers   |                                                  |
| enable_dhcp       | False                                            |
| gateway_ip        | 10.68.16.1                                       |
| host_routes       |                                                  |
| id                | 7c51532a-167c-4894-b5e6-8dc4861b2247             |
| ip_version        | 4                                                |
| ipv6_address_mode |                                                  |
| ipv6_ra_mode      |                                                  |
| name              | compat-subnet                                    |
| network_id        | f6905bee-8c2c-4b3c-babe-17273694b4da             |
| subnetpool_id     |                                                  |
| tenant_id         | admin                                            |
| updated_at        | 2018-09-11T10:43:50                              |
+-------------------+--------------------------------------------------+
root@cloudcontrol1003:~# neutron port-create --name compat-port --fixed-ip subnet_id=compat-subnet,ip_address=10.68.23.253 compat-net
Created a new port:
+-----------------------+-------------------------------------------------------------------------------------+
| Field                 | Value                                                                               |
+-----------------------+-------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                |
| allowed_address_pairs |                                                                                     |
| binding:host_id       |                                                                                     |
| binding:profile       | {}                                                                                  |
| binding:vif_details   | {}                                                                                  |
| binding:vif_type      | unbound                                                                             |
| binding:vnic_type     | normal                                                                              |
| created_at            | 2018-09-11T10:44:20                                                                 |
| description           |                                                                                     |
| device_id             |                                                                                     |
| device_owner          |                                                                                     |
| dns_name              |                                                                                     |
| extra_dhcp_opts       |                                                                                     |
| fixed_ips             | {"subnet_id": "7c51532a-167c-4894-b5e6-8dc4861b2247", "ip_address": "10.68.23.253"} |
| id                    | ac5739f5-db3d-4582-8174-0dfc070a67f6                                                |
| mac_address           | fa:16:3e:e1:bf:5d                                                                   |
| name                  | compat-port                                                                         |
| network_id            | f6905bee-8c2c-4b3c-babe-17273694b4da                                                |
| port_security_enabled | True                                                                                |
| security_groups       | a7fea49f-047b-4cf5-a434-d4b213dd0c4f                                                |
| status                | DOWN                                                                                |
| tenant_id             | admin                                                                               |
| updated_at            | 2018-09-11T10:44:20                                                                 |
+-----------------------+-------------------------------------------------------------------------------------+
root@cloudcontrol1003:~# neutron router-interface-add cloudinstances2b-gw port=compat-port
Added interface ac5739f5-db3d-4582-8174-0dfc070a67f6 to router cloudinstances2b-gw.

At this point I restarted the neutron-linuxbridge-agent service in both cloudnet1003 and cloudnet1004 because I was seeing log messages like these:

2018-09-11 10:44:45.989 36764 ERROR neutron.plugins.ml2.managers [req-3f984a39-c85a-40e3-afb1-c24f0309b29f - - - - -] Failed to bind port ac5739f5-db3d-4582-8174-0dfc070a67f6 on host cloudnet1004 for vnic_type normal using segments [{'segmentation_id': None, 'physical_network': u'br-compat', 'id': u'136161dc-86f6-47a0-82b2-593b9e46d743', 'network_type': u'flat'}]
2018-09-11 10:44:46.002 36764 ERROR neutron.plugins.ml2.managers [req-3f984a39-c85a-40e3-afb1-c24f0309b29f - - - - -] Failed to bind port ac5739f5-db3d-4582-8174-0dfc070a67f6 on host cloudnet1004 for vnic_type normal using segments [{'segmentation_id': None, 'physical_network': u'br-compat', 'id': u'136161dc-86f6-47a0-82b2-593b9e46d743', 'network_type': u'flat'}]
root@cloudcontrol1003:~# neutron port-show compat-port
+-----------------------+-------------------------------------------------------------------------------------+
| Field                 | Value                                                                               |
+-----------------------+-------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                |
| allowed_address_pairs |                                                                                     |
| binding:host_id       | cloudnet1004                                                                        |
| binding:profile       | {}                                                                                  |
| binding:vif_details   | {}                                                                                  |
| binding:vif_type      | binding_failed                                                                      |
| binding:vnic_type     | normal                                                                              |
| created_at            | 2018-09-11T10:44:20                                                                 |
| description           |                                                                                     |
| device_id             | d93771ba-2711-4f88-804a-8df6fd03978a                                                |
| device_owner          | network:router_interface                                                            |
| dns_name              |                                                                                     |
| extra_dhcp_opts       |                                                                                     |
| fixed_ips             | {"subnet_id": "7c51532a-167c-4894-b5e6-8dc4861b2247", "ip_address": "10.68.23.253"} |
| id                    | ac5739f5-db3d-4582-8174-0dfc070a67f6                                                |
| mac_address           | fa:16:3e:e1:bf:5d                                                                   |
| name                  | compat-port                                                                         |
| network_id            | f6905bee-8c2c-4b3c-babe-17273694b4da                                                |
| port_security_enabled | True                                                                                |
| security_groups       | a7fea49f-047b-4cf5-a434-d4b213dd0c4f                                                |
| status                | DOWN                                                                                |
| tenant_id             | admin                                                                               |
| updated_at            | 2018-09-11T10:44:45                                                                 |
+-----------------------+-------------------------------------------------------------------------------------+

After the restart, all seems to work:

root@cloudnet1004:~# ip netns exec qrouter-d93771ba-2711-4f88-804a-8df6fd03978a ip a | grep 10.68.23
    inet 10.68.23.253/21 scope global qr-ac5739f5-db
root@cloudnet1004:~# brctl show br-compat
bridge name	bridge id		STP enabled	interfaces
br-compat		8000.1a1fbd837351	no		eth1.1102
							tapac5739f5-db

root@cloudcontrol1003:~# neutron port-show compat-port
+-----------------------+-------------------------------------------------------------------------------------+
| Field                 | Value                                                                               |
+-----------------------+-------------------------------------------------------------------------------------+
| admin_state_up        | True                                                                                |
| allowed_address_pairs |                                                                                     |
| binding:host_id       | cloudnet1004                                                                        |
| binding:profile       | {}                                                                                  |
| binding:vif_details   | {"port_filter": true}                                                               |
| binding:vif_type      | bridge                                                                              |
| binding:vnic_type     | normal                                                                              |
| created_at            | 2018-09-11T10:44:20                                                                 |
| description           |                                                                                     |
| device_id             | d93771ba-2711-4f88-804a-8df6fd03978a                                                |
| device_owner          | network:router_interface                                                            |
| dns_name              |                                                                                     |
| extra_dhcp_opts       |                                                                                     |
| fixed_ips             | {"subnet_id": "7c51532a-167c-4894-b5e6-8dc4861b2247", "ip_address": "10.68.23.253"} |
| id                    | ac5739f5-db3d-4582-8174-0dfc070a67f6                                                |
| mac_address           | fa:16:3e:e1:bf:5d                                                                   |
| name                  | compat-port                                                                         |
| network_id            | f6905bee-8c2c-4b3c-babe-17273694b4da                                                |
| port_security_enabled | True                                                                                |
| security_groups       | a7fea49f-047b-4cf5-a434-d4b213dd0c4f                                                |
| status                | ACTIVE                                                                              |
| tenant_id             | admin                                                                               |
| updated_at            | 2018-09-11T10:51:07                                                                 |
+-----------------------+-------------------------------------------------------------------------------------+

So, this should all be working fine now.

In any case DHCP clients need a new DHCP lease to know the new static route.

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T16:08:17Z] <arturo> restarted nova-network after killing all dnsmasq procs in labnet1001 for T202636

Change 459816 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: main: don't offer the static route in DHCP for the compat network

https://gerrit.wikimedia.org/r/459816

Change 459816 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: main: don't offer the static route in DHCP for the compat network

https://gerrit.wikimedia.org/r/459816

Mentioned in SAL (#wikimedia-cloud) [2018-09-11T16:52:14Z] <arturo> again, restarted nova-network after killing all dnsmasq procs in labnet1001 for T202636

Update on state of things.

When setting the classless-static-route extra-option in the dnsmasq config in labnet1001.eqiad.wmnet (for the main deployment), DHCP clients (VMs) in this deployment can't no longer be reached (after they negotiate DHCP with this new config activated).
This is a big contrast with the labtest deployment, in which this setup worked right away.

After much debugging, I tracked down the issue to DHCP clients don't setting their default gateway when the classless-static-route extra-option is sent by the DHCP server.

The RFC3442, which defines the classless-static-route extra-option we are using reads:

DHCP Client Behavior
[...]
   If the DHCP server returns both a Classless Static Routes option and
   a Router option, the DHCP client MUST ignore the Router option.

(https://tools.ietf.org/html/rfc3442)
I'm not sure why that behavior exists, but I'm afraid we should stick to RFCs behaviors.
It seems the workaround/solution is to push 2 routes to DHCP clients using classless-static-route options (default + compat network).

How is possible this setup worked in labtest?

How is possible this setup worked in labtest?

This doesn't work in labtest either!
What I tested in labtest was to renew DHCP leases. There is a different behavior when renewing and when requesting a brand new lease (which happens at boot/reboot).

I'm convinced we can solve this by simply pushing 2 static routes: the default route and the compat route.

Change 459998 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: nova-network: also distribute default route in dnsmasq

https://gerrit.wikimedia.org/r/459998

Change 459998 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: nova-network: also distribute default route in dnsmasq

https://gerrit.wikimedia.org/r/459998

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T11:20:39Z] <arturo> T202636 distributing default routes using classless-static-route for all VMs in main/labtest (dnsmasq/nova-network)

My initial tests rebooting machines in both labtest/main (remember, different DHCP query than a release) shows this is now working!

Mentioned in SAL (#wikimedia-cloud) [2018-09-12T11:22:10Z] <arturo> rebooting bastion-restricted-01 to test new setup from T202636

aborrero closed this task as Resolved.Wed, Sep 12, 3:31 PM
aborrero added a subscriber: Andrew.

@Andrew did some tests and all seems working.

Andrew reopened this task as Open.Fri, Sep 14, 9:35 PM

This is working, but I note that the originating IP is getting natted when I go between regions:

root@bastion-restricted-eqiad1-01:~# tcpdump -n port 4444
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:32:10.664549 IP 208.80.155.255.49432 > 172.16.1.135.4444: Flags [S], seq 2261175948, win 29200, options [mss 1460,sackOK,TS val 805828756 ecr 0,nop,wscale 9], length 0
21:32:10.664637 IP 172.16.1.135.4444 > 208.80.155.255.49432: Flags [R.], seq 0, ack 2261175949, win 0, length 0

That's traffic coming in from 10.68.17.149 in the eqiad region -- it shows as originating from 208.80.155.255. If getting that to work properly is hard then maybe we can live with it, but it's potentially confusing when people are setting up firewalls.

(The good news is that the originating IP is correct when coming from a VM with a public ip -- which the bastions have. So this is moot for most use cases.)

Using NAT means the eqiad-->eqiad1 traffic is reaching the router (i.e, nova-network @ labnet1001), which means the static route is not being applied.

I confirmed this by jumping into the machine:

aborrero@util-abogott-stretch:~$ ip r
default via 10.68.16.1 dev eth0 
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 
10.68.16.0/21 dev eth0 proto kernel scope link src 10.68.17.149

we don't have there our custom static route, which probably means this machine didn't ask for DHCP in a while.

Forcing it:

aborrero@util-abogott-stretch:~$ sudo ifup --force eth0
Internet Systems Consortium DHCP Client 4.3.5
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/eth0/fa:16:3e:62:0e:94
Sending on   LPF/eth0/fa:16:3e:62:0e:94
Sending on   Socket/fallback
DHCPREQUEST of 10.68.17.149 on eth0 to 255.255.255.255 port 67
DHCPACK of 10.68.17.149 from 10.68.16.1
RTNETLINK answers: File exists
bound to 10.68.17.149 -- renewal in 17457 seconds.
Sending network state change signal to nslcd...done.

aborrero@util-abogott-stretch:~$ ip r
default via 10.68.16.1 dev eth0 
10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 
10.68.16.0/21 dev eth0 proto kernel scope link src 10.68.17.149 
172.16.0.0/21 via 10.68.23.253 dev eth0

Now, the tcpdump from the bastion-resctricted-eqiad1-01 POV:

before:
09:38:32.303135 IP 208.80.155.255 > 172.16.1.135: ICMP echo request, id 10798, seq 52, length 64
09:38:32.303234 IP 172.16.1.135 > 208.80.155.255: ICMP echo reply, id 10798, seq 52, length 64
after:
09:40:15.799446 IP 10.68.17.149 > 172.16.1.135: ICMP echo request, id 10923, seq 1, length 64
09:40:15.799542 IP 172.16.1.135 > 10.68.17.149: ICMP echo reply, id 10923, seq 1, length 64

Conclusion: if a machine is not properly using the compat network, let's force a new DHCP renew.

@arturo thanks! I'm pondering whether or not we should just do that cloud-wide... maybe just the bastions and the proxies.

There is logic in the NAT controller to exclude certain destination addresses from NAT (IIUC) and potentially dding the 10. range for eqiad instances would allow preservation of source addresses here?