We can almost live without this, but routing between the different regions will make a lot of things (e.g. bastions and proxies) a lot easier to deal with.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T53494 Use Beta cluster as a true canary for code deployments (epic) | |||
| Open | None | T87220 Minimize infrastructure differences between Beta Cluster and production | |||
| Open | None | T196662 Set up LVS in beta like prod | |||
| Resolved | bd808 | T166396 Program 1 Outcome 4: VPS hosting | |||
| Resolved | None | T167293 Nova-network to Neutron migration | |||
| Resolved | aborrero | T202636 Allow routing between eqiad and eqiad1 regions |
Event Timeline
Comment Actions
We talked today about this problem and the most practical solution for this migration from bastions through toolforge seems to be allowing connectivity between the old instance network and the new one at the neutron virtual router layer. This would mean main instances can connect to 172.x instances through the neutron router and vice versa for eqiad1 instances. There seem to be 4 steps involved here:
- Add the cloud-instance1-eqiad network to the cloudnet trunks
- Add a LAN interface for cloud-instance1-eqiad to the virtual neutron router that exists now
- Add a route via DHCP option 121 (I think?) that tells cloud-instance1-eqiad instances to go through the neutron virtual router IP on the cloud-instance1-eqiad network.
- Make changes to relevant security groups
Comment Actions
@ayounsi could you handle this trunking updates?
We would like to add VLAN 1102 to both cloudnet1003.eqiad.wmnet and cloudnet1004.eqiad.wmnet in their eth1 NIC.
(Please, confirm v1102 is [labs|cloud]-instances1-b-eqiad with addressing 10.68.16.0/21).
Comment Actions
Hold on, I would like to test this setup first in labtest/labtestn, so please do this instead:
Add VLAN 2102 to both labtestneutron2001.codfw.wmnet and labtestneutron2002.codfw.wmnet in their eth1 NIC.
Other VLANs in this trunk are: 2105 and 2120, so we will end up with 3.
(Please confirm v2102 is labs-instances1-b-codfw with addressing 10.196.16.0/24).
Comment Actions
From DNS, it's 10.196.16.0/21 but close enough :) vlan name is correct.
I can easily trunk that new vlan, let me know when to do it.
[edit interfaces ge-5/0/42 unit 0 family ethernet-switching vlan] - members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw ]; + members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw cloud-instances1-b-codfw ]; [edit interfaces ge-8/0/11 unit 0 family ethernet-switching vlan] - members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw ]; + members [ cloud-instances2-b-codfw cloud-instance-transport1-b-codfw cloud-instances1-b-codfw ];
Comment Actions
Please, go ahead. I understand adding a new trunk won't affect any traffic in the machine.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2018-08-27T18:12:00Z] <XioNoX> trunking cloud-instances1-b-codfw to labtestneutron2002:eth1 and labtestneutron2001:eth1 - T202636
Comment Actions
Change 455615 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: labtestn: add network link with labtest deployment network
Comment Actions
Change 455615 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: labtestn: add network link with labtest deployment network
Comment Actions
Trying stuff (in labtest/labtestn). I used the name compat to refer to all this sutff.
Register the nova-network subnet into neutron:
oot@labtestcontrol2003:~# neutron subnet-create --no-gateway --name compat-subnet --ip-version 4 --disable-dhcp --allocation-pool start=10.196.16.3,end=10.196.16.3 flatcloudinstancesb 10.196.16.0/24
Created a new subnet:
+-------------------+------------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------------+
| allocation_pools | {"start": "10.196.16.3", "end": "10.196.16.3"} |
| cidr | 10.196.16.0/24 |
| created_at | 2018-08-28T12:21:52 |
| description | |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | |
| host_routes | |
| id | 27f55952-7698-4253-8e40-72d17761ecf9 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | compat-subnet |
| network_id | 05a5494a-184f-4d5c-9e98-77ae61c56daa |
| subnetpool_id | |
| tenant_id | admin |
| updated_at | 2018-08-28T12:21:52 |
+-------------------+------------------------------------------------+How network objects look:
root@labtestcontrol2003:~# neutron net-list +--------------------------------------+-------------------------+-------------------------------------------------------+ | id | name | subnets | +--------------------------------------+-------------------------+-------------------------------------------------------+ | d967e056-efc3-46f2-b75b-c906bb5322dc | HA network tenant admin | 651250de-53ca-4487-97ce-e6f65dc4b8ec 169.254.192.0/18 | | 05a5494a-184f-4d5c-9e98-77ae61c56daa | flatcloudinstancesb | 27f55952-7698-4253-8e40-72d17761ecf9 10.196.16.0/24 | | | | 7adfcebe-b3d0-4315-92fe-e8365cc80668 172.16.128.0/24 | | 57017d7c-3817-429a-8aa3-b028de82cdcc | flattransportb | 5f646219-ce2c-4eb8-8a40-14848c4aab22 10.192.22.0/24 | | | | 9dd8c6f6-9b58-4a14-a920-72b201c6b325 172.16.129.0/24 | +--------------------------------------+-------------------------+-------------------------------------------------------+
Create our compat port:
root@labtestcontrol2003:~# neutron port-create --name compat-port --fixed-ip subnet_id=27f55952-7698-4253-8e40-72d17761ecf9,ip_address=10.196.16.3 flatcloudinstancesb
Created a new port:
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| created_at | 2018-08-28T12:29:43 |
| description | |
| device_id | |
| device_owner | |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "27f55952-7698-4253-8e40-72d17761ecf9", "ip_address": "10.196.16.3"} |
| id | aed2634f-efc1-46c7-ac32-f80781451e9a |
| mac_address | fa:16:3e:dd:22:39 |
| name | compat-port |
| network_id | 05a5494a-184f-4d5c-9e98-77ae61c56daa |
| port_security_enabled | True |
| security_groups | 6167edf8-87a0-457a-935d-7fd7fb6e22b2 |
| status | DOWN |
| tenant_id | admin |
| updated_at | 2018-08-28T12:29:43 |
+-----------------------+------------------------------------------------------------------------------------+Associate port with virtual router:
root@labtestcontrol2003:~# neutron router-interface-add cloudinstances2b-gw port=compat-port Added interface aed2634f-efc1-46c7-ac32-f80781451e9a to router cloudinstances2b-gw.
The compat IP address now is present in the virtual router:
aborrero@labtestneutron2002:~ $ sudo ip netns exec qrouter-5712e22e-134a-40d3-a75a-1c9b441717ad ip a | grep 10.196.16.3
inet 10.196.16.3/24 scope global qr-aed2634f-efRouting in the virtual router:
aborrero@labtestneutron2002:~ $ sudo ip netns exec qrouter-5712e22e-134a-40d3-a75a-1c9b441717ad ip r default via 10.192.22.1 dev qg-666fcda7-04 10.192.22.0/24 dev qg-666fcda7-04 proto kernel scope link src 10.192.22.4 10.196.16.0/24 dev qr-aed2634f-ef proto kernel scope link src 10.196.16.3 <---- 169.254.0.0/24 dev ha-e7fcaf0d-ec proto kernel scope link src 169.254.0.1 169.254.192.0/18 dev ha-e7fcaf0d-ec proto kernel scope link src 169.254.192.1 172.16.128.0/24 dev qr-21e10025-d4 proto kernel scope link src 172.16.128.1 172.16.129.0/24 dev qg-666fcda7-04 scope link
Not sure if all this setup works. Will test it now.
Comment Actions
My comments from IRC:
you added another subnet
you want to remove teh subnet from that network
add a new network
add the new subnet to it
have the new network tie in at the linuxbridge layer like you have
you want the virtual router to add a new subinterface for a new network
that plugs into the correct subinterface on the host
keep in mind those host subinterfaces are setup in puppet not by neutron
Comment Actions
root@labtestcontrol2003:~# source /root/novaenv.sh root@labtestcontrol2003:~# openstack network list +--------------------------------------+-------------------------+----------------------------------------------------------------------------+ | ID | Name | Subnets | +--------------------------------------+-------------------------+----------------------------------------------------------------------------+ | 57017d7c-3817-429a-8aa3-b028de82cdcc | flattransportb | 5f646219-ce2c-4eb8-8a40-14848c4aab22, 9dd8c6f6-9b58-4a14-a920-72b201c6b325 | | d967e056-efc3-46f2-b75b-c906bb5322dc | HA network tenant admin | 651250de-53ca-4487-97ce-e6f65dc4b8ec | | 05a5494a-184f-4d5c-9e98-77ae61c56daa | flatcloudinstancesb | 27f55952-7698-4253-8e40-72d17761ecf9, 7adfcebe-b3d0-4315-92fe-e8365cc80668 | +--------------------------------------+-------------------------+----------------------------------------------------------------------------+ root@labtestcontrol2003:~# openstack network show 05a5494a-184f-4d5c-9e98-77ae61c56daa | grep cloud | name | flatcloudinstancesb | | provider:physical_network | cloudinstances2b | root@labtestcontrol2003:~#
Comment Actions
Change 455873 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: labtestn: fix bridge mapping
Comment Actions
Change 455873 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: labtestn: fix bridge mapping
Comment Actions
From an instance in labtest (nova-network) I can ping the virtual router in labtestn (neutron) using this setup:
root@labtestcontrol2003:~# neutron net-create compat-net --router:external=true --provider:network_type=flat --provider:physical_network=br-compat --shared
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2018-08-28T17:15:35 |
| description | |
| id | a5e6b90d-57a0-4906-8abe-c270680d7250 |
| ipv4_address_scope | |
| ipv6_address_scope | |
| is_default | False |
| mtu | 1500 |
| name | compat-net |
| port_security_enabled | True |
| provider:network_type | flat |
| provider:physical_network | br-compat |
| provider:segmentation_id | |
| router:external | True |
| shared | True |
| status | ACTIVE |
| subnets | |
| tags | |
| tenant_id | admin |
| updated_at | 2018-08-28T17:15:35 |
+---------------------------+--------------------------------------+
root@labtestcontrol2003:~# neutron subnet-create --gateway 10.196.16.1 --name compat-subnet --ip-version 4 --disable-dhcp --allocation-pool start=10.196.16.3,end=10.196.16.3 compat-net 10.196.16.0/24
Created a new subnet:
+-------------------+------------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------------+
| allocation_pools | {"start": "10.196.16.3", "end": "10.196.16.3"} |
| cidr | 10.196.16.0/24 |
| created_at | 2018-08-28T17:16:51 |
| description | |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 10.196.16.1 |
| host_routes | |
| id | 5f5fc653-f98f-4034-b5a0-51d4ecfa34ca |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | compat-subnet |
| network_id | a5e6b90d-57a0-4906-8abe-c270680d7250 |
| subnetpool_id | |
| tenant_id | admin |
| updated_at | 2018-08-28T17:16:51 |
+-------------------+------------------------------------------------+
root@labtestcontrol2003:~# neutron port-create --name compat-port --fixed-ip subnet_id=compat-subnet,ip_address=10.196.16.3 compat-net
Created a new port:
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| created_at | 2018-08-28T17:18:29 |
| description | |
| device_id | |
| device_owner | |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "5f5fc653-f98f-4034-b5a0-51d4ecfa34ca", "ip_address": "10.196.16.3"} |
| id | 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa |
| mac_address | fa:16:3e:82:43:f8 |
| name | compat-port |
| network_id | a5e6b90d-57a0-4906-8abe-c270680d7250 |
| port_security_enabled | True |
| security_groups | 6167edf8-87a0-457a-935d-7fd7fb6e22b2 |
| status | DOWN |
| tenant_id | admin |
| updated_at | 2018-08-28T17:18:29 |
+-----------------------+------------------------------------------------------------------------------------+
root@labtestcontrol2003:~# neutron router-interface-add cloudinstances2b-gw port=compat-port
Added interface 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa to router cloudinstances2b-gw.
root@labtestcontrol2003:~# neutron port-show 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | labtestneutron2001 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true} |
| binding:vif_type | bridge |
| binding:vnic_type | normal |
| created_at | 2018-08-28T17:18:29 |
| description | |
| device_id | 5712e22e-134a-40d3-a75a-1c9b441717ad |
| device_owner | network:router_interface |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "5f5fc653-f98f-4034-b5a0-51d4ecfa34ca", "ip_address": "10.196.16.3"} |
| id | 9b7a3bed-6b4a-44b6-b4fd-37bee9663eaa |
| mac_address | fa:16:3e:82:43:f8 |
| name | compat-port |
| network_id | a5e6b90d-57a0-4906-8abe-c270680d7250 |
| port_security_enabled | True |
| security_groups | 6167edf8-87a0-457a-935d-7fd7fb6e22b2 |
| status | ACTIVE |
| tenant_id | admin |
| updated_at | 2018-08-28T17:19:14 |
+-----------------------+------------------------------------------------------------------------------------+Comment Actions
Change 456127 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: nova-network: allow extra dnsmasq/dhcp option
Comment Actions
Live testing the above patch, I see clients getting the new static route:
root@aborrero-test4:~# ip r default via 10.196.16.1 dev eth0 10.196.16.0/24 dev eth0 proto kernel scope link src 10.196.16.46 172.16.128.0/24 via 10.196.16.3 dev eth0 root@aborrero-test4:~# traceroute 172.16.128.1 traceroute to 172.16.128.1 (172.16.128.1), 30 hops max, 60 byte packets 1 172.16.128.1 (172.16.128.1) 1.447 ms 1.390 ms 1.376 ms
Comment Actions
And a packet nova-network --> neutron uses the right route:
root@aborrero-test4:~# traceroute 172.16.128.197 traceroute to 172.16.128.197 (172.16.128.197), 30 hops max, 60 byte packets 1 10.196.16.3 (10.196.16.3) 0.518 ms 0.469 ms 0.442 ms 2 172.16.128.197 (172.16.128.197) 0.785 ms 0.763 ms 0.738 ms
Comment Actions
And a packet neutron --> nova-network uses the right route:
root@aborrero-test2:~# ping 10.196.16.46 -c1 PING 10.196.16.46 (10.196.16.46) 56(84) bytes of data. 64 bytes from 10.196.16.46: icmp_seq=1 ttl=63 time=0.739 ms --- 10.196.16.46 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 0.739/0.739/0.739/0.000 ms root@aborrero-test2:~# traceroute -I 10.196.16.46 traceroute to 10.196.16.46 (10.196.16.46), 30 hops max, 60 byte packets 1 172.16.128.1 (172.16.128.1) 0.519 ms 0.515 ms 0.515 ms 2 10.196.16.46 (10.196.16.46) 1.440 ms 1.437 ms 1.435 ms
So we can think on moving this setup to eqiad1/main
Comment Actions
Change 456127 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: nova-network: allow extra dnsmasq/dhcp option
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-08-30T13:07:56Z] <arturo> T202636 internal network routing now exists in labtest/labtestn for VM to communicate with each other
Comment Actions
Note: I would like to explore neutron firewalling options to limit which traffic is able to transit the virtual router (probably only bastion traffic)
Comment Actions
So we'd be blocking all other kinds of traffic between instances sitting on nova-network vs. neutron? Doesn't that mostly defeat the purpose of this?
Comment Actions
I'm not sure this is a good idea (or at least worth doing). Individual projects already have default restrictive firewall rules... it will be baffling to users if the UI-displayed rules aren't the whole story and there's an additional behind-the-scenes filter. Also I'm sure there are a lot of inter-project communication corner cases; having clear routing between regions should at least make those corner cases fixable by users.
Comment Actions
Ok, I will withdraw any special firewalling and will proceed to migrating this setup to main/eqiad1 this week.
@ayounsi could we move forward with this?
Comment Actions
Change 459573 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: eqiad1: add compat network link with main deployment network
Comment Actions
Confirmed.
ayounsi@asw2-b-eqiad# show | compare [edit interfaces interface-range cloud-net-trunk unit 0 family ethernet-switching vlan] - members [ cloud-instances2-b-eqiad cloud-instance-transport1-b-eqiad ]; + members [ cloud-instances2-b-eqiad cloud-instance-transport1-b-eqiad cloud-instances1-b-eqiad ];
To be pushed.
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2018-09-10T17:42:47Z] <XioNoX> trunk cloud-instances1-b-eqiad to cloudnet1003/4:eth1 - T202636
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:22:40Z] <arturo> added myself as project admin while investigating where 10.68.16.2 is assigned and why (T202636)
Comment Actions
I would like to use the address 10.68.16.2 for the virtual router, but that seems assigned.
Will use 10.68.23.253 instead.
root@labcontrol1001:~# nova fixed-ip-get 10.68.23.253 +--------------+---------------+----------+------+----------+ | address | cidr | hostname | host | reserved | +--------------+---------------+----------+------+----------+ | 10.68.23.253 | 10.68.16.0/21 | - | - | False | +--------------+---------------+----------+------+----------+ root@labcontrol1001:~# nova fixed-ip-reserve 10.68.23.253 root@labcontrol1001:~# nova fixed-ip-get 10.68.23.253 +--------------+---------------+----------+------+----------+ | address | cidr | hostname | host | reserved | +--------------+---------------+----------+------+----------+ | 10.68.23.253 | 10.68.16.0/21 | - | - | True | +--------------+---------------+----------+------+----------+
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:33:42Z] <arturo> T202636 manually reserve 10.68.23.253 (in nova-network)
Comment Actions
Change 459573 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: eqiad1: add compat network link with main deployment network
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:36:54Z] <arturo> T202636 creating br-compat bridge in eqiad1 for the compat network
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-11T10:53:28Z] <arturo> T202636 creating all the compat-network configuration in neutron
Comment Actions
This is what I did:
root@cloudcontrol1003:~# neutron net-create compat-net --router:external=true --provider:network_type=flat --provider:physical_network=br-compat --shared
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| availability_zone_hints | |
| availability_zones | |
| created_at | 2018-09-11T10:41:59 |
| description | |
| id | f6905bee-8c2c-4b3c-babe-17273694b4da |
| ipv4_address_scope | |
| ipv6_address_scope | |
| is_default | False |
| mtu | 1500 |
| name | compat-net |
| port_security_enabled | True |
| provider:network_type | flat |
| provider:physical_network | br-compat |
| provider:segmentation_id | |
| router:external | True |
| shared | True |
| status | ACTIVE |
| subnets | |
| tags | |
| tenant_id | admin |
| updated_at | 2018-09-11T10:41:59 |
+---------------------------+--------------------------------------+
root@cloudcontrol1003:~# neutron subnet-create --gateway 10.68.16.1 --name compat-subnet --ip-version 4 --disable-dhcp --allocation-pool start=10.68.23.253,end=10.68.23.253 compat-net 10.68.16.0/21
Created a new subnet:
+-------------------+--------------------------------------------------+
| Field | Value |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.68.23.253", "end": "10.68.23.253"} |
| cidr | 10.68.16.0/21 |
| created_at | 2018-09-11T10:43:50 |
| description | |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | 10.68.16.1 |
| host_routes | |
| id | 7c51532a-167c-4894-b5e6-8dc4861b2247 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | compat-subnet |
| network_id | f6905bee-8c2c-4b3c-babe-17273694b4da |
| subnetpool_id | |
| tenant_id | admin |
| updated_at | 2018-09-11T10:43:50 |
+-------------------+--------------------------------------------------+
root@cloudcontrol1003:~# neutron port-create --name compat-port --fixed-ip subnet_id=compat-subnet,ip_address=10.68.23.253 compat-net
Created a new port:
+-----------------------+-------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | unbound |
| binding:vnic_type | normal |
| created_at | 2018-09-11T10:44:20 |
| description | |
| device_id | |
| device_owner | |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "7c51532a-167c-4894-b5e6-8dc4861b2247", "ip_address": "10.68.23.253"} |
| id | ac5739f5-db3d-4582-8174-0dfc070a67f6 |
| mac_address | fa:16:3e:e1:bf:5d |
| name | compat-port |
| network_id | f6905bee-8c2c-4b3c-babe-17273694b4da |
| port_security_enabled | True |
| security_groups | a7fea49f-047b-4cf5-a434-d4b213dd0c4f |
| status | DOWN |
| tenant_id | admin |
| updated_at | 2018-09-11T10:44:20 |
+-----------------------+-------------------------------------------------------------------------------------+
root@cloudcontrol1003:~# neutron router-interface-add cloudinstances2b-gw port=compat-port
Added interface ac5739f5-db3d-4582-8174-0dfc070a67f6 to router cloudinstances2b-gw.At this point I restarted the neutron-linuxbridge-agent service in both cloudnet1003 and cloudnet1004 because I was seeing log messages like these:
2018-09-11 10:44:45.989 36764 ERROR neutron.plugins.ml2.managers [req-3f984a39-c85a-40e3-afb1-c24f0309b29f - - - - -] Failed to bind port ac5739f5-db3d-4582-8174-0dfc070a67f6 on host cloudnet1004 for vnic_type normal using segments [{'segmentation_id': None, 'physical_network': u'br-compat', 'id': u'136161dc-86f6-47a0-82b2-593b9e46d743', 'network_type': u'flat'}]
2018-09-11 10:44:46.002 36764 ERROR neutron.plugins.ml2.managers [req-3f984a39-c85a-40e3-afb1-c24f0309b29f - - - - -] Failed to bind port ac5739f5-db3d-4582-8174-0dfc070a67f6 on host cloudnet1004 for vnic_type normal using segments [{'segmentation_id': None, 'physical_network': u'br-compat', 'id': u'136161dc-86f6-47a0-82b2-593b9e46d743', 'network_type': u'flat'}]root@cloudcontrol1003:~# neutron port-show compat-port
+-----------------------+-------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | cloudnet1004 |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | binding_failed |
| binding:vnic_type | normal |
| created_at | 2018-09-11T10:44:20 |
| description | |
| device_id | d93771ba-2711-4f88-804a-8df6fd03978a |
| device_owner | network:router_interface |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "7c51532a-167c-4894-b5e6-8dc4861b2247", "ip_address": "10.68.23.253"} |
| id | ac5739f5-db3d-4582-8174-0dfc070a67f6 |
| mac_address | fa:16:3e:e1:bf:5d |
| name | compat-port |
| network_id | f6905bee-8c2c-4b3c-babe-17273694b4da |
| port_security_enabled | True |
| security_groups | a7fea49f-047b-4cf5-a434-d4b213dd0c4f |
| status | DOWN |
| tenant_id | admin |
| updated_at | 2018-09-11T10:44:45 |
+-----------------------+-------------------------------------------------------------------------------------+After the restart, all seems to work:
root@cloudnet1004:~# ip netns exec qrouter-d93771ba-2711-4f88-804a-8df6fd03978a ip a | grep 10.68.23
inet 10.68.23.253/21 scope global qr-ac5739f5-db
root@cloudnet1004:~# brctl show br-compat
bridge name bridge id STP enabled interfaces
br-compat 8000.1a1fbd837351 no eth1.1102
tapac5739f5-db
root@cloudcontrol1003:~# neutron port-show compat-port
+-----------------------+-------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+-------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | cloudnet1004 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true} |
| binding:vif_type | bridge |
| binding:vnic_type | normal |
| created_at | 2018-09-11T10:44:20 |
| description | |
| device_id | d93771ba-2711-4f88-804a-8df6fd03978a |
| device_owner | network:router_interface |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "7c51532a-167c-4894-b5e6-8dc4861b2247", "ip_address": "10.68.23.253"} |
| id | ac5739f5-db3d-4582-8174-0dfc070a67f6 |
| mac_address | fa:16:3e:e1:bf:5d |
| name | compat-port |
| network_id | f6905bee-8c2c-4b3c-babe-17273694b4da |
| port_security_enabled | True |
| security_groups | a7fea49f-047b-4cf5-a434-d4b213dd0c4f |
| status | ACTIVE |
| tenant_id | admin |
| updated_at | 2018-09-11T10:51:07 |
+-----------------------+-------------------------------------------------------------------------------------+So, this should all be working fine now.
In any case DHCP clients need a new DHCP lease to know the new static route.
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-11T16:08:17Z] <arturo> restarted nova-network after killing all dnsmasq procs in labnet1001 for T202636
Comment Actions
Change 459816 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: main: don't offer the static route in DHCP for the compat network
Comment Actions
Change 459816 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: main: don't offer the static route in DHCP for the compat network
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-11T16:52:14Z] <arturo> again, restarted nova-network after killing all dnsmasq procs in labnet1001 for T202636
Comment Actions
Update on state of things.
When setting the classless-static-route extra-option in the dnsmasq config in labnet1001.eqiad.wmnet (for the main deployment), DHCP clients (VMs) in this deployment can't no longer be reached (after they negotiate DHCP with this new config activated).
This is a big contrast with the labtest deployment, in which this setup worked right away.
After much debugging, I tracked down the issue to DHCP clients don't setting their default gateway when the classless-static-route extra-option is sent by the DHCP server.
The RFC3442, which defines the classless-static-route extra-option we are using reads:
DHCP Client Behavior [...] If the DHCP server returns both a Classless Static Routes option and a Router option, the DHCP client MUST ignore the Router option.
(https://tools.ietf.org/html/rfc3442)
I'm not sure why that behavior exists, but I'm afraid we should stick to RFCs behaviors.
It seems the workaround/solution is to push 2 routes to DHCP clients using classless-static-route options (default + compat network).
How is possible this setup worked in labtest?
Comment Actions
This doesn't work in labtest either!
What I tested in labtest was to renew DHCP leases. There is a different behavior when renewing and when requesting a brand new lease (which happens at boot/reboot).
I'm convinced we can solve this by simply pushing 2 static routes: the default route and the compat route.
Comment Actions
Change 459998 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudvps: nova-network: also distribute default route in dnsmasq
Comment Actions
Change 459998 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudvps: nova-network: also distribute default route in dnsmasq
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-12T11:20:39Z] <arturo> T202636 distributing default routes using classless-static-route for all VMs in main/labtest (dnsmasq/nova-network)
Comment Actions
My initial tests rebooting machines in both labtest/main (remember, different DHCP query than a release) shows this is now working!
Comment Actions
Mentioned in SAL (#wikimedia-cloud) [2018-09-12T11:22:10Z] <arturo> rebooting bastion-restricted-01 to test new setup from T202636
Comment Actions
This is working, but I note that the originating IP is getting natted when I go between regions:
root@bastion-restricted-eqiad1-01:~# tcpdump -n port 4444
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:32:10.664549 IP 208.80.155.255.49432 > 172.16.1.135.4444: Flags [S], seq 2261175948, win 29200, options [mss 1460,sackOK,TS val 805828756 ecr 0,nop,wscale 9], length 0
21:32:10.664637 IP 172.16.1.135.4444 > 208.80.155.255.49432: Flags [R.], seq 0, ack 2261175949, win 0, length 0
That's traffic coming in from 10.68.17.149 in the eqiad region -- it shows as originating from 208.80.155.255. If getting that to work properly is hard then maybe we can live with it, but it's potentially confusing when people are setting up firewalls.
(The good news is that the originating IP is correct when coming from a VM with a public ip -- which the bastions have. So this is moot for most use cases.)
Comment Actions
Using NAT means the eqiad-->eqiad1 traffic is reaching the router (i.e, nova-network @ labnet1001), which means the static route is not being applied.
I confirmed this by jumping into the machine:
aborrero@util-abogott-stretch:~$ ip r default via 10.68.16.1 dev eth0 10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 10.68.16.0/21 dev eth0 proto kernel scope link src 10.68.17.149
we don't have there our custom static route, which probably means this machine didn't ask for DHCP in a while.
Forcing it:
aborrero@util-abogott-stretch:~$ sudo ifup --force eth0 Internet Systems Consortium DHCP Client 4.3.5 Copyright 2004-2016 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ Listening on LPF/eth0/fa:16:3e:62:0e:94 Sending on LPF/eth0/fa:16:3e:62:0e:94 Sending on Socket/fallback DHCPREQUEST of 10.68.17.149 on eth0 to 255.255.255.255 port 67 DHCPACK of 10.68.17.149 from 10.68.16.1 RTNETLINK answers: File exists bound to 10.68.17.149 -- renewal in 17457 seconds. Sending network state change signal to nslcd...done. aborrero@util-abogott-stretch:~$ ip r default via 10.68.16.1 dev eth0 10.0.3.0/24 dev lxcbr0 proto kernel scope link src 10.0.3.1 linkdown 10.68.16.0/21 dev eth0 proto kernel scope link src 10.68.17.149 172.16.0.0/21 via 10.68.23.253 dev eth0
Now, the tcpdump from the bastion-resctricted-eqiad1-01 POV:
before: 09:38:32.303135 IP 208.80.155.255 > 172.16.1.135: ICMP echo request, id 10798, seq 52, length 64 09:38:32.303234 IP 172.16.1.135 > 208.80.155.255: ICMP echo reply, id 10798, seq 52, length 64 after: 09:40:15.799446 IP 10.68.17.149 > 172.16.1.135: ICMP echo request, id 10923, seq 1, length 64 09:40:15.799542 IP 172.16.1.135 > 10.68.17.149: ICMP echo reply, id 10923, seq 1, length 64
Conclusion: if a machine is not properly using the compat network, let's force a new DHCP renew.
Comment Actions
@arturo thanks! I'm pondering whether or not we should just do that cloud-wide... maybe just the bastions and the proxies.
Comment Actions
There is logic in the NAT controller to exclude certain destination addresses from NAT (IIUC) and potentially dding the 10. range for eqiad instances would allow preservation of source addresses here?