This task is for migrating the eqiad1 Neutron L3 routers to the OVS agent. This will cause some brief service outages, so it will need to be performed in a pre-announced maintenance window.
These should be done far in advance:
- Prepare Puppet patches for the planned operations below, get them reviewed
These should be done just before the announced maintenance window:
- Take the inactive cloudnet out of service (disable Puppet, stop Neutron services)
- Reimage that host as a spare (this will need to use the no-firewall spare class)
- Delete the Linuxbridge agent for the now-spare host from Neutron (openstack network agent delete)
At this point there is one spare host, and one active Linuxbridge host. Rollback plan at this point is to apply the previous role to the now-spare host.
Wait for announced maintenance window to begin.
- Make the now-spare host as an OVS-based cloudnet in Puppet
- Immediately disable Puppet and the Neutron services on the OVS host
- sudo systemctl stop neutron-metadata-agent.service neutron-dhcp-agent.service neutron-l3-agent.service neutron-openvswitch-agent.service
At this point there is one active Linuxbridge host, and one inactive OVS host. Rollback plan at this point is to delete the new OVS agent from the Neutron database, and to reimage the now-OVS host back to a Linuxbridge host.
The following steps will cause some minor downtime (~30 sec in codfw1dev):
- Disable Puppet and stop Neutron services on the active Linuxbridge host
- sudo systemctl stop neutron-metadata-agent.service neutron-dhcp-agent.service neutron-l3-agent.service neutron-linuxbridge-agent.service
- Update Neutron ports associated with any routers to use the OVS VIF type in the database (T358761#9780138)
- sudo mariadb -u root neutron
- SELECT * FROM ml2_port_bindings WHERE port_id IN (SELECT port_id FROM routerports WHERE router_id IN (SELECT id FROM routers))\G
- UPDATE ml2_port_bindings SET vif_type = 'ovs' WHERE port_id IN (SELECT port_id FROM routerports WHERE router_id IN (SELECT id FROM routers));
- to rollback: UPDATE ml2_port_bindings SET vif_type = 'bridge' WHERE port_id IN (SELECT port_id FROM routerports WHERE router_id IN (SELECT id FROM routers));
- Start Neutron services on the new OVS host
- sudo systemctl start neutron-metadata-agent.service neutron-dhcp-agent.service neutron-l3-agent.service neutron-openvswitch-agent.service
At this point there is one active OVS host and one inactive Linuxbridge host. Rollback plan at this point is to follow this section section but in the opposite direction (replace OVS with Linuxbridge and the other way around in the instructions).
- Reimage old Linuxbridge host as a spare
- Delete the old Linuxbridge agent from the Neutron database
- Apply the OVS cloudnet role to the spare host
At this point there is one active OVS host and one standby OVS host. Rollback plan at this point is to follow this entire procedure but in the opposite direction.