Page MenuHomePhabricator
Create Task
Maniphest T397783

Rabbitmq, neutron-openvswitch-agent, and network outages
Closed, InvalidPublic
Actions

Description

Just now I did some routine-ish rabbitmq maintenance (rebooting hosts in turn) and that caused the neutron-openvswitch-agents to crash.

That's a neutron bug, as those agents should fail over between non-working and working rabbitmq backends; and also they should just wait even if they can't reach one.

But, anyway, when those workers crashed, the cloud-vps VMs on associated cloudvirts fell off the internet.

Up until today I believed that openstack services were only involved in changing state and not in maintaining active workloads. Is that untrue of neutron-openvswitch-agent? Is it an active router which touches every packet? If so, we will need to change several of our maintenance practices.

Details

Other Assignee
taavi

Related Objects

Event Timeline

Andrew created this task.Jun 24 2025, 9:23 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 24 2025, 9:23 PM
Andrew triaged this task as High priority.Jun 24 2025, 9:23 PM
bd808 added subscribers: • aborrero, bd808.Jun 24 2025, 10:06 PM

Something I noticed linked from https://wikitech.wikimedia.org/wiki/Incidents/2024-11-26_WMCS_network_problems when I searched Wikitech for notes on neutron-openvswitch-agent:

T380972: openstack: prevent puppet from restarting neutron-openvswitch-agent

In T380972#10361884, @aborrero wrote:
In T380972#10361862, @Andrew wrote:

Is there any theory about why restarting openvswitch-agent is more delicate than restarting the old linuxbridge agent?

I'm in favor of avoiding outages, but because the agent runs in many places (cloudvirts), decoupling it from puppet can result in agent state being out of sync with config which also seems bad.

my current theory is that the linuxbridge agent was stateless, whereas openvswitch is stateful.

Andrew closed this task as Invalid.Aug 28 2025, 7:48 PM

I stopped tried stopping neutron-openvswitch-agent on all cloudvirts in codfw1dev and that did not interrupt my ssh connection to a codfw1dev VM.

Then I restarted it in cloudvirts and stopped in cloudnet nodes; this also didn't interrupt the VM.

Whatever caused this outage wasnt't just neutron-openvswitch-agent going down.l

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits