Page MenuHomePhabricator
Create Task
Maniphest T318816

Test, understand, improve rabbitmq HA
Closed, ResolvedPublic
Actions

Description

We have three dedicated rabbitmq nodes, and they're configured as a shared cluster with all the recommended settings for HA.

Nevertheless, HA only barely works. Often when a single rabbitmq node goes down a bunch of nova services get stuck forever pining for the missing node rather than failing over. If we restart all three nodes gradually in series, a bunch of messages get lost even though the whole idea of a rabbitmq cluster is that the messages are mirrored persistent.

This needs a deep dive with test cases and research to figure out what's going on and how to stabilize things.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Revert "oslo_messaging_rabbit: increase retry and backoff by a lot"operations/puppetproduction+2 -2
oslo_messaging_rabbit: kombu_reconnect_delay=0.1operations/puppetproduction+1 -0
oslo_messaging_rabbit: increase retry and backoff by a lotoperations/puppetproduction+2 -2
OpenStack: use rabbitmq Quorum queuesoperations/puppetproduction+1 -3
Openstack config: move oslo_messaging_rabbit into a shared templateoperations/puppetproduction+31 -76
Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Sep 28 2022, 2:31 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 28 2022, 2:31 PM
dcaro added a project: User-dcaro.Sep 28 2022, 2:36 PM
Andrew added a comment.Oct 3 2022, 7:24 PM

I am able to reproduce this issue in codfw1dev and have not found an obvious cause. The next steps should/could be:

  • Switch to using quorum queues, which generally seem better designed than the current mirroring behavior
  • Upgrade to Openstack version Y. There are quite a few changes in the Oslo code between X ad Y and this issue /might/ be handled there.
• aborrero added a subtask: T320232: nova-fullstack leaking VMs: rabbitmq may have problems.Oct 7 2022, 10:11 AM
fnegri changed the status of subtask T320232: nova-fullstack leaking VMs: rabbitmq may have problems from Open to In Progress.Oct 7 2022, 1:41 PM
fnegri closed subtask T320232: nova-fullstack leaking VMs: rabbitmq may have problems as Resolved.Oct 7 2022, 3:40 PM
Andrew added a comment.Nov 29 2022, 3:45 AM

With Yoga packages I'm unable to reproduce this issue.

I may try moving us to quorum queues anyway since that's more future-proof.

gerritbot added a comment.Nov 29 2022, 4:58 PM

Change 861890 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Openstack config: move oslo_messaging_rabbit into a shared template

https://gerrit.wikimedia.org/r/861890

gerritbot added a project: Patch-For-Review.Nov 29 2022, 4:58 PM
gerritbot added a comment.Nov 29 2022, 11:43 PM

Change 861890 merged by Andrew Bogott:

[operations/puppet@production] Openstack config: move oslo_messaging_rabbit into a shared template

https://gerrit.wikimedia.org/r/861890

Maintenance_bot removed a project: Patch-For-Review.Nov 30 2022, 12:30 AM
Andrew claimed this task.Nov 30 2022, 7:21 PM
gerritbot added a comment.Nov 30 2022, 7:30 PM

Change 862323 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] OpenStack: use rabbitmq Quorum queues

https://gerrit.wikimedia.org/r/862323

gerritbot added a project: Patch-For-Review.Nov 30 2022, 7:30 PM
gerritbot added a comment.Nov 30 2022, 7:33 PM

Change 862323 merged by Andrew Bogott:

[operations/puppet@production] OpenStack: use rabbitmq Quorum queues

https://gerrit.wikimedia.org/r/862323

Stashbot added a comment.Nov 30 2022, 8:03 PM

Mentioned in SAL (#wikimedia-cloud) [2022-11-30T20:03:11Z] <andrewbogott> changing all rabbitmq queues to quorum queues. Will be noisy! T318816

Maintenance_bot removed a project: Patch-For-Review.Nov 30 2022, 8:30 PM
gerritbot added a comment.Dec 1 2022, 12:19 AM

Change 862389 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] oslo_messaging_rabbit: increase retry and backoff by a lot

https://gerrit.wikimedia.org/r/862389

gerritbot added a project: Patch-For-Review.Dec 1 2022, 12:19 AM
gerritbot added a comment.Dec 1 2022, 12:21 AM

Change 862389 merged by Andrew Bogott:

[operations/puppet@production] oslo_messaging_rabbit: increase retry and backoff by a lot

https://gerrit.wikimedia.org/r/862389

Maintenance_bot removed a project: Patch-For-Review.Dec 1 2022, 12:30 AM
Andrew added a comment.Dec 2 2022, 4:13 AM

There's an upstream bug in the interaction between oslo-messaging and kombu -- Oslo throws an additional exception that almost always prevents kombu from failing over to a different backend.

I'm pretty sure this issue is:

https://bugs.launchpad.net/oslo.messaging/+bug/1993149

...which seems to have a pending fix (although at the moment it's a blind revert of a previous patch).

For my future reference: the code that /should/ cause the failover is in kombu.connection:

self.maybe_switch_next()  # select next host

That line is never executed because the errback call above calls oslo_messaging/_drivers/impl_rabbit.py _recoverable_error_callback which raises an exception at

timer.check_return(_raise_timeout)
gerritbot added a comment.Dec 2 2022, 4:17 AM

Change 863090 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] Revert "oslo_messaging_rabbit: increase retry and backoff by a lot"

https://gerrit.wikimedia.org/r/863090

gerritbot added a project: Patch-For-Review.Dec 2 2022, 4:17 AM
gerritbot added a comment.Dec 5 2022, 2:25 AM

Change 864321 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] oslo_messaging_rabbit: kombu_reconnect_delay=0.1

https://gerrit.wikimedia.org/r/864321

gerritbot added a comment.Dec 5 2022, 2:36 PM

Change 864321 merged by Andrew Bogott:

[operations/puppet@production] oslo_messaging_rabbit: kombu_reconnect_delay=0.1

https://gerrit.wikimedia.org/r/864321

gerritbot added a comment.Dec 5 2022, 2:41 PM

Change 863090 merged by Andrew Bogott:

[operations/puppet@production] Revert "oslo_messaging_rabbit: increase retry and backoff by a lot"

https://gerrit.wikimedia.org/r/863090

Andrew closed this task as Resolved.Dec 6 2022, 6:04 PM

I'm still chasing down the upstream fix for this but the above patches should resolve the issue for us.

Andrew closed subtask T324208: Unexpected pages during rabbitmq outage as Declined.Jul 24 2024, 2:10 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 24 2024, 2:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits