Page MenuHomePhabricator
Create Task
Maniphest T356621

[nova-api,cloudrabbit] Connectivity issues from all cloudcontrols to all cloudrabbit nodes
Open, HighPublic
Actions

Description

We are seeing connectivity issues from all of the cloudcontrols to all of the cloudrabbit nodes.

This seems to still work, but the services keep reconnecting.

Some logs:

root@cloudcontrol1006:~# journalctl -n 100 -u nova-api
...
Feb 05 10:00:02 cloudcontrol1006 nova-api-wsgi[902532]: 2024-02-05 10:00:02.420 902532 ERROR oslo.messaging._drivers.impl_rabbit [-] [01586216-7dd4-4b9a-b924-d706dcf3a8a5] AMQP server on rabbitmq03.eqiad1.wikimediacloud.org:5671 is unreachable: Server unexpectedly closed connection. Trying again in 0 seconds.: OSError: Server unexpectedly closed connection
Feb 05 10:00:02 cloudcontrol1006 nova-api-wsgi[902532]: 2024-02-05 10:00:02.555 902532 INFO oslo.messaging._drivers.impl_rabbit [-] [01586216-7dd4-4b9a-b924-d706dcf3a8a5] Reconnected to AMQP server on rabbitmq03.eqiad1.wikimediacloud.org:5671 via [amqp] client with port 34844.
Feb 05 10:00:27 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:00:27.710 985534 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: EOF occurred in violation of protocol (_ssl.c:2393)
Feb 05 10:00:27 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:00:27.713 985534 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: EOF occurred in violation of protocol (_ssl.c:2393)
Feb 05 10:01:15 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:01:15.894 985534 ERROR oslo.messaging._drivers.impl_rabbit [-] [8ea3e25f-35ca-42a1-83a3-31c95bf8e76d] AMQP server on rabbitmq01.eqiad1.wikimediacloud.org:5671 is unreachable: EOF occurred in violation of protocol (_ssl.c:2393). Trying again in 0 seconds.: ssl.SSLEOFError: EOF occurred in violation of protocol >
Feb 05 10:01:16 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:01:16.027 985534 INFO oslo.messaging._drivers.impl_rabbit [-] [8ea3e25f-35ca-42a1-83a3-31c95bf8e76d] Reconnected to AMQP server on rabbitmq01.eqiad1.wikimediacloud.org:5671 via [amqp] client with port 46626.
Feb 05 10:05:02 cloudcontrol1006 nova-api-wsgi[902532]: 2024-02-05 10:05:02.592 902532 ERROR oslo.messaging._drivers.impl_rabbit [-] [01586216-7dd4-4b9a-b924-d706dcf3a8a5] AMQP server on rabbitmq03.eqiad1.wikimediacloud.org:5671 is unreachable: EOF occurred in violation of protocol (_ssl.c:2393). Trying again in 0 seconds.: ssl.SSLEOFError: EOF occurred in violation of protocol >
Feb 05 10:05:02 cloudcontrol1006 nova-api-wsgi[902532]: 2024-02-05 10:05:02.750 902532 INFO oslo.messaging._drivers.impl_rabbit [-] [01586216-7dd4-4b9a-b924-d706dcf3a8a5] Reconnected to AMQP server on rabbitmq03.eqiad1.wikimediacloud.org:5671 via [amqp] client with port 52616.
Feb 05 10:07:50 cloudcontrol1006 nova-api-wsgi[902532]: 2024-02-05 10:07:50.794 902532 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: EOF occurred in violation of protocol (_ssl.c:2393)
Feb 05 10:10:01 cloudcontrol1006 nova-api-wsgi[1341495]: 2024-02-05 10:10:01.279 1341495 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: EOF occurred in violation of protocol (_ssl.c:2393)
Feb 05 10:21:31 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:21:31.344 985534 ERROR oslo.messaging._drivers.impl_rabbit [-] [8ea3e25f-35ca-42a1-83a3-31c95bf8e76d] AMQP server on rabbitmq01.eqiad1.wikimediacloud.org:5671 is unreachable: Too many heartbeats missed. Trying again in 0 seconds.: amqp.exceptions.ConnectionForced: Too many heartbeats missed
Feb 05 10:21:31 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:21:31.345 985534 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: EOF occurred in violation of protocol (_ssl.c:2393)
Feb 05 10:21:31 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:21:31.348 985534 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: EOF occurred in violation of protocol (_ssl.c:2393)
Feb 05 10:21:31 cloudcontrol1006 nova-api-wsgi[985534]: 2024-02-05 10:21:31.470 985534 INFO oslo.messaging._drivers.impl_rabbit [-] [8ea3e25f-35ca-42a1-83a3-31c95bf8e76d] Reconnected to AMQP server on rabbitmq01.eqiad1.wikimediacloud.org:5671 via [amqp] client with port 54352.
Feb 05 10:23:01 cloudcontrol1006 nova-api-wsgi[1341495]: 2024-02-05 10:23:01.282 1341495 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: EOF occurred in violation of protocol (_ssl.c:2393)

After getting a pcap of the connection flow, I can see many retransmitted packages and duplicated ACKs:

Screenshot from 2024-02-05 11-32-02.png (431×1 px, 173 KB)

There's several bumps on the nova-api response times, though might not be related:

image.png (410×1 px, 89 KB)

Details

SubjectRepoBranchLines +/-
rabbitmq: increase heartbeat timeout and number of heartbeatsoperations/puppetproduction+7 -2
Customize query in gerrit

Event Timeline

dcaro triaged this task as High priority.Feb 5 2024, 10:36 AM
dcaro created this task.
dcaro updated the task description. (Show Details)Feb 5 2024, 10:37 AM
aborrero added a project: User-aborrero.Feb 5 2024, 10:41 AM
aborrero subscribed.
aborrero unsubscribed.
aborrero subscribed.
dcaro added a comment.Feb 5 2024, 11:41 AM

The retransmitted packages are seen by both ends (cloudcontrol and cloudrabbit), and on both ends the package is from the other end.

So it seems that somehow the switch is duplicating packages.

taavi added a project: Cloud-VPS.Feb 6 2024, 12:08 PM
dcaro removed projects: Cloud-VPS, User-aborrero.Feb 6 2024, 1:27 PM

I think that the duplicated packages come from capturing on different interfaces (vlan1152/1154 and ens3f0np0), as the packages have to go through both, looking

gerritbot added a comment.Feb 6 2024, 5:18 PM

Change 997921 had a related patch set uploaded (by Andrew Bogott; author: Andrew Bogott):

[operations/puppet@production] rabbitmq: increase heartbeat timeout and number of heartbeats

https://gerrit.wikimedia.org/r/997921

gerritbot added a project: Patch-For-Review.Feb 6 2024, 5:18 PM
dcaro updated the task description. (Show Details)Feb 6 2024, 5:21 PM
gerritbot added a comment.Feb 6 2024, 5:24 PM

Change 997921 merged by Andrew Bogott:

[operations/puppet@production] rabbitmq: increase heartbeat timeout and number of heartbeats

https://gerrit.wikimedia.org/r/997921

Maintenance_bot removed a project: Patch-For-Review.Feb 6 2024, 5:31 PM
dcaro added a comment.Feb 7 2024, 1:51 PM

Unfortunately this did not seem to help :/

Now we get '180s' on the timeout messages, but still hapenning quite often

fnegri edited projects, added cloud-services-team (FY2024/2025-Q1-Q2); removed cloud-services-team (FY2023/2024-Q3-Q4).Aug 19 2024, 9:19 AM
taavi added a project: Cloud-VPS.Sep 28 2024, 12:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits