I believe this alert is spurious. I suspect this cron is running on the inactive node and causing alarms:
# Puppet Name: drain and log rabbit notifications.error queue 35 * * * * /usr/local/sbin/drain_queue notifications.error >> /var/log/rabbitmq/notifications_error.log 2>&1
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/puppet | production | +41 -14 | openstack: only run rabbitmq cleanup on active control node |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T294906 Puppet Improvements 2021/2022 | |||
| Open | jbond | T265138 Work required to prepare for puppet 6 | |||
| Open | None | T273673 replace all puppet crons with systemd timers | |||
| Open | None | T132324 Tracking and Reducing cron-spam to root@ | |||
| Resolved | • chasemp | T183144 labcontrol1002 Error: unable to connect to node rabbit@labcontrol1002: nodedown |
Change 398900 had a related patch set uploaded (by Rush; owner: cpettet):
[operations/puppet@production] openstack: only run rabbitmq cleanup on active control node
Change 398900 merged by Rush:
[operations/puppet@production] openstack: only run rabbitmq cleanup on active control node
Notice: /Stage[main]/Rabbitmq::Cleanup/Cron[drain and log rabbit notifications.error queue]/ensure: removed Notice: Finished catalog run in 11.22 seconds root@labcontrol1002