Right now those messages just pile up forever.
On RabbitMQ Grafana the queue notifications.error has messages pilling up and they are apparently not being consumed:
Description
Description
Details
Details
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| rabbitmq: drain and log notifications.error hourly | operations/puppet | production | +9 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Andrew | T170492 figure out if nodepool is overwhelming rabbitmq and/or nova | |||
| Resolved | Andrew | T175029 rabbitmq: Consume and log messages sent to notifications.error |
Event Timeline
Comment Actions
OpenStack SearchLight looks like a simple brick to consume the notifications and push them to ElasticSearch:
https://docs.openstack.org/searchlight/latest/index.html
https://wiki.openstack.org/wiki/Searchlight
https://www.youtube.com/watch?v=ExzULavwvNQ
It comes with a RabbitMQ consumer, an API and an Horizon plugin for the UI.
Comment Actions
Change 381450 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] rabbitmq: drain and log notifications.error hourly
Comment Actions
Change 381450 merged by Andrew Bogott:
[operations/puppet@production] rabbitmq: drain and log notifications.error hourly