Page MenuHomePhabricator

openstack-nova running out of database connections
Open, HighPublic

Description

This morning I responded to a full-stack alert; every few hours nova has been failing to schedule new VM creations. This appears to be a result of us running out of DB connections. From nova-conductor logs:

2019-11-03 14:40:17.386 27023 ERROR oslo_messaging.rpc.server OperationalError: (_mysql_exceptions.OperationalError) (1226, "User 'nova' has exceeded the 'max_user_connections' resource (current value: 100)")

I addressed this issue last time by reducing the number of worker nodes (for several services) with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/541407/

That improved things quite a bit, but apparently we still don't have enough headroom. We might be leaking connections, but more likely we just need to raise the limit.

Event Timeline

Andrew created this task.Nov 3 2019, 3:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 3 2019, 3:21 PM
Andrew triaged this task as High priority.Nov 3 2019, 3:21 PM
Paladox added a subscriber: Paladox.Nov 3 2019, 3:42 PM