Page MenuHomePhabricator

openstack-nova running out of database connections
Open, HighPublic

Description

This morning I responded to a full-stack alert; every few hours nova has been failing to schedule new VM creations. This appears to be a result of us running out of DB connections. From nova-conductor logs:

2019-11-03 14:40:17.386 27023 ERROR oslo_messaging.rpc.server OperationalError: (_mysql_exceptions.OperationalError) (1226, "User 'nova' has exceeded the 'max_user_connections' resource (current value: 100)")

I addressed this issue last time by reducing the number of worker nodes (for several services) with https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/541407/

That improved things quite a bit, but apparently we still don't have enough headroom. We might be leaking connections, but more likely we just need to raise the limit.

Event Timeline

Andrew created this task.Sun, Nov 3, 3:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSun, Nov 3, 3:21 PM
Andrew triaged this task as High priority.Sun, Nov 3, 3:21 PM
Paladox added a subscriber: Paladox.Sun, Nov 3, 3:42 PM