Page MenuHomePhabricator
Paste P6770

Nova Conductor logs for db1009 overload
ActivePublic
Actions

Authored by • madhuvishy on Mar 1 2018, 7:25 AM.
Tags
None
Referenced Files
F14162424: Nova Conductor logs for db1009 overload
Mar 1 2018, 7:25 AM
Subscribers
None
# I see this error pop up a couple hours before the db usage spiked once
```
2018-03-01 03:54:36.489 2143 INFO nova.db.sqlalchemy.api [req-7bed5cd8-5a0a-4431-89a5-f02094f02532 nodepoolmanager contintcloud - - -] quota_usages out of sync, updating. project_id: contintcloud, user_id: None, resource: fixed_ips, tracked usage: 15, actual usage: 11
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", line 88, in _report_state
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db service.service_ref.save()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 250, in save
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 153, in service_update
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 525, in service_update
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db service_ref = _service_get(context, service_id, session=session)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 431, in _service_get
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db result = query.first()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2445, in first
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db ret = list(self[0:1])
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2281, in __getitem__
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return list(res)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in _execute_and_instances
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db close_with_result=True)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in _connection_from_session
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db **kw)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in connection
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db execution_options=execution_options)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 887, in _connection_for_bind
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db engine, execution_options)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None),
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db e, dialect, self)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return fn()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db rec = pool._do_get()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db self._dec_overflow()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return self._create_connection()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return _ConnectionRecord(self)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db self.connection = self.__connect()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db connection = self.__pool._invoke_creator(self)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs)
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 626, in __init__
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db self._connect()
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 818, in _connect
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db 2003, "Can't connect to MySQL server on %r (%s)" % (self.host, e))
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'm5-master.eqiad.wmnet' ((1040, u'Too many connections'))")
2018-03-01 03:54:43.626 2139 ERROR nova.servicegroup.drivers.db
2018-03-01 03:54:53.477 2139 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2018-03-01 03:55:43.423 2143 INFO nova.db.sqlalchemy.api [req-df9befc0-a0d3-4ec2-915c-5eeda4c8ca28 osstackcanary admin-monitoring - - -] quota_usages out of sync, updating. project_id: admin-monitoring, user_id: None, resource: fixed_ips, tracked usage: 3, actual usage: 2
```
# In a couple hours
```
2018-03-01 05:59:56.833 2140 INFO nova.db.sqlalchemy.api [req-2aadba58-06d5-4723-9e38-403dec017629 nodepoolmanager contintcloud - - -] quota_usages out of sync, updating. project_id: contintcloud, user_id: None, resource: fixed_ips, tracked usage: 18, actual usage: 15
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db [-] Unexpected error while reporting service status
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db Traceback (most recent call last):
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", line 88, in _report_state
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db service.service_ref.save()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 250, in save
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 153, in service_update
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 146, in wrapper
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 136, in wrapper
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 525, in service_update
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db service_ref = _service_get(context, service_id, session=session)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 431, in _service_get
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db result = query.first()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2445, in first
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db ret = list(self[0:1])
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2281, in __getitem__
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return list(res)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in _execute_and_instances
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db close_with_result=True)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in _connection_from_session
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db **kw)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in connection
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db execution_options=execution_options)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 887, in _connection_for_bind
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db engine, execution_options)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None),
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db e, dialect, self)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return fn()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db rec = pool._do_get()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db self._dec_overflow()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return self._create_connection()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return _ConnectionRecord(self)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db self.connection = self.__connect()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db connection = self.__pool._invoke_creator(self)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs)
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 626, in __init__
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db self._connect()
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 818, in _connect
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db 2003, "Can't connect to MySQL server on %r (%s)" % (self.host, e))
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'm5-master.eqiad.wmnet' ((1040, u'Too many connections'))")
2018-03-01 06:00:31.183 2140 ERROR nova.servicegroup.drivers.db
2018-03-01 06:00:41.074 2140 INFO nova.servicegroup.drivers.db [-] Recovered from being unable to report status.
2018-03-01 06:00:49.714 2136 INFO nova.db.sqlalchemy.api [req-2f860f2c-cca7-44ef-8301-efe6b8d07b62 nodepoolmanager contintcloud - - -] quota_usages out of sync, updating. project_id: contintcloud, user_id: None, resource: fixed_ips, tracked usage: 15, actual usage: 12
2018-03-01 06:02:57.315 2142 INFO nova.db.sqlalchemy.api [req-2aadba58-06d5-4723-9e38-403dec017629 nodepoolmanager contintcloud - - -] quota_usages out of sync, updating. project_id: contintcloud, user_id: None, resource: fixed_ips, tracked usage: 12, actual usage: 11
```
# And this keeps repeating until we restarted it at 6:57
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL