Mar 21 19:30:01 cloudcontrol1003 systemd[1]: Starting Designate Floating IP PTR records updater... Mar 21 19:30:01 cloudcontrol1003 systemd[1]: Started Designate Floating IP PTR records updater. Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: Traceback (most recent call last): Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: File "/etc/dns-floating-ip-updater.py", line 90, in <module> Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: for server in nova_client.servers.list(): Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: File "/usr/lib/python2.7/dist-packages/novaclient/v2/servers.py", line 750, in list Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: "servers") Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 242, in _list Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: resp, body = self.api.client.get(url) Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 173, in get Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: return self.request(url, 'GET', **kwargs) Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 94, in request Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: raise exceptions.from_response(resp, body, url, method) Mar 21 19:32:37 cloudcontrol1003 dns-floating-ip-updater.py[37033]: novaclient.exceptions.ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-3aac256f-7e57-4b3a-b4b2-c5a873f035bb) Mar 21 19:32:37 cloudcontrol1003 systemd[1]: designate_floating_ip_ptr_records_updater.service: main process exited, code=exited, status=1/FAILURE Mar 21 19:32:37 cloudcontrol1003 systemd[1]: Unit designate_floating_ip_ptr_records_updater.service entered failed state.
Description
Description
Event Timeline
Comment Actions
The time is not exactly the same so I can't guarantee there's any relation but here are the logs.
cloudservice1003 /var/log/designate/designate-sink.log:
2019-03-21 19:30:59.379 1324 WARNING nova_fixed_multi.novamulti [-] NovaFixedHandler creating records - compute.instance.create.end
2019-03-21 19:30:59.474 1324 WARNING nova_fixed_multi.base [req-57c98486-8432-40fe-815d-0b5b75703927 - - - - -] Creating reverse record in 6990e139-49e6-466c-9421-46cf45f05842 / 5b234015-dd00-425f-9389-59f293d4f59
3 with values {'managed_plugin_type': 'handler', 'managed_plugin_name': 'nova_fixed_multi', 'managed_resource_id': u'd5104194-2135-4746-9e43-1ebc9c17501c', 'managed_resource_type': 'instance', 'managed': True, 'da
ta': u'fullstackd-1553196647.admin-monitoring.eqiad.wmflabs.'}
2019-03-21 19:30:59.635 1324 WARNING nova_fixed_multi.base [req-57c98486-8432-40fe-815d-0b5b75703927 - - - - -] Creating record in 114f1333-c2c1-44d3-beb4-ebed1a91742b / e255ae2c-e9ac-4765-a081-717f998442e6 with v
alues {'managed_plugin_type': 'handler', 'managed_plugin_name': 'nova_fixed_multi', 'managed_resource_id': u'd5104194-2135-4746-9e43-1ebc9c17501c', 'managed_resource_type': 'instance', 'managed': True, 'data': u'1
72.16.0.250'}
2019-03-21 19:30:59.833 1324 WARNING nova_fixed_multi.base [req-57c98486-8432-40fe-815d-0b5b75703927 - - - - -] Creating record in 114f1333-c2c1-44d3-beb4-ebed1a91742b / 199778a0-1b50-4293-a1ec-0243d5f18044 with v
alues {'managed_plugin_type': 'handler', 'managed_plugin_name': 'nova_fixed_multi', 'managed_resource_id': u'd5104194-2135-4746-9e43-1ebc9c17501c', 'managed_resource_type': 'instance', 'managed': True, 'data': u'1
72.16.0.250'}
2019-03-21 19:33:50.835 1324 WARNING wmf_sink.base [req-4f888f56-e524-4b33-8d4c-7925e43f2cbc - - - - -] Caught exception when deleting proxy records
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base Traceback (most recent call last):
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/wmf_sink/base.py", line 113, in _delete
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base self._delete_proxies_for_ip(data['project_name'], ip)
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/wmf_sink/base.py", line 144, in _delete_proxies_for_ip
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base project_proxies = self._get_proxy_list_for_project(project)
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/wmf_sink/base.py", line 193, in _get_proxy_list_for_project
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base resp = requests.get(requrl + '/mapping')
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/requests/api.py", line 70, in get
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base return request('get', url, params=params, **kwargs)
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/requests/api.py", line 56, in request
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base return session.request(method=method, url=url, **kwargs)
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 488, in request
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base resp = self.send(prep, **send_kwargs)
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 609, in send
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base r = adapter.send(request, **kwargs)
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 487, in send
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base raise ConnectionError(e, request=request)
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.base ConnectionError: HTTPConnectionPool(host='proxy-eqiad.wmflabs.org', port=5668): Max retries exceeded with url: /dynamicproxy-api/v1/admin-monitoring/mapping (Caused
by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fdaedded250>: Failed to establish a new connection: [Errno -2] Name or service not known',))
2019-03-21 19:33:50.835 1324 ERROR wmf_sink.baseComment Actions
Mentioned in SAL (#wikimedia-cloud) [2019-03-21T19:58:54Z] <Krenair> Set proxy-eqiad.wmflabs.org to CNAME to proxy-eqiad1.wmflabs.org T218938
Comment Actions
That was partially my fault in https://phabricator.wikimedia.org/T218633#5037322 though I have no idea how it worked before as the old proxy IP no longer appeared to function at the time I was doing that. It should be better now?