Page MenuHomePhabricator

wmcs-wikireplica-dns: error: circular reference detected
Closed, ResolvedPublic

Description

When running the wmcs-wikireplica-dns script in cloudcontrol1003, we see:

root@cloudcontrol1003:~# wmcs-wikireplica-dns 
2019-05-21T12:49:51Z mwopenstackclients.DnsManager WARNING : Creating s6.{u'status': u'ACTIVE', u'masters': [], u'name': u'analytics.db.svc.eqiad.wmflabs.', u'links': {u'self': u'http://cloudservices1003.wikimedia.org:9001/v2/zones/04c45c1f-214d-450b-a733-028dcdc87a12'}, u'transferred_at': None, u'created_at': u'2017-09-15T23:16:54.000000', u'pool_id': u'794ccc2c-d751-44fe-b57f-8894c9f5c842', u'updated_at': u'2019-04-23T20:55:06.000000', u'email': u'root@wmflabs.org', u'version': 2864, u'ttl': 60, u'action': u'NONE', u'attributes': {}, u'serial': 1556052260, u'project_id': u'noauth-project', u'type': u'PRIMARY', u'id': u'04c45c1f-214d-450b-a733-028dcdc87a12', u'description': u'long running wiki replica queries'}
2019-05-21T12:49:51Z root         WARNING : Error response from http://cloudservices1003.wikimedia.org:9001/v2/zones/04c45c1f-214d-450b-a733-028dcdc87a12/recordsets:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 206, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
    response = req.get_response(self._app)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_middleware/base.py", line 114, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/oslo_middleware/base.py", line 114, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/designate/api/middleware.py", line 262, in __call__
    return self._handle_exception(request, e, status, response)
  File "/usr/lib/python2.7/dist-packages/designate/api/middleware.py", line 312, in _handle_exception
    response=json.dumps(response))
  File "/usr/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 178, in dumps
    return json.dumps(obj, default=default, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected

Traceback (most recent call last):
  File "/usr/local/sbin/wmcs-wikireplica-dns", line 146, in <module>
    main()
  File "/usr/local/sbin/wmcs-wikireplica-dns", line 117, in main
    dns.ensure_recordset(zone_id, fqdn, 'A', ips)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 329, in ensure_recordset
    rs = self.create_recordset(zone, name, type_, records)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 305, in create_recordset
    r = self._post('/v2/zones/{}/recordsets'.format(uuid), data=data)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 232, in _post
    return self._req('POST', *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 224, in _req
    r.raise_for_status()
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 893, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://cloudservices1003.wikimedia.org:9001/v2/zones/04c45c1f-214d-450b-a733-028dcdc87a12/recordsets

The most recent patch merged to that script has been: https://gerrit.wikimedia.org/r/c/operations/puppet/+/506019 by @bd808

Event Timeline

aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.

For the record, I discovered this while working on T223992: toolforge: redis issue.

HTTP 5xx from the Designate API... Can we get the full error message from the logs on the server?

This just blocked T224062
That's very bad because the server is running out of space. I got a bunch of room cleared up, but it will again in due time.

Change 512425 had a related patch set uploaded (by Bstorm; owner: Bstorm):
[operations/puppet@production] wmcs: stop rewriting the zone variable for cnames

https://gerrit.wikimedia.org/r/512425

Change 512425 merged by Bstorm:
[operations/puppet@production] wmcs: stop rewriting the zone variable for cnames

https://gerrit.wikimedia.org/r/512425

Bstorm claimed this task.

And that last one seems to have done the trick.

I've tested and modern Designate appears to do better error handling of this:

krenair@labs-t224000-alex-osdev:~/devstack$ cat test_mwoc.py
import mwopenstackclients
dns_mgr = mwopenstackclients.DnsManager(mwopenstackclients.Clients(
    username="admin",
    password="password",
    url="http://labs-t224000-alex-osdev/identity/v3",
    project="demo"
), None)
dns_mgr.create_recordset('2bc35734-0fd6-4b04-83dd-88e1672dbfb6', "asd.{u'status': u'ACTIVE'}.alextest.wmflabs.org.", 'A', ['127.0.0.1'])
#dns_mgr.create_recordset('2bc35734-0fd6-4b04-83dd-88e1672dbfb6', "asd.abcstatusdefugACTIVEhi.alextest.wmflabs.org.", 'A', ['127.0.0.1'])

krenair@labs-t224000-alex-osdev:~/devstack$ python test_mwoc.py
/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py:224: UserWarning: Using keystoneclient sessions has been deprecated. Please update your software to use keystoneauth1.
  warnings.warn('Using keystoneclient sessions has been deprecated. '
WARNING:root:Error response from http://172.16.6.108:9001//v2/zones/2bc35734-0fd6-4b04-83dd-88e1672dbfb6/recordsets:
{"message": "Provided object is not valid. Got a ValueError error with message Host name asd.{u'status': u'ACTIVE'}.alextest.wmflabs.org. is not match", "code": 400, "type": "invalid_object", "request_id": "req-eb0fee59-0469-44d9-94e9-4e5831dd5a20"}
Traceback (most recent call last):
  File "test_mwoc.py", line 8, in <module>
    dns_mgr.create_recordset('2bc35734-0fd6-4b04-83dd-88e1672dbfb6', "asd.{u'status': u'ACTIVE'}.alextest.wmflabs.org.", 'A', ['127.0.0.1'])
  File "/home/krenair/devstack/mwopenstackclients.py", line 305, in create_recordset
    r = self._post('/v2/zones/{}/recordsets'.format(uuid), data=data)
  File "/home/krenair/devstack/mwopenstackclients.py", line 232, in _post
    return self._req('POST', *args, **kwargs)
  File "/home/krenair/devstack/mwopenstackclients.py", line 224, in _req
    r.raise_for_status()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: BAD REQUEST for url: http://172.16.6.108:9001//v2/zones/2bc35734-0fd6-4b04-83dd-88e1672dbfb6/recordsets

or alternatively for the example that was failing here, the same but about it being too long instead:
{"message": "Provided object is not valid. Got a ValueError error with message Value too long for name", "code": 400, "type": "invalid_object", "request_id": "req-8ea4b0c3-9eac-41d1-b0ae-1c048fad8e33"

designateclient catches this stuff too, but that's T224708: Drop most of mwopenstackclients.DnsManager in favour of designateclient

@Krenair Can I safely delete the labs-t224000-alex-osdev.openstack VM now?