When running the wmcs-wikireplica-dns script in cloudcontrol1003, we see:
root@cloudcontrol1003:~# wmcs-wikireplica-dns 2019-05-21T12:49:51Z mwopenstackclients.DnsManager WARNING : Creating s6.{u'status': u'ACTIVE', u'masters': [], u'name': u'analytics.db.svc.eqiad.wmflabs.', u'links': {u'self': u'http://cloudservices1003.wikimedia.org:9001/v2/zones/04c45c1f-214d-450b-a733-028dcdc87a12'}, u'transferred_at': None, u'created_at': u'2017-09-15T23:16:54.000000', u'pool_id': u'794ccc2c-d751-44fe-b57f-8894c9f5c842', u'updated_at': u'2019-04-23T20:55:06.000000', u'email': u'root@wmflabs.org', u'version': 2864, u'ttl': 60, u'action': u'NONE', u'attributes': {}, u'serial': 1556052260, u'project_id': u'noauth-project', u'type': u'PRIMARY', u'id': u'04c45c1f-214d-450b-a733-028dcdc87a12', u'description': u'long running wiki replica queries'} 2019-05-21T12:49:51Z root WARNING : Error response from http://cloudservices1003.wikimedia.org:9001/v2/zones/04c45c1f-214d-450b-a733-028dcdc87a12/recordsets: Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response result = self.application(self.environ, start_response) File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 206, in __call__ return app(environ, start_response) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__ response = req.get_response(self._app) File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send application, catch_exc_info=False) File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/oslo_middleware/base.py", line 114, in __call__ response = req.get_response(self.application) File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send application, catch_exc_info=False) File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/oslo_middleware/base.py", line 114, in __call__ response = req.get_response(self.application) File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1299, in send application, catch_exc_info=False) File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application app_iter = application(self.environ, start_response) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__ resp = self.call_func(req, *args, **self.kwargs) File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func return self.func(req, *args, **kwargs) File "/usr/lib/python2.7/dist-packages/designate/api/middleware.py", line 262, in __call__ return self._handle_exception(request, e, status, response) File "/usr/lib/python2.7/dist-packages/designate/api/middleware.py", line 312, in _handle_exception response=json.dumps(response)) File "/usr/lib/python2.7/dist-packages/oslo_serialization/jsonutils.py", line 178, in dumps return json.dumps(obj, default=default, **kwargs) File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps sort_keys=sort_keys, **kw).encode(obj) File "/usr/lib/python2.7/json/encoder.py", line 207, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode return _iterencode(o, 0) ValueError: Circular reference detected Traceback (most recent call last): File "/usr/local/sbin/wmcs-wikireplica-dns", line 146, in <module> main() File "/usr/local/sbin/wmcs-wikireplica-dns", line 117, in main dns.ensure_recordset(zone_id, fqdn, 'A', ips) File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 329, in ensure_recordset rs = self.create_recordset(zone, name, type_, records) File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 305, in create_recordset r = self._post('/v2/zones/{}/recordsets'.format(uuid), data=data) File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 232, in _post return self._req('POST', *args, **kwargs) File "/usr/lib/python2.7/dist-packages/mwopenstackclients.py", line 224, in _req r.raise_for_status() File "/usr/lib/python2.7/dist-packages/requests/models.py", line 893, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http://cloudservices1003.wikimedia.org:9001/v2/zones/04c45c1f-214d-450b-a733-028dcdc87a12/recordsets
The most recent patch merged to that script has been: https://gerrit.wikimedia.org/r/c/operations/puppet/+/506019 by @bd808