Page MenuHomePhabricator

wmfsink designate handler not running during vm deletes
Closed, ResolvedPublic

Description

The wmfsink handler needs to be updated with the new designate keystone authtoken format.

2019-10-09 19:58:49.183 6279 ERROR oslo_messaging.notify.dispatcher     return wrapped(*args, **kwargs)
2019-10-09 19:58:49.183 6279 ERROR oslo_messaging.notify.dispatcher   File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 420, in request
2019-10-09 19:58:49.183 6279 ERROR oslo_messaging.notify.dispatcher     raise exceptions.from_response(resp, method, url)
2019-10-09 19:58:49.183 6279 ERROR oslo_messaging.notify.dispatcher BadRequest: Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-a4a9fc08-0394-4d86- 9617-41acc347c0ac)

The last successful run was on Fri Sep 27. This has unfortunately left DNS records and puppet certificates that need to be cleaned up.

Event Timeline

Change 541913 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: update designate wmfsink handler for newton

https://gerrit.wikimedia.org/r/541913

Change 541913 merged by Jhedden:
[operations/puppet@production] openstack: update designate wmfsink handler for newton

https://gerrit.wikimedia.org/r/541913

Mentioned in SAL (#wikimedia-cloud) [2019-10-09T20:53:26Z] <jeh> restarting designate-sink in eqiad1 with updated wmfsink handler T235127

Mentioned in SAL (#wikimedia-cloud) [2019-10-09T21:33:19Z] <jeh> cleanup designate dns leaks in eqiad1 T235127

Mentioned in SAL (#wikimedia-cloud) [2019-10-10T13:32:11Z] <jeh> resuming cleanup of designate dns leaks in eqiad1 T235127