Openstack: designate API issues
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	aborrero
	Oct 18 2019, 12:45 PM

Description

Horizon seems to have issues connecting to the designate endpoints. The following was registered in the labweb1001/1002 logs when trying to create/modify DNS records in horizon:

2019-10-18 12:32:41.842098 HTTP exception with no status/code
2019-10-18 12:32:41.842244 Traceback (most recent call last):
2019-10-18 12:32:41.842255   File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/openstack_dashboard/api/rest/utils.py", line 125, in _wrapped
2019-10-18 12:32:41.842263     data = function(self, request, *args, **kw)
2019-10-18 12:32:41.842270   File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/designatedashboard/api/rest/passthrough.py", line 105, in post
2019-10-18 12:32:41.842277     return passthrough_post(path, request, data).json()
2019-10-18 12:32:41.842284   File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/designatedashboard/api/rest/passthrough.py", line 58, in _passthrough_request
2019-10-18 12:32:41.842292     response.raise_for_status()
2019-10-18 12:32:41.842298   File "/srv/deployment/horizon/venv/lib/python3.5/site-packages/requests/models.py", line 893, in raise_for_status
2019-10-18 12:32:41.842305     raise HTTPError(http_error_msg, response=self)
2019-10-18 12:32:41.842315 requests.exceptions.HTTPError: 403 Client Error: FORBIDDEN for url: http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones/
2019-10-18 12:32:41.842329

Also, the CLI utils shows some issues, but this may be unrelated:

root@cloudcontrol1004:~# openstack zone list --all-projects
Unexpected exception for http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones?: Header value True must be of type str or bytes, not <type 'bool'>

Details

	Subject	Repo	Branch	Lines +/-
	openstack: patch python-designateclient header values	operations/puppet	production	+158 -0

Customize query in gerrit

Related Objects

Mentioned Here: T235846: wikimedia.cloud: setup new domain

Event Timeline

aborrero created this task.Oct 18 2019, 12:45 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 18 2019, 12:45 PM

aborrero triaged this task as High priority.Oct 18 2019, 12:45 PM

aborrero moved this task from Inbox to Soon! on the cloud-services-team (Kanban) board.

@aborrero can you reproduce this now? I just ran some tests (in the 'testlabs' project) and things seemed ok.

root@cloudcontrol1004:~# openstack zone list --all-projects
Unexpected exception for http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones?: Header value True must be of type str or bytes, not <type 'bool'>

During the Newton upgrade the python-requests package was upgraded which changed the way headers are handled [0], python-designateclient was patched [1] a few years ago but it unfortunately didn't make it into the debian package.

This only happens when using the all-projects or all-tenants flag.

[0] https://github.com/psf/requests/issues/3491
[1] https://review.opendev.org/#/c/390965/

In T235863#5586981, @Andrew wrote:

@aborrero can you reproduce this now? I just ran some tests (in the 'testlabs' project) and things seemed ok.

I was just able to create a new zone for T235846: wikimedia.cloud: setup new domain. Whatever you did, it worked!

In T235863#5586991, @JHedden wrote:
root@cloudcontrol1004:~# openstack zone list --all-projects
Unexpected exception for http://openstack.eqiad1.wikimediacloud.org:9001/v2/zones?: Header value True must be of type str or bytes, not <type 'bool'>
During the Newton upgrade the python-requests package was upgraded which changed the way headers are handled [0], python-designateclient was patched [1] a few years ago but it unfortunately didn't make it into the debian package.

This only happens when using the all-projects or all-tenants flag.

[0] https://github.com/psf/requests/issues/3491
[1] https://review.opendev.org/#/c/390965/

The patch seems simple enough that perhaps we can do the patching ourselves? What do you think?

--- a/designateclient/v2/client.py
+++ b/designateclient/v2/client.py
@@ -58,7 +58,7 @@ class DesignateAdapter(adapter.LegacyJsonAdapter):
         if self.all_projects:
             kwargs['headers'].setdefault(
                 'X-Auth-All-Projects',
-                self.all_projects
+                str(self.all_projects)
             )
 
         if self.edit_managed:

In T235863#5587207, @aborrero wrote:

The patch seems simple enough that perhaps we can do the patching ourselves? What do you think?

--- a/designateclient/v2/client.py
+++ b/designateclient/v2/client.py
@@ -58,7 +58,7 @@ class DesignateAdapter(adapter.LegacyJsonAdapter):
         if self.all_projects:
             kwargs['headers'].setdefault(
                 'X-Auth-All-Projects',
-                self.all_projects
+                str(self.all_projects)
             )
 
         if self.edit_managed:

I think that's a good idea. What's our best approach to adding a patch to the python-designateclient=2.3.0-2 package? I was looking at using dpkg-source --commit but I'm not sure how we should increment the version.

In T235863#5587266, @JHedden wrote:
In T235863#5587207, @aborrero wrote:
The patch seems simple enough that perhaps we can do the patching ourselves? What do you think?
--- a/designateclient/v2/client.py
+++ b/designateclient/v2/client.py
@@ -58,7 +58,7 @@ class DesignateAdapter(adapter.LegacyJsonAdapter):
         if self.all_projects:
             kwargs['headers'].setdefault(
                 'X-Auth-All-Projects',
-                self.all_projects
+                str(self.all_projects)
             )
 
         if self.edit_managed:
I think that's a good idea. What's our best approach to adding a patch to the python-designateclient=2.3.0-2 package? I was looking at using dpkg-source --commit but I'm not sure how we should increment the version.

Instead of patching the .deb package, rebuilding, uploading to the repo, etc, the usual shortcut is usually to just overwrite the file by using puppet. That's what we do for our custom neutron l3 agent hacks. Not very elegant, but it works..

Change 545072 had a related patch set uploaded (by Jhedden; owner: Jhedden):
[operations/puppet@production] openstack: patch python-designateclient header values

https://gerrit.wikimedia.org/r/545072

gerritbot added a project: Patch-For-Review.Oct 21 2019, 10:08 PM

Change 545072 merged by Jhedden:
[operations/puppet@production] openstack: patch python-designateclient header values