Page MenuHomePhabricator

Cannot create web proxy because of Duplicate RecordSet
Closed, ResolvedPublic

Description

In the wikicommunityhealth, we have two VPS instances.

I would like to add a Web Proxy configuration to direct https://wikicommunityhealth.wmcloud.org to one of those servers.

However, I am getting a Duplicate RecordSet when trying to add the web proxy.

Event Timeline

The dns entry is there:

04:33 PM ~/Work/wikimedia/analytics_quarry_web  (master|✔)
dcaro@vulcanus$ dig +short wikicommunityhealth.wmcloud.org
185.15.56.49

04:39 PM ~/Work/wikimedia/analytics_quarry_web  (master|✔)
dcaro@vulcanus$ dig +short -x 185.15.56.49
instance-proxy-01.project-proxy.wmflabs.org.

Checked on the proxy side, but there's no entry for it:

root@proxy-01:~# redis-cli
127.0.0.1:6379> keys *
...
127.0.0.1:6379> get frontend:wikicommunityhealth.wmcloud.org
(nil)

It does not show on the DNS->proxies page on horizon though, so probably a stale DNS entry somewhere.
Looking

dcaro triaged this task as Medium priority.Aug 16 2021, 2:56 PM

I can't see the recordset on the horizon ui or in the cli:

root@cloudcontrol1003:~# wmcs-openstack recordset list --os-project-id=cloudinfra wmcloud.org. | grep wikicommunityhealth
| cbcf4738-33b6-424a-8343-4128db8d744d | wikicommunityhealth-dev.wmcloud.org.                   | A     | 185.15.56.49                                                                              | ACTIVE | NONE   |

(that's the dev one, not the one being added).
looking

@Andrew I'm seeing the entry in the dns, and on designate logs, but openstack seems not to have it in it's database, the docs point to a 'designate' cli that seems not available anymore, any guidance on how to continue? (I can doodle with the DB, but before messing with the internals of anything I'd like your input). Thanks!

Hi @dcaro, @Andrew,

is there anything we can do to help with this? Any info we can provide?
Let us know and thanks in advance.

Running sudo wmcs-openstack zone list --all-projects to cast a really wide net seems to suggest a reason:
<snip>

| 8bf50f05-a5ed-46ce-9f92-5dfb48c5ae3a | wikicommunityhealth            | wikicommunityhealth.eqiad1.wmcloud.org.                 | PRIMARY | 1604510307 | ACTIVE | NONE   |
| 7347b52a-0599-4d4d-a843-d07f3d8677bd | wikicommunityhealth            | svc.wikicommunityhealth.eqiad1.wikimedia.cloud.         | PRIMARY | 1604510318 | ACTIVE | NONE   |
| ac589c17-806e-4db4-9712-0f9fa702d5cd | wikicommunityhealth            | wikicommunityhealth.wmcloud.org.                        | PRIMARY | 1617716287 | ACTIVE | NONE   |
| 0f4d3e65-c9be-4515-ac2b-bccb2c57c6f4 | wikicommunityhealth            | wikicommunityhealth.wmflabs.org.                        | PRIMARY | 1604510341 | ACTIVE | NONE   |

</snip>
That's the zone name because it's the project. So it has a stub record with that name. @bd808 remembered dealing with that problem before here T260388 There's probably doc updates needed around this so we can reference it easier when it comes up again.

So this is in that recordset: | 9170b14a-7fa2-44a4-ab2b-af25f913f04b | wikicommunityhealth.wmcloud.org. | A | 185.15.56.49 | ACTIVE | NONE |

I'm getting that using sudo wmcs-openstack --os-project-id wikicommunityhealth recordset list ac589c17-806e-4db4-9712-0f9fa702d5cd. Going to poke a little more to see what the disconnect is with what is showing up for other people.

Here is the record that @Bstorm found shown in the Horizon UI:

Screen Shot 2021-08-19 at 10.56.57 AM.png (1×1 px, 253 KB)

I used the UI tools there to delete this record. After I did that I was able to create a new proxy with the desired name:

Screen Shot 2021-08-19 at 10.58.50 AM.png (1×1 px, 204 KB)

I have deleted that new proxy as well to clean up after my test. @CristianCantoro, you should now be able to login to Horizon and create the proxy with the correct settings.

My sense making about what happened here is that the proxy was partially created by having it's DNS record setup but not getting the proper state data about that pushed into the wmcloud.org front proxy configuration. If you read the code that actually does this work you can see that the DNS record is created first and then the config change is sent to the front proxy for storage. There is an error handler block for the update to the config, but when it detects a failure it does not roll back the DNS record. This feels like a thing that could be improved.