I think possibly @Andrew just fixed this but maybe it wouldn't have effected historical artifacts.

I dumped a list of all the known proxies using a script I have on silver. A grep of that dump doesn't show any matches for discovery-beta. The dump matches the list shown at https://tools.wmflabs.org/openstack-browser/project/shiny-r at the moment:

• discovery-experimental.wmflabs.org
• sentimentalk.wmflabs.org
• discovery-dev.wmflabs.org

I just tried it again and I'm still getting the same error. I also tried it after logging out and logging back in (might as well try that, right?) but also to no success.

Update: tried it again this morning (in case giving it more time to reset the behind-the-scenes cache or whatever might help) but also to no success.

Mentioned in SAL (#wikimedia-cloud) [2017-06-16T19:30:14Z] <bd808> Added self (BryanDavis) as project admin to debug T167985

The only log message on the server is:

[Fri Jun 16 19:32:16.106226 2017] [:error] [pid 1715] Recoverable error: Duplicate RecordSet

I got this by trying as myself to create the "discovery-beta" proxy pointing to discovery-testing:3838.

This is looking like it might be an error coming from Designate (the OpenStack DNS management layer). See similar message in an upstream report.

Our dns server has records for both discovery-beta and discovery that point to the Labs proxy ip:

$ dig discovery-beta.wmflabs.org

; <<>> DiG 9.8.3-P1 <<>> discovery-beta.wmflabs.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42708
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;discovery-beta.wmflabs.org.    IN      A

;; ANSWER SECTION:
discovery-beta.wmflabs.org. 86399 IN    A       208.80.155.156

;; Query time: 98 msec
;; SERVER: 192.168.11.1#53(192.168.11.1)
;; WHEN: Fri Jun 16 14:02:56 2017
;; MSG SIZE  rcvd: 60

$ dig discovery.wmflabs.org

; <<>> DiG 9.8.3-P1 <<>> discovery.wmflabs.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19458
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;discovery.wmflabs.org.         IN      A

;; ANSWER SECTION:
discovery.wmflabs.org.  41735   IN      A       208.80.155.156

;; Query time: 33 msec
;; SERVER: 192.168.11.1#53(192.168.11.1)
;; WHEN: Fri Jun 16 14:04:41 2017
;; MSG SIZE  rcvd: 55

The proxy however has no record, so these were leaked from prior use and now horizon can't see them properly.

I'm in the process of making a more comprehensive fix for this issue, but in the meantime I've tried a one-off fix... @mpopov, does it work now?