During a recent incident, one of the issues that came up (though not an old issue), was that we had no immediate visibility of where each of our services in discovery are served from. That information is easily obtainable via confctl, when everything is onfire, a human consuming the following information is not easy:
{"codfw": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=eventgate-analytics-external"}
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=kartotherian"}
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=parsoid-php"}
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=mw-web"}
{"codfw": {"pooled": true, "references": [], "ttl": 10}, "tags": "dnsdisc=api-ro"}
{"codfw": {"pooled": true, "references": [], "ttl": 10}, "tags": "dnsdisc=appservers-ro"}
{"codfw": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=echostore"}
{"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=mw-api-ext"}
{"codfw": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=eventstreams-internal"}
{"codfw": {"pooled": true, "references": [], "ttl": 300}, "tags": "dnsdisc=recommendation-api"}
<snip>Moreover, the above data does not allow us to know if a service is active-active or active passive.
One solution to this could be polling confd for that information and representing them on grafana,