Background
Although DNS resolution latency issue in T422455 was not the direct result of load on coredns, the specific way that resolution will typically (i.e., without dot suffixing) fan out to a sequence of search-path-suffixed queries means that, in the presence of an issue affecting a fraction of coredns queries, the probability of a given resolution being impacted is increased.
Proposal
We should look for low-hanging fruit where we can avoid these often-unnecessary (i.e., in the case of an external name) queries (one example solution again being dot-suffixing). Of particular interest are queries that are high volume, critical, or unlikely to be cached.
Two potential examples (T422455#11808708): non-mesh services referenced in wmf-config/ProductionServices.php and envoy upstream cluster configuration rendered by the mesh.configuration module.