We currently rely on a rr-dns entry to connect to services running in Kubernetes clusters.
While this does work in general it needs manual configuration in DNS and it is different from how traffic reaches clusters in production.
With upcoming ingress, we should add an LVS service k8s-ingress-staging pointing to the istio-ingressgateway on staging kubernetes nodes (as we would do it in production).
That LVS service should be active/passive (eqiad being active by default) to retain the functionality of the "hidden" staging cluster used by SRE's to test infrastructure changes/updates etc.
LVS is now setup up and the staging clusters ingressgateway can be reached via k8s-ingress-staging.discovery.wmnet.
miscweb, being the only service currently enabled, can be accessed by with proper SNI:
curl -I --resolve "miscweb.staging.discovery.wmnet:30443:$(dig +short k8s-ingress-staging.discovery.wmnet)" 'https://miscweb.staging.discovery.wmnet:30443'
As next step, I would like to add a wildcard record to DNS (like *.k8s-staging.discovery.wmnet, name is just a proposal and what I currently used on the ingress side - easy to change that, though) pointing to whatever k8s-ingress-staging.discovery.wmnet currently points to. This would make setting up new services in staging clusters completely self-service (after the initial k8s user/namespace creation by SRE).
All that only applies to staging ofc. Production services would still need an entry in services.yaml for monitoring and individual depooling etc.