It seems like the prometheus setup for the aux cluster is missing bits like DNS, LVS, grafana datasource:
➔ rgrep k8s-dse *
hieradata/common/kubernetes.yaml: name: k8s-dse
hieradata/role/common/prometheus.yaml: - prometheus@k8s-dse
modules/profile/files/grafana/production-datasources.yaml: - name: eqiad prometheus/k8s-dse
modules/profile/files/grafana/production-datasources.yaml: url: http://prometheus.svc.eqiad.wmnet/k8s-dse
modules/profile/files/rsyslog/lookup_table_output.json: {"index" : "prometheus@k8s-dse", "value" : "kafka local ecs_170" },
modules/prometheus/files/provision-fs.sh:provision k8s-dse 50g
```➔ rgrep k8s-aux *
hieradata/common/kubernetes.yaml: name: k8s-aux
modules/prometheus/files/provision-fs.sh:provision k8s-aux 50g