As evidenced in https://phabricator.wikimedia.org/T371885 there are pods which can't be scraped by prometheus k8s, either because of misconfiguration or actual problems.
Therefore we should be alerting on pods that are not available for scraping, i.e. the JobUnavailable equivalent we have in production, but for pods.
The query is 1 - (count by (kubernetes_namespace, app, prometheus) (up{app!=""} == 0) / count by (kubernetes_namespace, app,prometheus) (up)) < 0.8 https://w.wiki/DW5V
As of March 20th the list looks like this:
{app="mediawiki", kubernetes_namespace="mw-videoscaler", prometheus="k8s"} 0.0916
{app="thumbor", kubernetes_namespace="thumbor", prometheus="k8s"} 0.1999
{app="developer-portal", kubernetes_namespace="developer-portal", prometheus="k8s"} 0.3333
{app="api-gateway", kubernetes_namespace="rest-gateway", prometheus="k8s"} 0.5
{app="mcrouter", kubernetes_namespace="mw-mcrouter", prometheus="k8s"} 0.5
{app="toolhub", kubernetes_namespace="toolhub", prometheus="k8s"} 0.6
{app="thumbor", kubernetes_namespace="thumbor", prometheus="k8s-staging"} 0.4
{app="api-gateway", kubernetes_namespace="rest-gateway", prometheus="k8s-staging"} 0.5
{app="toolhub", kubernetes_namespace="toolhub", prometheus="k8s-staging"} 0.6
{app="spark-history", kubernetes_namespace="spark-history", prometheus="k8s-dse"} 0
{app="spark-history", kubernetes_namespace="spark-history-test", prometheus="k8s-dse"} 0
{app="airflow", kubernetes_namespace="airflow-platform-eng", prometheus="k8s-dse"} 0.25
{app="airflow", kubernetes_namespace="airflow-analytics-product", prometheus="k8s-dse"} 0.3000
{app="airflow", kubernetes_namespace="airflow-analytics-test", prometheus="k8s-dse"} 0.3000
{app="airflow", kubernetes_namespace="airflow-main", prometheus="k8s-dse"} 0.3000
{app="airflow", kubernetes_namespace="airflow-research", prometheus="k8s-dse"} 0.3000
{app="airflow", kubernetes_namespace="airflow-search", prometheus="k8s-dse"} 0.3000
{app="airflow", kubernetes_namespace="airflow-test-k8s", prometheus="k8s-dse"} 0.3000
{app="airflow", kubernetes_namespace="airflow-wmde", prometheus="k8s-dse"} 0.3000
{app="airflow", kubernetes_namespace="airflow-ml", prometheus="k8s-dse"} 0.3333
{app="mpic", kubernetes_namespace="mpic", prometheus="k8s-dse"} 0.6666
{app="mpic", kubernetes_namespace="mpic-next", prometheus="k8s-dse"} 0.6666
{app="ores-legacy", kubernetes_namespace="ores-legacy", prometheus="k8s-mlserve"} 0.3333
{app="recommendation-api-ng", kubernetes_namespace="recommendation-api-ng", prometheus="k8s-mlserve"} 0.3333
{app="net-istio-controller", kubernetes_namespace="knative-serving", prometheus="k8s-mlserve"} 0.5
{app="ores-legacy", kubernetes_namespace="ores-legacy", prometheus="k8s-mlstaging"} 0.3333
{app="recommendation-api-ng", kubernetes_namespace="recommendation-api-ng", prometheus="k8s-mlstaging"} 0.3333
{app="net-istio-controller", kubernetes_namespace="knative-serving", prometheus="k8s-mlstaging"} 0.5
{app="developer-portal", kubernetes_namespace="developer-portal", prometheus="k8s-staging"} 0.3333