While investigating T309182: Evaluate 'pint' for Prometheus alerts pint flagged a problem with poolcounter alerts:
pint_problem{filename="/srv/alerts/ops/team-sre_poolcounter.yaml", instance="localhost:9123", job="pint", kind="alerting", name="PoolcounterFullQueues", problem="rate() should only be used with counters but "poolcounter_full_queues" is a gauge according to metrics metadata from prometheus "ops" at http://127.0.0.1:9900/ops", prometheus="ops", reporter="promql/rate", severity="bug", site="codfw"}And indeed poolcounter_full_queues in our deployment is a gauge now, it was turned into a counter here: https://github.com/Wikia/poolcounter-prometheus-exporter/commit/f7bc8a2e13dff8e94bb6531cd523d92e9c0621d2
Updating our version of poolcounter-prometheus-exporter will fix the problem, cc @akosiaris what do you think ?