Follow-up for T222112 and https://wikitech.wikimedia.org/wiki/Incident_documentation/20190425-prometheus.
bbrazil says we might be helped by upgrading to >= 2.9.2, citing "various improvements to the [tsdb] postings logic" made between 2.7.2 and then.
+1 to testing/PoC 2.9.2; we're using Debian Prometheus packages mostly verbatim, but adding back the k8s discovery + dependencies back as they are not shipped in Debian. For testing purposes we can even simply use upstream's binary tho.
I took the change of Bullseye upcoming upgrade to build a Prometheus 2.24.1 + k8s package in the wmf/bullseye in the operations/debs/prometheus repo, the package works fine on Buster too so we can use it to upgrade Prometheus across the board.