Out current scrape config for k8s pods (`job="k8s-pods"`) creates one scrape target for every containerPort of every container if the `prometheus.io/port` annotation is not set. This leads to quite some (intentionally) down targets and should be avoided.
We already defines the schema of adding the suffix `-metrics` to container ports that should be scraped by prometheus, so we could limit the targets created to just them (as long as `prometheus.io/port` is not set).
This has one issue though: If we want to scrape multiple ports in one pod (e.g. a single `prometheus.io/port` is not an option) and the metrics are exposed via the generic application port, we would have to suffix that port with `-metitics` which is not very intuitive. Multiple `containerPort` entries for the same port are unfortunately not possible, but maybe there is a good way around this that I have not thought of yet.