Out current scrape config for k8s pods (`job="k8s-pods"`) creates one scrape target for every containerPort of every container if the `prometheus.io/port` annotation is not set. This leads to quite some (intentionally) down targets and should be avoided.
We already defines the schema of adding the suffix `-metrics` to container ports that should be scraped by prometheus, so we could limit the targets created to just them (as long as `prometheus.io/port` is not set).
This has one issue though: If we want to scrape multiple ports in one pod (e.g. a single `prometheus.io/port` is not an option) and the metrics are exposed via the generic application port, we would have to suffix that port with `-metrics` which is not very intuitive. Multiple `containerPort` entries for the same port are unfortunately not possible, but maybe there is a good way around this that I have not thought of yet.
One idea to work around this issue could be to have multiple k8s-pods scrape jobs, one per port. That would effectively sill hardcoding the maximum number of scrape targets per pod, though:
```
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/scrape0: "true"
prometheus.io/port0: 1234
prometheus.io/path0: "/foo/metrics"
prometheus.io/scrape1: "true"
prometheus.io/port1: 4321
...
```
```
- job_name: k8s-pods-0
relabel_configs:
- action: keep
source_labels: ['__meta_kubernetes_pod_annotations_prometheus_io_scape0']
regex: 'true'
# no idea if the following is possible
- action: replace
source_labels: ['job']
regex: '.*'
replacement: 'k8s-pods'
target_label: 'job'
...
- job_name: k8s-pods-1
relabel_configs:
- action: keep
source_labels: ['__meta_kubernetes_pod_annotations_prometheus_io_scape1']
regex: 'true'
# no idea if the following is possible
- action: replace
source_labels: ['job']
regex: '.*'
replacement: 'k8s-pods'
target_label: 'job'
...
```