Our current scrape config for k8s pods (job="k8s-pods") creates one scrape target for every containerPort of every container if the prometheus.io/port annotation is not set. This leads to quite some (intentionally) down targets and should be avoided.
We already define the schema by adding the suffix -metrics to container ports that should be scraped by prometheus, so we could limit the targets created to just them (as long as prometheus.io/port is not set).
This has one issue though: If we want to scrape multiple ports in one pod (e.g. a single prometheus.io/port is not an option) and the metrics are exposed via the generic application port, we would have to suffix that port with -metrics which is not very intuitive. Multiple containerPort entries for the same port are unfortunately not possible, but maybe there is a good way around this that I have not thought of yet.
One idea to work around this issue could be to have multiple k8s-pods scrape jobs, one per port. We would still be hardcoding the maximum number of scrape targets per pod, though:
apiVersion: v1
kind: Pod
metadata:
annotations:
prometheus.io/scrape0: "true"
prometheus.io/port0: 1234
prometheus.io/path0: "/foo/metrics"
prometheus.io/scrape1: "true"
prometheus.io/port1: 4321
...- job_name: k8s-pods-0
relabel_configs:
- action: keep
source_labels: ['__meta_kubernetes_pod_annotations_prometheus_io_scape0']
regex: 'true'
# no idea if the following is possible
- action: replace
source_labels: ['job']
regex: '.*'
replacement: 'k8s-pods'
target_label: 'job'
...
- job_name: k8s-pods-1
relabel_configs:
- action: keep
source_labels: ['__meta_kubernetes_pod_annotations_prometheus_io_scape1']
regex: 'true'
# no idea if the following is possible
- action: replace
source_labels: ['job']
regex: '.*'
replacement: 'k8s-pods'
target_label: 'job'
...