Hi Joe,

Let us know if there is any support you'd like from our team on this task, otherwise moving to Radar for now.

In T271822#6773823, @lmata wrote:

Hi Joe,

Let us know if there is any support you'd like from our team on this task, otherwise moving to Radar for now.

Hi, I would probably like some advice on which direction to go with. Specifically:

• Is there a way to tell prometheus to read from multiple ports from the same pod in the same job definition? I think there isn't, but maybe I'm missing something
• If that's not possible, we would need to add one scraping of the kubernetes api for each additional exporter after the first one

As an alternative, I could proxy all exporters to a single one collecting the metrics of all of them. After all, all those exporters are more or less concurring to report the functionality of this single application.

In T271822#6790319, @Joe wrote:

• Is there a way to tell prometheus to read from multiple ports from the same pod in the same job definition? I think there isn't, but maybe I'm missing something

For sake of completeness: We had a discussion about that in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/651757/3/_scaffold/templates/_containers.tpl#178 and the outcome was that it looks doable (though not tested afaik).

I dug a bit into upstream Prometheus issues and this is relevant: https://github.com/prometheus/prometheus/issues/3756 . One of the proposed solution is indeed the same/very similar as the result of https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/651757/3/_scaffold/templates/_containers.tpl#178 : namely to scrape ports with a certain naming scheme. In the case of https://github.com/grafana/jsonnet-libs/blob/master/prometheus-ksonnet/lib/prometheus-config.libsonnet#L72 for example the suffix for port name is -metrics

Some status update after a good chat with the o11y folks:

• We will go down the path @fgiunchedi and @JMeybohm suggested, thus marking all ports that prometheus should scrape with a name ending in -metrics
• We will first add the new job to prometheus in puppet, so that it will start scraping, but it will just come up empty as no port currently has a compliant name
• We will convert one chart by removing the prometheus.io/port annotation and adding a name to the statsd-exporter port
• Once we can confirm that we're still gathering the data, we will convert all current charts (scaffolding is already take care of with https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/651757)

Change 919053 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] prometheus/k8s: add selective scraping of ports

https://gerrit.wikimedia.org/r/919053

Change 919055 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] base.meta.pod_annotations: support annotations for prometheus scraping

https://gerrit.wikimedia.org/r/919055

Change 919058 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] mediawiki: update modules, enable named ports

https://gerrit.wikimedia.org/r/919058

Change 919155 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/deployment-charts@master] shellbox: update modules, enable named ports

https://gerrit.wikimedia.org/r/919155

Change 919055 merged by jenkins-bot:

[operations/deployment-charts@master] base.meta.pod_annotations: support annotations for prometheus scraping

https://gerrit.wikimedia.org/r/919055

Change 919053 merged by Giuseppe Lavagetto:

[operations/puppet@production] prometheus/k8s: add selective scraping of ports in staging

https://gerrit.wikimedia.org/r/919053

Change 919155 merged by jenkins-bot:

[operations/deployment-charts@master] shellbox: update modules, enable named ports

https://gerrit.wikimedia.org/r/919155

Change 920636 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] prometheus: add the named port scraping option to all wikikube clusters

https://gerrit.wikimedia.org/r/920636

Change 920636 merged by Giuseppe Lavagetto:

[operations/puppet@production] prometheus: add the named port scraping option to all clusters

https://gerrit.wikimedia.org/r/920636

Change 919058 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: enable named ports

https://gerrit.wikimedia.org/r/919058