Change Details

== Intro == The `instance` label is automatically added by Prometheus and (typically) it is in the form of `hostname:port` from which metrics have been fetched from. For example node-exporter is on port 9100 thus all of its metrics have `instance=HOST:9100`. Icinga compatibility alerts (prefixed with `Icinga/`) don't have the port (though we could change that easily). I have inquired Karma upstream about this situation in https://github.com/prymitive/karma/issues/3938 == Problem statement == From the alerts dashboard we'd like to allow for filtering/grouping by host (e.g. to show all active alerts of a single host). In the dashboard UI clicking a label adds said label to the current filters; therefore clicking a `host:port` label will show all alerts for that specific port and not the host. Showing all alerts per-host would mean changing the filter from `instance=HOST:PORT` to `instance=~^HOST:.*` (for example). == Solutions == Below a list of possible solutions and the tradeoffs involved: ==== 1. Strip port from instance at ingestion time ==== In this case we'd have `instance` to be without a port at ingestion time (i.e. Prometheus stores metrics without the port). This solution is quite invasive (likely dashboards need to be adapted), we'd have 100% new metrics since the `instance` label changes, and having port in `instance` does have its use cases (e.g. when co-hosting multiple instances of the same software). ==== 2. String port from instance for outgoing alerts ==== We would strip the port from `instance` only when sending alerts to alertmanager. The solution is not invasive and allows for the easy grouping mentioned above. Downsides include the fact that the alert's labels don't reflect the underlying expression labels anymore, leading to potential confusion. Another point of confusion might be when metrics with different ports (but same host) are alerting (e.g. search has multiple ES instances on the same hw) ==== 3. Add a new label `host` based on `instance` to alerts ==== We would add a new label `host` to alerts (adding it to the metrics is possible but we'd incur in metrics churn described above). The solution has the advantage of a brand new label (i.e. no confusion), however the hostname would be shown twice, once in `instance` and once in `host` ==== 4. Keep port in `instance` ==== In this case we strive for consistency between alerts and their underlying metrics, and would add a (bogus) port to Icinga / LibreNMS alerts. While the grouping is achieved via a different filter (i.e. non-default from the dashboard UI) this is the least invasive solution and the most "consistent" one. For "quality of life" we could ask the dashboard UI (Karma) upstream if they are willing to implement different filters on click; this way we could still have one-click filtering/grouping to select all alerts for a given host.

== Intro == The `instance` label is automatically added by Prometheus and (typically) it is in the form of `hostname:port` from which metrics have been fetched from. For example node-exporter is on port 9100 thus all of its metrics have `instance=HOST:9100`. Icinga compatibility alerts (prefixed with `Icinga/`) don't have the port (though we could change that easily). I have inquired Karma upstream about this situation in https://github.com/prymitive/karma/issues/3938 == Problem statement == From the alerts dashboard we'd like to allow for filtering/grouping by host (e.g. to show all active alerts of a single host). In the dashboard UI clicking a label adds said label to the current filters; therefore clicking a `host:port` label will show all alerts for that specific port and not the host. Showing all alerts per-host would mean changing the filter from `instance=HOST:PORT` to `instance=~^HOST:.*` (for example). == Solutions == Below a list of possible solutions and the tradeoffs involved: ==== 1. Strip port from instance at ingestion time ==== In this case we'd have `instance` to be without a port at ingestion time (i.e. Prometheus stores metrics without the port). This solution is quite invasive (likely dashboards need to be adapted), we'd have 100% new metrics since the `instance` label changes, and having port in `instance` does have its use cases (e.g. when co-hosting multiple instances of the same software). ==== 2. String port from instance for outgoing alerts ==== We would strip the port from `instance` only when sending alerts to alertmanager. The solution is not invasive and allows for the easy grouping mentioned above. Downsides include the fact that the alert's labels don't reflect the underlying expression labels anymore, leading to potential confusion. Another point of confusion might be when metrics with different ports (but same host) are alerting (e.g. search has multiple ES instances on the same hw) ==== 3. Add a new label `host` based on `instance` to alerts ==== We would add a new label `host` to alerts (adding it to the metrics is possible but we'd incur in metrics churn described above). The solution has the advantage of a brand new label (i.e. no confusion), however the hostname would be shown twice, once in `instance` and once in `host` ==== 4. Keep port in `instance` ==== In this case we strive for consistency between alerts and their underlying metrics, and would add a (bogus) port to Icinga / LibreNMS alerts. While the grouping is achieved via a different filter (i.e. non-default from the dashboard UI) this is the least invasive solution and the most "consistent" one. For "quality of life" we could ask the dashboard UI (Karma) upstream if they are willing to implement different filters on click; this way we could still have one-click filtering/grouping to select all alerts for a given host. ==== 5. Add a new label `host` based on `instance` to metrics ==== We would add a new `host` label based on `instance` to each job in Prometheus. The upside is that we don't have to worry about discrepancies between alerts and metrics (e.g. in solution 3), downsides include (similar to solution 1): changing each job in puppet, new metrics created (i.e. losing history of the current metrics), in some contexts (e.g. k8s) the `instance` label might not be a (real) hostname

== Intro == The `instance` label is automatically added by Prometheus and (typically) it is in the form of `hostname:port` from which metrics have been fetched from. For example node-exporter is on port 9100 thus all of its metrics have `instance=HOST:9100`. Icinga compatibility alerts (prefixed with `Icinga/`) don't have the port (though we could change that easily). I have inquired Karma upstream about this situation in https://github.com/prymitive/karma/issues/3938 == Problem statement == From the alerts dashboard we'd like to allow for filtering/grouping by host (e.g. to show all active alerts of a single host). In the dashboard UI clicking a label adds said label to the current filters; therefore clicking a `host:port` label will show all alerts for that specific port and not the host. Showing all alerts per-host would mean changing the filter from `instance=HOST:PORT` to `instance=~^HOST:.*` (for example). == Solutions == Below a list of possible solutions and the tradeoffs involved: ==== 1. Strip port from instance at ingestion time ==== In this case we'd have `instance` to be without a port at ingestion time (i.e. Prometheus stores metrics without the port). This solution is quite invasive (likely dashboards need to be adapted), we'd have 100% new metrics since the `instance` label changes, and having port in `instance` does have its use cases (e.g. when co-hosting multiple instances of the same software). ==== 2. String port from instance for outgoing alerts ==== We would strip the port from `instance` only when sending alerts to alertmanager. The solution is not invasive and allows for the easy grouping mentioned above. Downsides include the fact that the alert's labels don't reflect the underlying expression labels anymore, leading to potential confusion. Another point of confusion might be when metrics with different ports (but same host) are alerting (e.g. search has multiple ES instances on the same hw) ==== 3. Add a new label `host` based on `instance` to alerts ==== We would add a new label `host` to alerts (adding it to the metrics is possible but we'd incur in metrics churn described above). The solution has the advantage of a brand new label (i.e. no confusion), however the hostname would be shown twice, once in `instance` and once in `host` ==== 4. Keep port in `instance` ==== In this case we strive for consistency between alerts and their underlying metrics, and would add a (bogus) port to Icinga / LibreNMS alerts. While the grouping is achieved via a different filter (i.e. non-default from the dashboard UI) this is the least invasive solution and the most "consistent" one. For "quality of life" we could ask the dashboard UI (Karma) upstream if they are willing to implement different filters on click; this way we could still have one-click filtering/grouping to select all alerts for a given host. ==== 5. Add a new label `host` based on `instance` to metrics ==== We would add a new `host` label based on `instance` to each job in Prometheus. The upside is that we don't have to worry about discrepancies between alerts and metrics (e.g. in solution 3), downsides include (similar to solution 1): changing each job in puppet, new metrics created (i.e. losing history of the current metrics), in some contexts (e.g. k8s) the `instance` label might not be a (real) hostname