Intro
The instance label is automatically added by Prometheus and (typically) it is in the form of hostname:port from which metrics have been fetched from. For example node-exporter is on port 9100 thus all of its metrics have instance=HOST:9100. Icinga compatibility alerts (prefixed with Icinga/) don't have the port (though we could change that easily). I have inquired Karma upstream about this situation in https://github.com/prymitive/karma/issues/3938
Problem statement
From the alerts dashboard we'd like to allow for filtering/grouping by host (e.g. to show all active alerts of a single host). In the dashboard UI clicking a label adds said label to the current filters; therefore clicking a host:port label will show all alerts for that specific port and not the host. Showing all alerts per-host would mean changing the filter from instance=HOST:PORT to instance=~^HOST:.* (for example).
Solutions
Below a list of possible solutions and the tradeoffs involved:
1. Strip port from instance at ingestion time
In this case we'd have instance to be without a port at ingestion time (i.e. Prometheus stores metrics without the port). This solution is quite invasive (likely dashboards need to be adapted), we'd have 100% new metrics since the instance label changes, and having port in instance does have its use cases (e.g. when co-hosting multiple instances of the same software).
2. Strip port from instance for outgoing alerts
We would strip the port from instance only when sending alerts to alertmanager. The solution is not invasive and allows for the easy grouping mentioned above. Downsides include the fact that the alert's labels don't reflect the underlying expression labels anymore, leading to potential confusion. Another point of confusion might be when metrics with different ports (but same host) are alerting (e.g. search has multiple ES instances on the same hw)
3. Add a new label host based on instance to alerts
We would add a new label host to alerts (adding it to the metrics is possible but we'd incur in metrics churn described above). The solution has the advantage of a brand new label (i.e. no confusion), however the hostname would be shown twice, once in instance and once in host
4. Keep port in instance
In this case we strive for consistency between alerts and their underlying metrics, and would add a (bogus) port to Icinga / LibreNMS alerts. While the grouping is achieved via a different filter (i.e. non-default from the dashboard UI) this is the least invasive solution and the most "consistent" one. For "quality of life" we could ask the dashboard UI (Karma) upstream if they are willing to implement different filters on click; this way we could still have one-click filtering/grouping to select all alerts for a given host.
5. Add a new label host based on instance to metrics
We would add a new host label based on instance to each job in Prometheus. The upside is that we don't have to worry about discrepancies between alerts and metrics (e.g. in solution 3), downsides include (similar to solution 1): changing each job in puppet, new metrics created (i.e. losing history of the current metrics), in some contexts (e.g. k8s) the instance label might not be a (real) hostname