The Prometheus metrics introduced in T376719: alerting: detect if a kernel had a panic to detect kernel panics and other kernel errors can overlap: for example a kernel error logged with priority=err and message=taint will increment both kernel_dmesg_err_priority and kernel_dmesg_taint.
Similarly, a message can have priority=err and also contain the word warning, incrementing both kernel_dmesg_err_priority and kernel_dmesg_warning.
This also makes it more difficult to define alerting rules in alertmanager without triggering two alerts for a single error.
We should define metrics that identify only a specific type of error, then we can tune the alerts based on that.
We can also use Prometheus labels to categorize the different types of messages.