On several occasions (T374608, T369025, T373639) we have had incidents which result in us saturating our link from codfw to eqsin, however we have not received any alerts when the circuit was maxed out and dropping packets.
The reason for this is unlike most of our transport circuits this is a sub-rated one, i.e. the carrier has a 4Gb/sec limit on the link so we saturate long before our normal alert (utilisation/link_speed) would fire.
We need to work out a good way to do this. The sub-rated speed is represented in Netbox for the link on the z-end termination (example). So in theory we could get this value and compare the usage to it, and alert on the percentage of the actual limit that way.
Another way may just be to alert on outbound tail drops on the router port, for example in the most recent issue:
These stats are coming from gnmic, but we also could use the 'discards' counter from LibreNMS for that. LibreNMS may also count discards for other reasons such as errors but I think they mostly reflect tail drops. We already have some rules in place for those, but we don't alert as we have frequent discards on our older switch models due to limited buffers.
