Following the examples in LoadLine (Googles Android performance teams tool) we should also pick page specific metrics that are not bimodal for our performance alerting.
For example today we alert on Largest Contentful Paint. However which element that is the largest content can change over time as we seen in T385381. There a banner caused the alert to fire. This is a problem we had for a long time with Speed Index too. The way we use to handle it was when we showed a banner, the alert would not fire. However the drawback with that approach is that if we have an alert during times we show banners, we will not see that.
A better way to handle this could be to either pick a specific article image (for example the info-box image on some pages), the first paragraph of text, or maybe the first headline. We need to pick a metric that we think is important for the user on that page. This could be our Wikipedia Web Vitals, metrics that we know is important for our users. We touched this before in for example T187684 and T334770.
We will continue to collect the current metrics but the new metrics will make it easier for us to know that we need to act on an alert.
When we do T373172 should make sure we pick metrics that match the user experience and make it easy for us to spot regressions.