Change Details

I've muted the Primary outbound port utilisation over 80% alert; since 21:21:05 on 2024-02-25 this has been repeatedly firing and resolving for cr2-codfw. Looking at [[ https://superset.wikimedia.org/superset/dashboard/p/pZxOp7RrVm0/ | superset ]] the extra traffic is largely from AS32934 (FB), but the people available didn't feel confident trying to work out which IP ranges to ratelimit with requestctl (and you can't use as_number as a filter). So we took the decision to mute the alert rather than having it keep paging on a Sunday evening, on the basis that if this becomes a wider issue more pages will fire. This will want putting back on Monday, though, I suspect.

I've muted the Primary outbound port utilisation over 80% alert; since 21:21:05 on 2024-02-25 this has been repeatedly firing and resolving for cr2-codfw. Looking at [[ https://superset.wikimedia.org/superset/dashboard/p/pZxOp7RrVm0/ | superset ]] the extra traffic is largely from AS32934 (FB), but the people available didn't feel confident trying to work out which IP ranges to ratelimit with requestctl (and you can't use as_number as a filter). So we took the decision to mute the alert rather than having it keep paging on a Sunday evening, on the basis that if this becomes a wider issue more pages will fire. This will want putting back on Monday, though, I suspect. Update 2024-02-26: Alert enabled again.

I've muted the Primary outbound port utilisation over 80% alert; since 21:21:05 on 2024-02-25 this has been repeatedly firing and resolving for cr2-codfw. Looking at [[ https://superset.wikimedia.org/superset/dashboard/p/pZxOp7RrVm0/ | superset ]] the extra traffic is largely from AS32934 (FB), but the people available didn't feel confident trying to work out which IP ranges to ratelimit with requestctl (and you can't use as_number as a filter). So we took the decision to mute the alert rather than having it keep paging on a Sunday evening, on the basis that if this becomes a wider issue more pages will fire. This will want putting back on Monday, though, I suspect. Update 2024-02-26: Alert enabled again.