Mentioned in SAL (#wikimedia-operations) [2024-04-19T18:50:34Z] <ryankemper> [WDQS] T363004 Restarted wdqs2010 and wdqs2024 to clear out their in-application-memory ban lists

Mentioned in SAL (#wikimedia-operations) [2024-04-19T19:15:59Z] <ryankemper> [WDQS] T363004 Restarted wdqs2012 to clear out its in-application-memory ban lists (it had pybal's twisted user agent banned)

Logstash entries show that the PyBal and Prometheus poller user agents ( Twisted PageGetter and prometheus-public-sparql-ep-check , respectively) are being throttled/banned by the WDQS throttling filter. The hosts are removed by pybal after it's banned by the filter, so user impact is minimal.

We've restarted the wdqs-blazegraph process to clear out the filters, but it's possible the filters will eventually turn back on. If we do nothing about this situation, we'll still get a lot of alerts but there won't be much user impact. We could also depool CODFW completely, which would keep alerts from firing. We'll do this if the problem resurfaces before shift end (~4 h or so).

Note that this is happening in the passive DC (codfw) only. I know we added a HTTP header recently to better identify our monitoring traffic, maybe this has something to do with it? It seems pretty unlikely, but I'm struggling to understand why this has never happened until now.