I think indeed the current system was primarily a "well this seems good enough" solution rather than taking a principled approach.

I don't remember whether the current approach is a 'buffer' or 'drop' - there is value in just giving up at some point (allowing more recent events to be emitted rather than old ones). At the very least it makes more sense to handle this on the *irc* end of the RedisQueue rather than the individual producers.

In T359753#9618016, @valhallasw wrote:

I don't remember whether the current approach is a 'buffer' or 'drop' - there is value in just giving up at some point (allowing more recent events to be emitted rather than old ones).

The current delay mechanism is to block on RedisQueue.put, so it is functionally a buffering approach. Additional events will not be read from the producer's origin until the put returns.

At the very least it makes more sense to handle this on the *irc* end of the RedisQueue rather than the individual producers.

This would be my inclination as well if we decide we still need a to ratelimit outbound from the python side. I just built an AsyncRedisQueue for the gerrit task changes I'm making to use. When the irc task is updated to use AsyncRedisQueue in the future we could look at adding a read delay to AsyncRedisQueue.get or in the bot right after it returns.

Alternately, irc3.IrcBot has flood_burst, flood_rate, and flood_rate_delay config knobs we can try turning to control the irc message production rate at the python outbound edge. The defaults look to be basically the same as the ZNC config defaults: flood_burst=4, flood_rate=1, flood_rate_delay=1. flood_burst here is the max number of lines to allow through before considering adding delay on each loop through the pending messages. After sending the available 1..flood_burst messages in a batch the bot checks to see if there are still pending messages in the queue. If there are it will sleep (flood_rate_delay / flood_rate) seconds, send the next message, and restart the loop. With the default settings then it seems that the bot should sustain at most a rate of 5 messages per 4 seconds: 4 messages + 4 second sleep + 1 message & repeat.

I just did an accidental test of flood protections using only the irc3 bot and znc limits from the tools.wikibugs-testing deployment. The irc bot there had lost connectivity with redis overnight. When I restarted the bot it pumped 72 messages towards the libera.chat servers. Those messages arrived at my client watching the ##wikibugs2 testing channel over a 4 minute interval from 15:24 to 15:28. That's approximately 18 messages per minute, or one every 3.33 seconds as a sustained rate.