Adding @greg, @TheresNoTime, and @LucasWerkmeister for visibility. These folks have been kindly restarting the bot job recently.

The storms of loss of connection to libera.chat are a bit confusing for me as I have not seen similar issues with Stashbot, Tool-bridgebot, or Jouncebot in the same period. This could be random luck of which libera.chat nodes these various IRC bots are connected to, or it could point to some quirk of configuration or implementation in redis2irc.py that makes it more sensitive to some sort of network delay or interruption. It could also just be some bias in my attention to things too.

The SULWatchers have had some connection problems lately as well, but not to the same frequency.

Is anything visible from other IRC clients? i.e. do they see a connection reset by peer, a forced close due to X, ...?

There could also be a server-specific component; from the SGE era I vaguely remember that wikibugs had trouble connecting to IRC from some specific hosts. E.g. too many IRC bots connecting from the same external IP and the server aggressively throttling.

Feb 06 07:06:09 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 06 08:00:59 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 07 10:49:59 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Ping timeout: 272 seconds)
Feb 08 00:21:46 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (*.net *.split)
Feb 08 13:00:54 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Read error: Connection reset by peer)
Feb 08 17:29:56 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 08 17:37:37 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 09 05:18:38 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 10 04:55:49 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Read error: Connection reset by peer)
Feb 10 12:14:50 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 12 16:16:00 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Read error: Connection reset by peer)
Feb 12 18:05:13 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 14 09:11:21 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 15 04:07:59 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Read error: Connection reset by peer)
Feb 15 05:49:23 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)
Feb 16 04:16:54 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Read error: Connection reset by peer)
Feb 16 06:16:41 ◀━━ Quits:	wikibugs (~wikibugs2@wikimedia/bot/pywikibugs) (Remote host closed the connection)

(times UTC-5)

In T357729#9552163, @bd808 wrote:

Until @valhallasw gets annoyed by me and kicks me out anyway. ;)

I, for one, welcome our new unicorn overlords.

The list from @AntiCompositeNumber seems a lot more limited than the logs from the bot - that suggests that the connection dies even before the bot has a chance to join a channel (consistent with the

2024-02-15 10:49:54,899 - irc3.wikibugs - DEBUG - CONNECT ping-pong ()
2024-02-15 10:50:25,884 - irc3.wikibugs - CRITICAL - connection lost (140684783862592): None

logs that suggest the connection is lost maybe before the full login cycle has completed?

Is there any way to get a tcpdump for the bot? There's obviously no root access in the container but maybe kubernetes has some magic for it. That would allow to see if a random RST is somehow happening (and from which direction)

In T357729#9552954, @valhallasw wrote:

Is there any way to get a tcpdump for the bot? There's obviously no root access in the container but maybe kubernetes has some magic for it. That would allow to see if a random RST is somehow happening (and from which direction)

It would be technically possible using Toolforge root super powers and the magic of nsenter. Tutorials like https://www.digitalocean.com/community/tutorials/how-to-inspect-kubernetes-networking and https://www.suse.com/support/kb/doc/?id=000021060 talk about how to get started with those types of investigations.

I have started working on a bouncer setup for the IRC client in https://gitlab.wikimedia.org/toolforge-repos/wikibugs2-znc as @Legoktm suggested in T357851#9565944.

bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2-znc/-/merge_requests/2

Make ZNC run

bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2-znc/-/merge_requests/2

Make ZNC run

Mentioned in SAL (#wikimedia-cloud) [2024-03-02T19:20:09Z] <wmbot~bd808@tools-sgebastion-11> Deployed znc and service (T357729)

In T357729#9593533, @Stashbot wrote:

Mentioned in SAL (#wikimedia-cloud) [2024-03-02T19:20:09Z] <wmbot~bd808@tools-sgebastion-11> Deployed znc and service (T357729)

I rolled this back for the moment. I'm having a hard time debugging the python irc bot with or without the znc in the mix.

The bot is now running with a znc instance between it and libera.chat. The hope here is that znc has more robust algorithms for detecting connection issues and reacting by attempting to reconnect. Many other small things have changed as well with the deployment of T357851: bd808's big pile of refactoring ideas. Let's see how things settle in in the next few days.

Some random observation: Some IRC notifications are notably delayed. I merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1008513 at 09:15 CET and the IRC notification was sent to #wikimedia-operations at 10:18 CET

In T357729#9599627, @MoritzMuehlenhoff wrote:

Some random observation: Some IRC notifications are notably delayed. I merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1008513 at 09:15 CET and the IRC notification was sent to #wikimedia-operations at 10:18 CET

That delay was most likely related to T359097: Frequent "Redis listener crashed; restarting in a few seconds." errors logged. Let's keep our eyes open however to see if long delays are repeatable now that in theory that particular bug is fixed.

The "connection lost" log event metric is looking much better, but with znc as a proxy I would have expected this no matter what.

$ for f in irc.log*; do echo -n $f " "; grep 'connection lost' $f | wc -l; done
irc.log  0
irc.log.2024-03-03  1
irc.log.2024-03-04  0
irc.log.2024-03-05  0

In the tools.wikibugs-testing deployment the problem I am seeing now is the irc bot losing it's connection to redis rather than to znc & libera.chat. I am starting to look at reusing the AsyncRedisQueue I added as part of T359096: Bot does not detect when ssh connection to Gerrit is interrupted in the irc bot. I have been able to experimentally demonstrate that AsyncRedisQueue reconnects to Redis robustly via local testing where I shutdown the Redis instance, let the bot thrash for a minute trying to reconnect, and then brought the instance back up resulting in a restored connection. The irc bot has logic that is trying to accomplish this, but it seems to be less robust.