Page MenuHomePhabricator

Restart pod if not all SULWatchers joined to IRC
Open, Needs TriagePublic

Description

Even with the PingServer ib3 mixin added in rTSTWe7a6f9e045e2: SULWatcher: add PingServer mixin to handle ping timeouts, it seems there is still a case where only one or two of the three bots rejoins the server. Sometimes it results in "Not connected." spam, but usually the connected bot(s) remain quiet. This leaves SULWatcher in an inconsistent state where it also can't be restarted from IRC.

If one bot disconnects from IRC and does not rejoin, the SULWatcher.py script should exit so that k8s can restart the pod.

Event Timeline

Unfortunately the log doesn't provide anything particularly useful:

2024-04-30T01:41:10
Running SULWatcher...
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: getConfig(password)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: getConfig(server)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: getConfig(channel)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: getConfig(nickname)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: buildRegex(self)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: buildWhitelist(self)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: getConfig(alias)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: buildRegex(self)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: buildWhitelist(self)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: getConfig(alias2)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: buildRegex(self)
2024-04-30T01:41:14Z SULWatcher MainThread DEBUG: buildWhitelist(self)
2024-04-30T01:41:14Z SULWatcher MainThread INFO: starting SULWatcher
2024-04-30T01:41:14Z SULWatcher MainThread INFO: starting SULWatcher2
2024-04-30T01:41:14Z SULWatcher MainThread INFO: starting SULWatcher3
2024-04-30T01:41:14Z SULWatcher MainThread INFO: starting EventStream
2024-04-30T01:41:17Z ib3.nick SULWatcher WARNING: Requested nick "SULWatcher" in use
2024-04-30T01:41:17Z ib3.nick SULWatcher2 WARNING: Requested nick "SULWatcher2" in use
2024-04-30T01:41:17Z ib3.nick SULWatcher3 WARNING: Requested nick "SULWatcher3" in use
2024-04-30T01:41:19Z ib3.auth SULWatcher INFO: Connected to server cadmium.libera.chat
2024-04-30T01:41:19Z ib3.auth SULWatcher2 INFO: Connected to server cadmium.libera.chat
2024-04-30T01:41:19Z ib3.mixins SULWatcher INFO: Joining #wikimedia-unifications
2024-04-30T01:41:19Z ib3.mixins SULWatcher2 INFO: Joining #wikimedia-unifications
2024-04-30T01:41:19Z ib3.auth SULWatcher3 INFO: Connected to server cadmium.libera.chat
2024-04-30T01:41:19Z ib3.mixins SULWatcher3 INFO: Joining #wikimedia-unifications
2024-04-30T18:41:14Z ib3.mixins SULWatcher WARNING: Connection timed out. Disconnecting.
2024-04-30T18:41:14Z ib3.mixins SULWatcher3 WARNING: Connection timed out. Disconnecting.
2024-04-30T18:41:14Z ib3.mixins SULWatcher2 WARNING: Connection timed out. Disconnecting.
2024-04-30T18:42:17Z ib3.auth SULWatcher2 INFO: Connected to server zinc.libera.chat
2024-04-30T18:42:17Z ib3.mixins SULWatcher2 INFO: Joining #wikimedia-unifications