• What actually is the TypeError: a coroutine was expected, got None exception telling us about the state of the system?
• Is connectivity to Redis actually lost this often?
• Is there some other expected scenario that could explain the frequency of this error?

In T359097#9597794, @bd808 wrote:

• What actually is the TypeError: a coroutine was expected, got None exception telling us about the state of the system?

Here's where the exception comes from, the asyncio.Task constructor:

def __init__(self, coro, *, loop=None, name=None):
    super().__init__(loop=loop)
    if self._source_traceback:
        del self._source_traceback[-1]
    if not coroutines.iscoroutine(coro):
        # raise after Future.__init__(), attrs are required for __del__
        # prevent logging for pending task in __del__
        self._log_destroy_pending = False
        raise TypeError(f"a coroutine was expected, got {coro!r}")

The call that leads to this in wikibugs2.irc.Redis2Irc is:

if useful_info.get("raw"):
    # Send a pre-formatted Gerrit message.
    bot.create_task(
        bot.privmsg_many(
            useful_info["channels"],
            useful_info["msg"],
        ),
    )

bot.privmsg_many is not marked as async, thus it should always fail the coroutines.iscoroutine test. I think this error happens every time that this branch is entered. Now the burning question is if the proper fix is marking bot.privmsg_many as async so that it returns a coroutine, or if instead the bot.create_task wrapper should be removed.

In T359097#9597794, @bd808 wrote:

• Is connectivity to Redis actually lost this often?

Yes, in the past 15 minutes the worker node where the irc task is running on seems to have created 80 Redis connections.

bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/9

irc: Mark Redis2Irc.privmsg_many as async

bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/9

irc: Mark Redis2Irc.privmsg_many as async

Mentioned in SAL (#wikimedia-cloud) [2024-03-05T18:46:36Z] <wmbot~bd808@tools-sgebastion-11> Restarting irc job to pick up fix for T359097