Page MenuHomePhabricator

wmcs-cookbooks: use a single bot for SAL logging
Closed, InvalidPublic

Description

We are currently using two separate bots to send log messages from WMCS cookbooks to SAL:

  • SALLogger from wmcs_libs.common (only used by WMCS cookbooks)
    • SALLogger writes to IRC by sending a message to wm-bot, which is running on wm-bot.wm-bot.wmcloud.org
  • sal_logger from spicerack/_log.py (used by both production and wmcs cookbooks, because it's embedded in Spicerack)
    • sal_logger writes to IRC by sending a message to tcpircbot (aka logmsgbot), which is running on alert1001.wikimedia.org
    • when we run WMCS cookbooks from a laptop, sal_logger is disabled

The log messages sent via the two bots are formatted slightly differently, and can appear in the wrong order (a message sent by bot A sometimes appear in SAL after a message sent by bot B, even if it was generated before).

It would be nice to use a single method. One blocker is that tcpircbot is currently not reachable from the public internet, so it cannot be used when running cookbooks from a laptop. This is tracked in T343336.

Event Timeline

@fnegri I suspect that in most cases the additional manual logging is done because the START/END messages were not logged. Now that they are I guess that most of those additional logs calls can be dropped by just implementing the runtime_description() property in most cookbooks that will log the same information in the START/END messages. But correct me if this assumption is wrong.

@Volans yes I think maybe 80% of the messages could be replaced by a runtime_description, but we will still have situations where we want to log multiple "steps" in a cookbook, something like:

  • START
  • doing step 1
  • doing step 2
  • END

It would be trivial to switch our current SALLogger calls for those situations to a call to sal_logger (the one provided by Spicerack). We could even do an easy search/replace of all the existing calls to SALLogger, and leave the migration to runtime_description for later.

The real blocker is that Spicerack's sal_logger doesn't work if you run cookbooks from a laptop... which is the current status quo. If we successfully move all our cookbook users to running them from cloudcumin, then we can remove our custom SALLogger and use sal_logger for everything.

Whether we can ask everyone to use cloudcumins will also depend on T343330.

fnegri added a subscriber: taavi.

After the recent patches by @taavi (https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/953596/ and https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/953597/), we are still using two bots, but we have unified the logging classes, the formatting of the messages is consistent and we no longer have the ordering problem (because logs are either sent all to tcpircbot or to wm-bot).

I am resolving this task as "Invalid".