(see also T384804: Reduce automatic messages on #wikimedia-operations from January 2025)

(for want of a better tag)

We could consider setting bots to use direct messages to reduce the amount of chatter in the channel without losing notification/highlight functionality outright. Off hand:

• Wikibugs [V:+/-1]
• Wikibugs (PS) notifications
• Bot help output
• Script awaiting input pings
• Detailed START/STATUS/DONE/PASS output. Along with a single "heads up" message to the channel, and follow-up messages only on failures
• Downtime cookbook

In T417163#11659366, @herron wrote:

We could consider setting bots to use direct messages to reduce the amount of chatter in the channel without losing notification/highlight functionality outright. Off hand:

• Wikibugs [V:+/-1]
• Wikibugs (PS) notifications

Wikibugs output is meant to inform everyone. Folks already can subscribe to email pings for these things if they want private notice. V:+1 messages in general might be reasonable to filter out... I thought there might already be a feature request for that in Wikibugs, but I'm not seeing one at the moment. V:-1 events seem pretty important as this is the signal that CI gives most often.

I think one easy thing to do would be to prune the Wikibugs config so that Phab ops(-.*)? and SRE(-.*)? projects no longer report to the channel. They would go to #wikimedia-dev by default if no other output channel was picked for them. There might be some gerrit repos that could be reported elsewhere too (wikimedia/bots/jouncebot jumps out at me).

Current configs:

"#wikimedia-operations":
  "ops(-.*)?":
  "puppet":
  "Blocked-on-Operations":
  "SRE(-.*)?":

"#wikimedia-operations":
  ".*":
    # All commits to a wmf/ branch goes here,
    # for co-ordinating production changes
    branch: "^wmf\/"
  "labs/private":
  "operations(?!/debs/wikistats).*":
    # Do not post betacluster stuff here
    branch: "^(?!betacluster)"
  "operations/software/gerrit.*":
    branch: "^deploy/"
  "wikimedia/bots/jouncebot":

# Does not include a #wikimedia-operations section at all

duplicate of T384804 fwiw

For reference see T384804 where I did a breakdown per bots and components they report, I built it based on the #wikimedia-operations channel logs. The bots / components they report on can then be split one by one, for example I have proposed as a proof of concept to move DCops Phab notifications to their dedicated channel: I proposed https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/49/diffs

I had enough push backs I have declined the task. Instead I have /ignore some bots and uses CSS tricks in my client to selectively hide messages and that fixed it for me ™

Quoting @hnowlan:

This makes tracking response difficult, and in some situations recently we have had to move to #wikimedia-sre in order to communicate properly. If we wish to pursue this as an official pattern, it needs to be documented and recorded.

This is documented at wikitech:Incident_response/Runbook:

If the alert may be caused by a (D)DoS or other attack or security issue, move to #mediawiki_security. If there’s too much alert noise, move to #wikimedia-sre. Otherwise, stay in #wikimedia-operations .

If we wanted to change that, and make it "coordinate in #wikimedia-sre by default," then I agree we'd need to publish that change.

In previous discussions of the same topic, we decided that in most incidents, it's good for our situational awareness to have bot updates in the same place as human conversation. Keeping the human conversation in #wikimedia-operations also means that deployers (who don't all monitor #wikimedia-sre) can tell when we're working on an incident and they shouldn't deploy. (The exception is cases where the alert traffic is so overwhelming that it's impossible to follow a conversation. That's less common, now that we don't have per-appserver icinga alerts, but still happens sometimes, e.g. the 350+ alert messages from T418840.)

That's why the existing practice is there, but we can change it if we decide that reasoning doesn't hold up anymore.

In T417163#11659366, @herron wrote:

• Detailed START/STATUS/DONE/PASS output. Along with a single "heads up" message to the channel, and follow-up messages only on failures

I renamed a host today and it logged 24 lines