Page MenuHomePhabricator

profile::icinga::ircbot trying to run icinga-wm without proper password from monitoring Cloud VPS project
Closed, ResolvedPublic

Description

[16:18]  <    glguy> The misconfigured bot running on nat.cloudgw.eqiad1.wikimediacloud.org repeatedly failing to authenticate as icinga-wm is still going which risk getting the whole host banned if the SASL failure allow-rate changes

Some poking around points to the profile::icinga::ircbot being applied on pontoon-icinga-01.monitoring.eqiad1.wikimedia.cloud as the likely root problem here

Event Timeline

bd808 renamed this task from profile::icinga::ircbot trying to run icinga-wm without proper password to profile::icinga::ircbot trying to run icinga-wm without proper password from monitoring Cloud VPS project.Jul 23 2021, 4:38 PM

I was going to try setting a hiera flag to disable the profile, but:

root@pontoon-icinga-01:/var/lib/puppet/state# puppet agent -tv
Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: 'filippo - filippo');
Use 'puppet agent --enable' to re-enable.
root@pontoon-icinga-01:~# cat /etc/icinga/.irc_secret
dummy

Mentioned in SAL (#wikimedia-cloud) [2021-07-23T16:45:38Z] <bd808> rm /usr/local/bin/ircecho as the worst fix for T287265

My bad! I'll make sure ircecho not starting is enforced by puppet

Change 708043 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] pontoon: disable ircecho/ircbot

https://gerrit.wikimedia.org/r/708043

Change 708043 merged by Filippo Giunchedi:

[operations/puppet@production] pontoon: disable ircecho/ircbot

https://gerrit.wikimedia.org/r/708043

Change 708073 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: fix ircbot::ensure logic

https://gerrit.wikimedia.org/r/708073

Change 708073 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: fix ircbot::ensure logic

https://gerrit.wikimedia.org/r/708073

fgiunchedi claimed this task.

Stubbed my toe into some active/passive logic and the hiera variable, ircecho is disabled for good now (and puppet runs again). Resolving but please reopen if sth is amiss