profile::icinga::ircbot trying to run icinga-wm without proper password from monitoring Cloud VPS project
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• bd808
	Jul 23 2021, 4:37 PM

Project Tags

Referenced Files

None

Subscribers

Description

[16:18]  <    glguy> The misconfigured bot running on nat.cloudgw.eqiad1.wikimediacloud.org repeatedly failing to authenticate as icinga-wm is still going which risk getting the whole host banned if the SASL failure allow-rate changes

Some poking around points to the profile::icinga::ircbot being applied on pontoon-icinga-01.monitoring.eqiad1.wikimedia.cloud as the likely root problem here

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	icinga: fix ircbot::ensure logic	operations/puppet	production	+5 -3
	pontoon: disable ircecho/ircbot	operations/puppet	production	+3 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	dcaro	T151704 Libera Chat may throttle bot connections from tools
Open	None	T278584 Promote use of SASL for Cloud VPS/Toolforge hosted Libera.chat / Freenode IRC bots
Resolved	fgiunchedi	T287265 profile::icinga::ircbot trying to run icinga-wm without proper password from monitoring Cloud VPS project

Event Timeline

• bd808 created this task.Jul 23 2021, 4:37 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 23 2021, 4:37 PM

• bd808 renamed this task from profile::icinga::ircbot trying to run icinga-wm without proper password to profile::icinga::ircbot trying to run icinga-wm without proper password from monitoring Cloud VPS project.Jul 23 2021, 4:38 PM

RhinosF1 subscribed.Jul 23 2021, 4:39 PM

I was going to try setting a hiera flag to disable the profile, but:

root@pontoon-icinga-01:/var/lib/puppet/state# puppet agent -tv
Notice: Skipping run of Puppet configuration client; administratively disabled (Reason: 'filippo - filippo');
Use 'puppet agent --enable' to re-enable.

root@pontoon-icinga-01:~# cat /etc/icinga/.irc_secret
dummy

Mentioned in SAL (#wikimedia-cloud) [2021-07-23T16:45:38Z] <bd808> rm /usr/local/bin/ircecho as the worst fix for T287265

• bd808 added a parent task: T278584: Promote use of SASL for Cloud VPS/Toolforge hosted Libera.chat / Freenode IRC bots.Jul 23 2021, 4:48 PM

Andrew mentioned this in T287269: pontoon: ever more Cloud VPS VMs unreachable via puppet or cumin.Jul 23 2021, 5:08 PM

My bad! I'll make sure ircecho not starting is enforced by puppet

Change 708043 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] pontoon: disable ircecho/ircbot

https://gerrit.wikimedia.org/r/708043

gerritbot added a project: Patch-For-Review.Jul 26 2021, 9:01 AM

Change 708043 merged by Filippo Giunchedi:

[operations/puppet@production] pontoon: disable ircecho/ircbot

https://gerrit.wikimedia.org/r/708043

Maintenance_bot removed a project: Patch-For-Review.Jul 26 2021, 9:10 AM

Change 708073 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: fix ircbot::ensure logic

https://gerrit.wikimedia.org/r/708073

gerritbot added a project: Patch-For-Review.Jul 26 2021, 9:43 AM

Change 708073 merged by Filippo Giunchedi: