Strongswan Icinga check: do not report issues about depooled hosts
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• ema
	Oct 24 2016, 2:59 PM

Description

Hosts with long running issues such as those described in T148891 currently affecting cp1052 cause the icinga check script for strongswan to report lots of critical errors. We do not want those errors to be reported when the machine is depooled.

Details

	Subject	Repo	Branch	Lines +/-
	profile: clean up ipsec aggregate check	operations/puppet	production	+0 -23
	profile: apply ipsec monitoring where enabled with ipsec_exporter	operations/puppet	production	+16 -0

Customize query in gerrit

Related Objects

Mentioned Here: T267581: Phase out "redis_sessions" cluster and away from memcached cluster
T196487: Upgrade eqiad rack D4 to 10G switch
T148891: cp1052 ethernet link down 2016-10-22 14:11

Event Timeline

• ema created this task.Oct 24 2016, 2:59 PM

Restricted Application added a project: SRE. · View Herald TranscriptOct 24 2016, 2:59 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• ema triaged this task as Medium priority.Oct 24 2016, 2:59 PM

fwiw, i saw these in Icinga web-ui but icinga-wm was apparently not talking about them on IRC, but i did not see "disabled notifications" or ACKs in the web ui. then i ACKed them all and it said that on IRC, that was a bit strange.

BBlack moved this task from Backlog to General on the Traffic board.Oct 24 2016, 11:48 PM

Icinga external commands include SCHEDULE_SVC_DOWNTIME, which seems handy. We could perhaps try writing a script that issues a SCHEDULE_SVC_DOWNTIME for the IPSec service for each host defined in the role::ipsec targets array?

• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 8:53 PM

This was mostly about cache nodes back when those had ipsec, I think. The remaining case that uses ipsec anymore is the memcached cluster. Does this matter there, is it worth fixing there? Unsetting priority to get this some Triage, and removing the traffic label.

• crusnov added projects: observability, serviceops.Sep 24 2020, 4:02 PM

herron moved this task from Inbox to Radar on the observability board.Sep 24 2020, 4:03 PM

In T148976#6488108, @BBlack wrote:

This was mostly about cache nodes back when those had ipsec, I think. The remaining case that uses ipsec anymore is the memcached cluster. Does this matter there, is it worth fixing there? Unsetting priority to get this some Triage, and removing the traffic label.

During T196487, with all eqiad row D hosts downtimed, we got this:

14:06 <+icinga-wm> PROBLEM - Aggregate IPsec Tunnel Status codfw on alert1001 is CRITICAL: instance={mc2033,mc2034,mc2035,mc2036} site=codfw tunnel={mc1033_v4,mc1034_v4,mc1035_v4,mc1036_v4}
                   https://wikitech.wikimedia.org/wiki/Monitoring/strongswan https://grafana.wikimedia.org/d/B9JpocKZz/ipsec-tunnel-status

So I would say that the issue is still there, and still worth fixing!

herron triaged this task as Medium priority.Sep 30 2020, 5:28 PM

herron moved this task from Radar to Inbox on the observability board.

Change 632738 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: apply ipsec monitoring where enabled with ipsec_exporter

https://gerrit.wikimedia.org/r/632738

Change 632739 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: clean up ipsec aggregate check

https://gerrit.wikimedia.org/r/632739

lmata moved this task from Inbox to Backlog on the observability board.Oct 19 2020, 3:28 PM

lmata edited projects, added SRE Observability; removed observability.Jul 12 2021, 2:21 AM

lmata moved this task from Inbox to Backlog on the SRE Observability board.Jul 15 2021, 4:09 AM

lmata edited projects, added Observability-Metrics; removed SRE Observability.Jan 31 2022, 1:43 PM

jijiki moved this task from Incoming 🐫 to 🙈🙉🙊Backlog on the serviceops board.Sep 28 2022, 2:27 PM

Strongswan is going away because we do not need it anymore. We were using it for redis_sessions T267581

$ sudo cumin --dry-run R:Class=strongswan
16 hosts will be targeted:
mc[2020,2022,2026,2030,2032,2034,2036,2038].codfw.wmnet,mc[1038,1040,1042,1044,1048,1050,1052,1054].eqiad.wmnet

lmata moved this task from Inbox to Done on the Observability-Metrics board.Jan 16 2023, 5:42 PM

Change 632738 abandoned by Cwhite:

[operations/puppet@production] profile: apply ipsec monitoring where enabled with ipsec_exporter

Reason:

task was declined

https://gerrit.wikimedia.org/r/632738

Change 632739 abandoned by Cwhite:

[operations/puppet@production] profile: clean up ipsec aggregate check