Page MenuHomePhabricator
Create Task
Maniphest T294564

Migrate Foundations Prometheus alerts to AlertManager
Closed, ResolvedPublic
Actions

Description

Followup from a chat with @Volans, there are a few Prometheus-based alerts in Puppet that should be straightforward to migrate to alerting rules (i.e. alerts.git or puppet if for some reason we want to set alerts "administratively" since alerts.git is self-service even for non-root / non-sre).

Specifically:

  • modules/profile/manifests/monitoring.pp: monitoring::check_prometheus { 'smart_healthy':
  • modules/profile/manifests/monitoring.pp: monitoring::check_prometheus { 'edac_correctable_errors':
  • modules/profile/manifests/monitoring.pp: monitoring::check_prometheus { 'edac_syslog_events':
  • modules/profile/manifests/monitoring.pp: monitoring::check_prometheus { 'filesystem_avail_bigger_than_size':
  • modules/profile/manifests/prometheus/alerts.pp: monitoring::check_prometheus { 'widespread-puppet-agent-fail':
  • modules/profile/manifests/prometheus/alerts.pp: monitoring::check_prometheus { 'widespread-puppet-agent-no-resources':

The EDAC checks will be dropped for some ipmi based monitoring

These could be either still valid (e.g. smart_healthy) or could be possibly discarded as not relevant anymore (e.g. filesystem_avail_bigger_than_size)

Details

SubjectRepoBranchLines +/-
profile: remove edac alertsoperations/puppetproduction+1 -38
sre: introduce cluster vs site wide puppet failuresoperations/alertsmaster+42 -23
team-sre/puppet-agent: Add widespread puppet failure (no resources) alertoperations/alertsmaster+35 -2
team-sre/puppet-agent: Add widespread puppet failure alertoperations/alertsmaster+32 -0
P:monitoring: drop check for filesystem_avail_bigger_than_sizeoperations/puppetproduction+0 -21
hardware: add Memory correctable errors (EDAC) alertoperations/alertsmaster+12 -0
P:monitoring: add docs and fix liniting errorsoperations/puppetproduction+15 -12
icinga: remove SMART checkoperations/puppetproduction+5 -23
smart_data_dump: skip over iDRAC devicesoperations/puppetproduction+12 -4
Add SmartNotHealthy to monitor for disk smart alertsoperations/alertsmaster+44 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Oct 28 2021, 2:14 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2021, 2:14 PM
lmata triaged this task as Medium priority.Nov 16 2021, 4:45 PM
fgiunchedi mentioned this in T196994: Open Phab tasks on SMART failure.Dec 15 2021, 9:23 AM
fgiunchedi moved this task from FY2021/2022-Q2 to Backlog on the SRE Observability board.Jan 17 2022, 2:08 PM
fgiunchedi edited projects, added SRE Observability; removed SRE Observability (FY2021/2022-Q2).
lmata edited projects, added Observability-Alerting; removed SRE Observability.Jan 17 2022, 11:07 PM
gerritbot added a comment.Jan 26 2022, 5:39 PM

Change 757489 had a related patch set uploaded (by Volans; author: Volans):

[operations/alerts@master] [WIP] team-sre: add hardware-related checks

https://gerrit.wikimedia.org/r/757489

gerritbot added a project: Patch-For-Review.Jan 26 2022, 5:39 PM
jhathaway added a subtask: T302639: How should we monitor for faulty memory modules?.Feb 27 2022, 11:25 PM
gerritbot added a comment.Feb 28 2022, 2:36 PM

Change 757489 merged by jenkins-bot:

[operations/alerts@master] Add SmartNotHealthy to monitor for disk smart alerts

https://gerrit.wikimedia.org/r/757489

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2022, 3:11 PM
jhathaway closed subtask T302687: Don't move filesystem_avail_bigger_than_size icinga check to alert manager? as Resolved.Mar 1 2022, 3:10 PM
lmata moved this task from Inbox to Prioritized on the Observability-Alerting board.Apr 5 2022, 12:27 PM
lmata moved this task from Prioritized to Radar on the Observability-Alerting board.Apr 5 2022, 12:29 PM
fgiunchedi added a subscriber: jhathaway.Apr 7 2022, 12:42 PM
In T294564#7741525, @gerritbot wrote:

Change 757489 merged by jenkins-bot:

[operations/alerts@master] Add SmartNotHealthy to monitor for disk smart alerts

https://gerrit.wikimedia.org/r/757489

@jhathaway I saw the alert firing today, looks like it is working as expected so that's great! I believe the old icinga alert can be removed now (i.e. both are firing e.g. for aqs1007 ATM)

jhathaway added a comment.EditedApr 8 2022, 7:49 PM
In T294564#7838028, @fgiunchedi wrote:

@jhathaway I saw the alert firing today, looks like it is working as expected so that's great! I believe the old icinga alert can be removed now (i.e. both are firing e.g. for aqs1007 ATM)

That is good news, however it appears that the two alerts are false positives. I dug into it a bit and it appears both drives are virtual drives exposed by the idrac:

root@aqs1007:~# lsblk -S /dev/{sdm,sdh}
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sdh  10:0:0:0   disk iDRAC    MAS001           0329 usb
sdm  11:0:0:0   disk iDRAC    MAS025           0329 usb

I then went digging in the Prometheus smartmon script, https://github.com/prometheus-community/node-exporter-textfile-collector-scripts/blob/master/smartmon.py, only to discover that we wrote our own.

However neither theirs nor ours seems to provide a way to skip a certain vendor or model of a drive.

I think we should augment the script to query the vendor via /sys/, i.e. cat /sys/block/sdm/device/vendor in addition to or instead of querying via smartctl, so that we can exclude idrac devices.

fgiunchedi added a comment.Apr 11 2022, 2:22 PM
In T294564#7842052, @jhathaway wrote:
In T294564#7838028, @fgiunchedi wrote:

@jhathaway I saw the alert firing today, looks like it is working as expected so that's great! I believe the old icinga alert can be removed now (i.e. both are firing e.g. for aqs1007 ATM)

That is good news, however it appears that the two alerts are false positives. I dug into it a bit and it appears both drives are virtual drives exposed by the idrac:

root@aqs1007:~# lsblk -S /dev/{sdm,sdh}
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sdh  10:0:0:0   disk iDRAC    MAS001           0329 usb
sdm  11:0:0:0   disk iDRAC    MAS025           0329 usb

I then went digging in the Prometheus smartmon script, https://github.com/prometheus-community/node-exporter-textfile-collector-scripts/blob/master/smartmon.py, only to discover that we wrote our own.

However neither theirs nor ours seems to provide a way to skip a certain vendor or model of a drive.

I think we should augment the script to query the vendor via /sys/, i.e. cat /sys/block/sdm/device/vendor in addition to or instead of querying via smartctl, so that we can exclude idrac devices.

+1 to exclude list, thank you for digging out the root cause. Just as an historical/contextual note I can't remember at the minute why we went with our implementation of smartmon.py (it is possible it wasn't available at the time though). Having said that, nowadays it might make sense to move to upstream' smartmon (100% out of scope for this task, but putting it out there)

The other thing to note is that aqs1007 and its idrac devices seem definitely a mis-setup of idrac / unexpected situation (e.g. aqs1008 doesn't have the idrac disks) and the host is pending decom (according to T304938, I can't find the decom task itself).

gerritbot added a comment.Apr 14 2022, 7:20 PM

Change 780990 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] smart_data_dump: skip over iDRAC devices

https://gerrit.wikimedia.org/r/780990

gerritbot added a project: Patch-For-Review.Apr 14 2022, 7:20 PM
jhathaway added a comment.Apr 14 2022, 7:24 PM

+1 to exclude list, thank you for digging out the root cause. Just as an historical/contextual note I can't remember at the minute why we went with our implementation of smartmon.py (it is possible it wasn't available at the time though). Having said that, nowadays it might make sense to move to upstream' smartmon (100% out of scope for this task, but putting it out there)

I added a patch to exclude iDRAC devices. It appears our code predates upstream's code and it looks like it would take a fair bit of work to the upstream code in order to reach feature parity with ours.

The other thing to note is that aqs1007 and its idrac devices seem definitely a mis-setup of idrac / unexpected situation (e.g. aqs1008 doesn't have the idrac disks) and the host is pending decom (according to T304938, I can't find the decom task itself).

Good point, however I suspect we may see these devices pop up from time to time.

gerritbot added a comment.Apr 19 2022, 7:48 PM

Change 780990 merged by JHathaway:

[operations/puppet@production] smart_data_dump: skip over iDRAC devices

https://gerrit.wikimedia.org/r/780990

Maintenance_bot removed a project: Patch-For-Review.Apr 19 2022, 8:31 PM
gerritbot added a comment.Apr 25 2022, 9:07 PM

Change 785921 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] icinga: remove SMART check

https://gerrit.wikimedia.org/r/785921

gerritbot added a project: Patch-For-Review.Apr 25 2022, 9:07 PM
gerritbot added a comment.Apr 26 2022, 5:44 PM

Change 785921 merged by JHathaway:

[operations/puppet@production] icinga: remove SMART check

https://gerrit.wikimedia.org/r/785921

Maintenance_bot removed a project: Patch-For-Review.Apr 26 2022, 6:31 PM
fgiunchedi edited parent tasks, added: T288622: All Prometheus based alerts move from Icinga to alert manager exclusively; removed: T281454: Onboard teams with Prometheus-based alerts to AlertManager.Jul 11 2022, 12:09 PM
fgiunchedi updated the task description. (Show Details)Jul 11 2022, 12:24 PM

I'm auditing puppet.git for candidate Prometheus-based alerts and realized the list in the task description was missing the puppet-related alerts (added now)

jbond added projects: SRE-Sprint-Week-Sustainability-March2023, Infrastructure-Foundations.Mar 21 2023, 9:53 AM
jbond updated the task description. (Show Details)
gerritbot added a comment.Mar 21 2023, 10:13 AM

Change 901546 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:monitoring: add docs and fix liniting errors

https://gerrit.wikimedia.org/r/901546

gerritbot added a project: Patch-For-Review.Mar 21 2023, 10:13 AM
gerritbot added a comment.Mar 21 2023, 10:16 AM

Change 901546 merged by Jbond:

[operations/puppet@production] P:monitoring: add docs and fix liniting errors

https://gerrit.wikimedia.org/r/901546

Joe removed a project: SRE-Sprint-Week-Sustainability-March2023.Mar 21 2023, 10:26 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 21 2023, 10:32 AM
gerritbot added a comment.Mar 21 2023, 11:25 AM

Change 901557 had a related patch set uploaded (by Jbond; author: jbond):

[operations/alerts@master] hardware: add Memory correctable errors (EDAC) alert

https://gerrit.wikimedia.org/r/901557

gerritbot added a project: Patch-For-Review.Mar 21 2023, 11:25 AM
gerritbot added a comment.Mar 21 2023, 1:33 PM

Change 901557 abandoned by Jbond:

[operations/alerts@master] hardware: add Memory correctable errors (EDAC) alert

Reason:

will create a new test based on impi data T302639

https://gerrit.wikimedia.org/r/901557

jbond changed the status of subtask T302639: How should we monitor for faulty memory modules? from Open to In Progress.Mar 21 2023, 1:50 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 21 2023, 2:12 PM
gerritbot added a comment.Mar 22 2023, 3:22 PM

Change 902110 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:monitoring: drop check for filesystem_avail_bigger_than_size

https://gerrit.wikimedia.org/r/902110

gerritbot added a project: Patch-For-Review.Mar 22 2023, 3:22 PM
gerritbot added a comment.Mar 22 2023, 4:37 PM

Change 902110 merged by Jbond:

[operations/puppet@production] P:monitoring: drop check for filesystem_avail_bigger_than_size

https://gerrit.wikimedia.org/r/902110

jbond updated the task description. (Show Details)Mar 22 2023, 4:38 PM
Maintenance_bot removed a project: Patch-For-Review.Mar 22 2023, 5:10 PM
jbond changed the task status from Open to In Progress.Mar 23 2023, 10:52 AM
jbond claimed this task.
jbond added a project: SRE-Sprint-Week-Sustainability-March2023.
jbond removed a project: SRE-Sprint-Week-Sustainability-March2023.
gerritbot added a comment.Mar 23 2023, 11:25 AM

Change 902323 had a related patch set uploaded (by Jbond; author: jbond):

[operations/alerts@master] team-sre/puppet-agent: Add widespread puppet failure alert

https://gerrit.wikimedia.org/r/902323

gerritbot added a project: Patch-For-Review.Mar 23 2023, 11:25 AM
gerritbot added a comment.Mar 23 2023, 11:59 AM

Change 902348 had a related patch set uploaded (by Jbond; author: jbond):

[operations/alerts@master] team-sre/puppet-agent: Add widespread puppet failure (no resources) alert

https://gerrit.wikimedia.org/r/902348

jbond added a comment.Mar 23 2023, 2:07 PM

changes pending for the two widespread-puppet checks

gerritbot added a comment.Mar 27 2023, 10:05 AM

Change 902323 merged by jenkins-bot:

[operations/alerts@master] team-sre/puppet-agent: Add widespread puppet failure alert

https://gerrit.wikimedia.org/r/902323

gerritbot added a comment.Mar 27 2023, 10:05 AM

Change 902348 merged by Jbond:

[operations/alerts@master] team-sre/puppet-agent: Add widespread puppet failure (no resources) alert

https://gerrit.wikimedia.org/r/902348

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2023, 10:10 AM
gerritbot added a comment.Mar 28 2023, 3:20 PM

Change 903687 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] sre: introduce cluster vs site wide puppet failures

https://gerrit.wikimedia.org/r/903687

gerritbot added a project: Patch-For-Review.Mar 28 2023, 3:20 PM
gerritbot added a comment.Mar 28 2023, 4:50 PM

Change 903687 merged by Jbond:

[operations/alerts@master] sre: introduce cluster vs site wide puppet failures

https://gerrit.wikimedia.org/r/903687

Maintenance_bot removed a project: Patch-For-Review.Mar 28 2023, 5:10 PM
gerritbot added a comment.Mar 29 2023, 1:46 PM

Change 904182 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] profile: remove edac alerts

https://gerrit.wikimedia.org/r/904182

gerritbot added a project: Patch-For-Review.Mar 29 2023, 1:46 PM
gerritbot added a comment.Mar 29 2023, 3:17 PM

Change 904182 merged by Filippo Giunchedi:

[operations/puppet@production] profile: remove edac alerts

https://gerrit.wikimedia.org/r/904182

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2023, 3:31 PM
fgiunchedi updated the task description. (Show Details)Mar 30 2023, 12:55 PM
jbond closed this task as Resolved.Mar 30 2023, 1:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL