Page MenuHomePhabricator
Create Task
Maniphest T328941

followups to unactionable NELHigh pages due to Telecom Italia outage, 2023-02-05
Open, MediumPublic
Actions

Description

In the event of long-duration issues in particular geographies, it would be nice if we could exclude those reports from the NEL metric used for alerting.

  • edit es_exporter config to also aggregate by country in addition to type -- it seems best to create a new metric aggregated this way
  • edit alertmanager rule to use the new metric
  • edit statograph configuration to use the new metric (verifying that the results of the query are the same as before)

Details

SubjectRepoBranchLines +/-
NEL: add alert by countryoperations/alertsmaster+70 -0
es_exporter: add NEL metrics by countryoperations/puppetproduction+39 -0
Customize query in gerrit

Event Timeline

CDanis created this task.Feb 6 2023, 5:16 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 6 2023, 5:16 PM
CDanis updated the task description. (Show Details)Feb 8 2023, 6:18 PM
CDanis added a project: Sustainability (Incident Followup).Feb 8 2023, 6:22 PM
LSobanski added a project: SRE-OnFire.Feb 16 2023, 12:50 PM
akosiaris removed a project: SRE.Mar 14 2023, 4:39 PM
akosiaris subscribed.

Removing SRE as the more specific working group is tagged already.

gerritbot added a comment.Mar 20 2023, 2:30 PM

Change 901220 had a related patch set uploaded (by Volans; author: Volans):

[operations/puppet@production] es_exporter: add NEL metrics by country

https://gerrit.wikimedia.org/r/901220

gerritbot added a project: Patch-For-Review.Mar 20 2023, 2:30 PM
Volans triaged this task as Medium priority.Mar 20 2023, 2:33 PM
Volans added a project: SRE-Sprint-Week-Sustainability-March2023.
Volans moved this task from Backlog to Doing on the SRE-Sprint-Week-Sustainability-March2023 board.
Volans claimed this task.Mar 21 2023, 10:34 AM
Volans added a comment.Mar 23 2023, 9:22 AM

In the meanwhile I've also added 2 graphs in the superset dashboards in the Geo tab with the number of NELs by Country and NELs by ISP. Hope that will be helpful.

gerritbot added a comment.Mar 23 2023, 9:53 AM

Change 901220 merged by Volans:

[operations/puppet@production] es_exporter: add NEL metrics by country

https://gerrit.wikimedia.org/r/901220

Volans added a comment.Mar 23 2023, 10:03 AM

Metrics are now being ingested by prometheus:

image.png (1×2 px, 480 KB)

Volans updated the task description. (Show Details)Mar 23 2023, 10:03 AM
Maintenance_bot removed a project: Patch-For-Review.Mar 23 2023, 10:10 AM
gerritbot added a comment.Mar 23 2023, 10:41 AM

Change 902316 had a related patch set uploaded (by Volans; author: Volans):

[operations/alerts@master] NEL: add alert by country

https://gerrit.wikimedia.org/r/902316

gerritbot added a project: Patch-For-Review.Mar 23 2023, 10:41 AM
Volans removed Volans as the assignee of this task.Mar 23 2023, 6:47 PM
Volans subscribed.

Removing myself from assignee as the sprint week effort ended. The above patch is pending review.

Volans moved this task from Doing to Backlog on the SRE-Sprint-Week-Sustainability-March2023 board.Mar 23 2023, 6:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL