Page MenuHomePhabricator
Create Task
Maniphest T350597

Audit and prioritize metrics for conversion to statslib that are used for graphite-based alerting
Closed, ResolvedPublic
Actions

Description

This task tracks the conversion of mw metrics used for graphite-based alerting in Icinga.

modules/icinga/manifests/monitor/elasticsearch/cirrus_cluster_checks.pp

AM patch: https://gerrit.wikimedia.org/r/c/operations/alerts/+/1054317/2
Patch to remove from puppet: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1054647

  • mediawiki_cirrus_update_rate_${site}
  • mediawiki_cirrus_pool_counter_rejections_rate
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.poolCounter.*.failureMs.sample_rate
  • mediawiki_cirrussearch_indices_high_fix_rate
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.{eqiad,codfw,cloudelastic}.sanitization.fixed.sum
modules/profile/manifests/graphite/alerts.pp
  • mediawiki_session_loss
    • Core: EditPage->incrementEditFailureStats()
    • MediaWiki.edit.failures.session_loss.rate
  • mediawiki_bad_token
    • Core: EditPage->incrementEditFailureStats()
    • MediaWiki.edit.failures.bad_token.rate
  • mediawiki_centralauth_errors
  • mediawiki_accountcreation_errors
modules/role/manifests/elasticsearch/alerts.pp

AM patch: https://gerrit.wikimedia.org/r/c/operations/alerts/+/1054374/
Puppet cleanup patch: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1054647

  • cirrussearch_eqiad_fulltext_95th_percentile
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.eqiad.requestTimeMs.full_text.p95
  • cirrussearch_eqiad_compsuggest_95th_percentile
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.eqiad.requestTimeMs.comp_suggest.p95
    • MediaWiki.CirrusSearch.eqiad.requestTimeMs.comp_suggest.sample_rate
  • cirrussearch_eqiad_morelike_95th_percentile
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.eqiad.requestTimeMs.more_like.p95
  • cirrussearch_codfw_fulltext_95th_percentile
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.codfw.requestTimeMs.full_text.p95
  • cirrussearch_codfw_compsuggest_95th_percentile
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.codfw.requestTimeMs.comp_suggest.p95
    • MediaWiki.CirrusSearch.codfw.requestTimeMs.comp_suggest.sample_rate
  • cirrussearch_codfw_morelike_95th_percentile
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.codfw.requestTimeMs.more_like.p95
  • search_backend_failure_count (related: T355795: Fix "requests triggering circuit breakers" Elastic alert) Using envoy telemetry now
    • CirrusSearch Extension
    • MediaWiki.CirrusSearch.eqiad.backend_failure.failed.count

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
team-sre: tweak MediaWikiLoginFailures thresholdoperations/alertsmaster+1 -1
graphite: remove mw graphite-based alertsoperations/puppetproduction+52 -107
mediawiki: port account creation failures alert from icinga/statsdoperations/alertsmaster+41 -0
mediawiki: port login failures alert from icinga/statsdoperations/alertsmaster+37 -2
AuthManager: increment Stats counters toomediawiki/extensions/WikimediaEventsmaster+69 -24
sre: move MediaWikiEditFailures alert to globaloperations/alertsmaster+43 -34
graphite: remove mw edit failures graphite alertsoperations/puppetproduction+0 -27
sre: add mw edit failures alertoperations/alertsmaster+34 -0
EditPage.php: convert edit failures count to new Stats librarymediawiki/coremaster+5 -2
Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Nov 6 2023, 4:21 PM
fgiunchedi updated the task description. (Show Details)Nov 6 2023, 4:39 PM
colewhite updated the task description. (Show Details)Nov 6 2023, 11:32 PM
gerritbot added a comment.Nov 7 2023, 11:50 AM

Change 972356 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[mediawiki/core@master] EditPage.php: convert edit failures count to new Stats library

https://gerrit.wikimedia.org/r/972356

gerritbot added a project: Patch-For-Review.Nov 7 2023, 11:50 AM
gerritbot added a comment.Jan 8 2024, 4:24 PM

Change 972356 merged by jenkins-bot:

[mediawiki/core@master] EditPage.php: convert edit failures count to new Stats library

https://gerrit.wikimedia.org/r/972356

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2024, 4:31 PM
ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.13; 2024-01-09 ).Jan 8 2024, 5:00 PM
• lmata edited projects, added SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).Jan 12 2024, 5:54 PM
gerritbot added a comment.Jan 16 2024, 1:28 PM

Change 991007 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] sre: add mw edit failures alert

https://gerrit.wikimedia.org/r/991007

gerritbot added a project: Patch-For-Review.Jan 16 2024, 1:28 PM
gerritbot added a comment.Jan 16 2024, 1:30 PM

Change 991008 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: remove mw edit failures graphite alerts

https://gerrit.wikimedia.org/r/991008

gerritbot added a comment.Jan 24 2024, 9:36 AM

Change 991007 merged by Filippo Giunchedi:

[operations/alerts@master] sre: add mw edit failures alert

https://gerrit.wikimedia.org/r/991007

gerritbot added a comment.Jan 24 2024, 9:36 AM

Change 991008 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: remove mw edit failures graphite alerts

https://gerrit.wikimedia.org/r/991008

Maintenance_bot removed a project: Patch-For-Review.Jan 24 2024, 10:30 AM
gerritbot added a comment.Jan 29 2024, 9:41 AM

Change 993661 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] sre: move MediaWikiEditFailures alert to global

https://gerrit.wikimedia.org/r/993661

gerritbot added a project: Patch-For-Review.Jan 29 2024, 9:41 AM
gerritbot added a comment.Jan 29 2024, 10:49 AM

Change 993661 merged by Filippo Giunchedi:

[operations/alerts@master] sre: move MediaWikiEditFailures alert to global

https://gerrit.wikimedia.org/r/993661

Maintenance_bot removed a project: Patch-For-Review.Jan 29 2024, 11:30 AM
fgiunchedi updated the task description. (Show Details)Jan 30 2024, 10:56 AM
gerritbot added a comment.Jan 30 2024, 3:10 PM

Change 994185 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[mediawiki/extensions/WikimediaEvents@master] AuthManager: increment Stats counters too

https://gerrit.wikimedia.org/r/994185

gerritbot added a project: Patch-For-Review.Jan 30 2024, 3:10 PM
fgiunchedi updated the task description. (Show Details)Jan 30 2024, 3:14 PM
fgiunchedi added a project: User-fgiunchedi.Jan 31 2024, 1:54 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.
fgiunchedi updated the task description. (Show Details)Jan 31 2024, 3:22 PM
gerritbot added a comment.Feb 21 2024, 4:46 PM

Change 994185 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] AuthManager: increment Stats counters too

https://gerrit.wikimedia.org/r/994185

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2024, 5:30 PM
ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.20; 2024-02-27); removed MW-1.42-notes (1.42.0-wmf.13; 2024-01-09 ).Feb 22 2024, 12:02 PM
Gehel added projects: Data-Platform-SRE, Discovery-Search.Feb 23 2024, 9:15 AM
Gehel moved this task from Incoming to Observability on the Data-Platform-SRE board.Feb 23 2024, 9:20 AM
bking subscribed.Feb 23 2024, 2:38 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.Feb 26 2024, 2:21 PM
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
fgiunchedi mentioned this in T355795: Fix "requests triggering circuit breakers" Elastic alert.Feb 29 2024, 3:13 PM
dcausse subscribed.Feb 29 2024, 3:24 PM
dcausse mentioned this in T359033: EPIC: Convert CirrusSearch metrics to statslib.Mar 4 2024, 10:05 AM
Gehel moved this task from Incoming to Blocked/Waiting on the Discovery-Search (Current work) board.Mar 4 2024, 4:35 PM
RKemper subscribed.Apr 3 2024, 4:39 PM
colewhite edited projects, added SRE Observability (FY2023/2024-Q4); removed SRE Observability (FY2023/2024-Q3).Apr 3 2024, 11:06 PM
colewhite moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q4) board.Apr 3 2024, 11:09 PM
colewhite mentioned this in T359264: Migrate MediaWiki.authmanager.* to statslib.Apr 4 2024, 9:49 PM
Gehel triaged this task as High priority.May 6 2024, 3:27 PM
Gehel moved this task from Blocked/Waiting to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
dcausse updated the task description. (Show Details)May 6 2024, 3:29 PM
• lmata updated the task description. (Show Details)May 13 2024, 8:00 PM
• lmata updated the task description. (Show Details)May 13 2024, 8:03 PM
andrea.denisse subscribed.May 15 2024, 2:21 PM
dcausse moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.Jun 24 2024, 7:33 AM
dcausse updated the task description. (Show Details)Jul 2 2024, 2:16 PM
colewhite edited projects, added SRE Observability (FY2024/2025-Q1); removed SRE Observability (FY2023/2024-Q4).Jul 2 2024, 6:01 PM
Gehel moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Jul 22 2024, 3:24 PM
dcausse updated the task description. (Show Details)Jul 22 2024, 3:29 PM
dr0ptp4kt assigned this task to EBernhardson.Jul 29 2024, 3:13 PM
bking added a subtask: T371061: Update CirrusSearch dashboards to use new metrics/refresh dashboards.Jul 30 2024, 8:26 PM
EBernhardson reassigned this task from EBernhardson to fgiunchedi.Jul 31 2024, 2:48 PM
EBernhardson updated the task description. (Show Details)
EBernhardson moved this task from Needs review to Needs Reporting on the Discovery-Search (Current work) board.
EBernhardson subscribed.

Not sure where to assign this, but from the search side our alerts are all moved over. The centralauth and account creation alerts appear to still exist in puppet.

fgiunchedi removed projects: Discovery-Search (Current work), Data-Platform-SRE, MW-1.42-notes (1.42.0-wmf.20; 2024-02-27).Aug 1 2024, 8:53 AM

Thank you @EBernhardson and team for this! I'll untag non-o11y teams now since we have to migrate centralauth alerts only at this point

gerritbot added a comment.Sep 6 2024, 10:21 AM

Change #1071161 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] mediawiki: port login failures alert from icinga/statsd

https://gerrit.wikimedia.org/r/1071161

gerritbot added a project: Patch-For-Review.Sep 6 2024, 10:21 AM
gerritbot added a comment.Sep 6 2024, 10:54 AM

Change #1071165 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] mediawiki: port account creation failures alert from icinga/statsd

https://gerrit.wikimedia.org/r/1071165

gerritbot added a comment.Sep 6 2024, 12:45 PM

Change #1071193 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] graphite: remove mw graphite-based alerts

https://gerrit.wikimedia.org/r/1071193

gerritbot added a comment.Sep 9 2024, 2:18 PM

Change #1071161 merged by Filippo Giunchedi:

[operations/alerts@master] mediawiki: port login failures alert from icinga/statsd

https://gerrit.wikimedia.org/r/1071161

gerritbot added a comment.Sep 9 2024, 2:18 PM

Change #1071165 merged by Filippo Giunchedi:

[operations/alerts@master] mediawiki: port account creation failures alert from icinga/statsd

https://gerrit.wikimedia.org/r/1071165

fgiunchedi added a subscriber: Clement_Goubert.Sep 9 2024, 2:29 PM

I have merged the ported mw alerts (thank you @Clement_Goubert !) and changed the referenced dashboard at https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts to show the prometheus metrics.

Once https://gerrit.wikimedia.org/r/c/operations/puppet/+/1071193 ships we can call this task done!

gerritbot added a comment.Sep 10 2024, 10:36 AM

Change #1071193 merged by Filippo Giunchedi:

[operations/puppet@production] graphite: remove mw graphite-based alerts

https://gerrit.wikimedia.org/r/1071193

fgiunchedi closed this task as Resolved.Sep 10 2024, 10:50 AM
fgiunchedi updated the task description. (Show Details)

This is done! The only other use of graphite_threshold is 'zuul_gearman_wait_queue' which will be addressed as part of T233089: Export zuul metrics to Prometheus. There are of course graphite-itself alerts left, which will be removed together with graphite.

Maintenance_bot removed a project: Patch-For-Review.Sep 10 2024, 11:31 AM
• lmata awarded a token.Sep 10 2024, 1:10 PM
gerritbot added a comment.Sep 13 2024, 6:43 AM

Change #1072657 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] team-sre: tweak MediaWikiLoginFailures threshold

https://gerrit.wikimedia.org/r/1072657

gerritbot added a project: Patch-For-Review.Sep 13 2024, 6:43 AM
gerritbot added a comment.Sep 13 2024, 12:27 PM

Change #1072657 merged by Filippo Giunchedi:

[operations/alerts@master] team-sre: tweak MediaWikiLoginFailures threshold

https://gerrit.wikimedia.org/r/1072657

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2024, 12:30 PM
• lmata moved this task from Inbox to Done on the SRE Observability (FY2024/2025-Q1) board.Nov 5 2024, 5:02 PM
bking closed subtask T371061: Update CirrusSearch dashboards to use new metrics/refresh dashboards as Resolved.Dec 4 2024, 3:38 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits