Page MenuHomePhabricator
Create Task
Maniphest T359033

EPIC: Convert CirrusSearch metrics to statslib
Closed, ResolvedPublic
Actions

Description

Task to track the efforts/progress made on converting CirrusSearch metrics from graphite to prometheus using statslib.

The work to do is:

The work does not have to be done all at once but working on metrics used by icinga should be prioritized according to T350597.

AC:

  • CirrusSearch no longer writes to (nor its operations depend on) graphite/icinga

Metrics initially identified:

  • MediaWiki.CirrusSearch.$cirrus_group.backend_failure.*
  • MediaWiki.CirrusSearch.$cirrus_group.backend_failure.*.rate
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.comp_suggest.p50
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.comp_suggest.p75
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.comp_suggest.p95
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.comp_suggest.p99
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.comp_suggest.sample_rate
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.full_text.p50
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.full_text.p75
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.full_text.p95
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.full_text.p99
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.full_text.sample_rate
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.more_like.p50
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.more_like.p75
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.more_like.p95
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.more_like.sample_rate
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.*.p50
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.prefix.p50
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.prefix.p75
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.prefix.p95
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.prefix.p99
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.prefix.sample_rate
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.*.sample_rate
  • MediaWiki.CirrusSearch.$cirrus_group.requestTimeMs.*.sum
  • MediaWiki.CirrusSearch.$cirrus_group.requestTime.p50
  • MediaWiki.CirrusSearch.$cirrus_group.requestTime.p75
  • MediaWiki.CirrusSearch.$cirrus_group.requestTime.p95
  • MediaWiki.CirrusSearch.$cirrus_group.requestTime.p99
  • MediaWiki.CirrusSearch.$cluster.backend_failure.failed.count
  • MediaWiki.CirrusSearch.$cluster.backend_failure.rejected.count
  • MediaWiki.CirrusSearch.$cluster.backend_failure.unknown.count
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.comp_suggest.p50
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.comp_suggest.p75
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.comp_suggest.p95
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.comp_suggest.p99
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.comp_suggest.sample_rate
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.full_text.p50
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.full_text.p75
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.full_text.p95
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.full_text.p99
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.full_text.sample_rate
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.more_like.p50
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.more_like.p75
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.more_like.p95
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.more_like.sample_rate
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.*.p50
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.prefix.p50
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.prefix.p75
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.prefix.p95
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.prefix.p99
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.prefix.sample_rate
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.*.sample_rate
  • MediaWiki.CirrusSearch.$cluster.requestTimeMs.*.sum
  • MediaWiki.CirrusSearch.$cluster.requestTime.p50
  • MediaWiki.CirrusSearch.$cluster.requestTime.p75
  • MediaWiki.CirrusSearch.$cluster.requestTime.p95
  • MediaWiki.CirrusSearch.$cluster.requestTime.p99
  • MediaWiki.CirrusSearch.$cluster.updates.all.lag.$change_type.mean
  • MediaWiki.CirrusSearch.cloudelastic.updates.all.*.rate
  • MediaWiki.CirrusSearch.codfw.requestTime.p95
  • MediaWiki.CirrusSearch.codfw.updates.all.*.rate
  • MediaWiki.CirrusSearch.codfw.updates.all.sent.rate
  • MediaWiki.CirrusSearch.codfw.updates.details.*.*.sent.rate
  • MediaWiki.CirrusSearch.eqiad.requestTime.p95
  • MediaWiki.CirrusSearch.eqiad.updates.all.lag.page_change.mean
  • MediaWiki.CirrusSearch.eqiad.updates.all.*.rate
  • MediaWiki.CirrusSearch.poolCounter.*
  • MediaWiki.CirrusSearch.poolCounter.$pool_counter.failureMs.sample_rate
  • MediaWiki.CirrusSearch.poolCounter.$pool_counter.successMs.sample_rate
  • MediaWiki.CirrusSearch.poolCounter.*.failureMs.sample_rate
  • MediaWiki.CirrusSearch.query_cache.more_like.hit.rate
  • MediaWiki.CirrusSearch.query_cache.more_like.miss.rate
  • MediaWiki.CirrusSearch.results.file_duplicates.count
  • MediaWiki.CirrusSearch.*.updates.all.doc_size.p95
  • MediaWiki.CirrusSearch.*.updates.all.doc_size.p99
  • MediaWiki.CirrusSearch.*.updates.all.doc_size.upper

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a project: Discovery-Search. · View Herald TranscriptMar 4 2024, 10:05 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
dcausse added a parent task: T343020: Converting MediaWiki Metrics to StatsLib.Mar 4 2024, 10:05 AM
Gehel triaged this task as High priority.Mar 4 2024, 4:28 PM
Gehel moved this task from needs triage to Current work on the Discovery-Search board.
Gehel edited projects, added Discovery-Search (Current work); removed Discovery-Search.
dcausse renamed this task from Convert CirrusSearch metrics to statslib to EPIC: Convert CirrusSearch metrics to statslib.Mar 4 2024, 4:52 PM
dcausse added a project: Epic.
dcausse moved this task from Incoming to Epics on the Discovery-Search (Current work) board.
• lmata awarded a token.Mar 4 2024, 7:39 PM
• lmata added a project: Observability-Metrics.
• lmata moved this task from Inbox to Radar on the Observability-Metrics board.
• lmata updated the task description. (Show Details)Mar 6 2024, 1:16 PM
• lmata mentioned this in T350592: EPIC: migrate in use metrics and dashboards to statslib.Mar 6 2024, 1:31 PM
bking subscribed.Apr 11 2024, 7:09 PM
gerritbot added a comment.Jun 19 2024, 1:52 PM

Change #1047513 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert DataSender metrics to new Stats library

https://gerrit.wikimedia.org/r/1047513

gerritbot added a project: Patch-For-Review.Jun 19 2024, 1:52 PM

Change #1047514 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert ElasticaWrite metrics to new Stats library

https://gerrit.wikimedia.org/r/1047514

gerritbot added a comment.Jun 19 2024, 1:52 PM

Change #1047515 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert Pool Counter metrics to new Stats library

https://gerrit.wikimedia.org/r/1047515

gerritbot added a comment.Jun 19 2024, 1:52 PM

Change #1047516 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert query cache metrics to new Stats library

https://gerrit.wikimedia.org/r/1047516

gerritbot added a comment.Jun 19 2024, 1:52 PM

Change #1047517 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert saneitizer metrics to new Stats library

https://gerrit.wikimedia.org/r/1047517

gerritbot added a comment.Jun 19 2024, 1:52 PM

Change #1047518 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert file duplicate metrics to new Stats library

https://gerrit.wikimedia.org/r/1047518

gerritbot added a comment.Jun 19 2024, 1:52 PM

Change #1047519 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert ElasticsearchIntermediary metrics to new Stats library

https://gerrit.wikimedia.org/r/1047519

gerritbot added a comment.Jun 19 2024, 1:52 PM

Change #1047520 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Convert deepcat sparql timing metrics to new Stats library

https://gerrit.wikimedia.org/r/1047520

gerritbot added a comment.Jun 24 2024, 4:23 PM

Change #1047514 abandoned by DCausse:

[mediawiki/extensions/CirrusSearch@master] Convert ElasticaWrite metrics to new Stats library

Reason:

squashed

https://gerrit.wikimedia.org/r/1047514

gerritbot added a comment.Jun 24 2024, 4:23 PM

Change #1047515 abandoned by DCausse:

[mediawiki/extensions/CirrusSearch@master] Convert Pool Counter metrics to new Stats library

Reason:

squashed

https://gerrit.wikimedia.org/r/1047515

gerritbot added a comment.Jun 24 2024, 4:23 PM

Change #1047516 abandoned by DCausse:

[mediawiki/extensions/CirrusSearch@master] Convert query cache metrics to new Stats library

Reason:

squashed

https://gerrit.wikimedia.org/r/1047516

gerritbot added a comment.Jun 24 2024, 4:24 PM

Change #1047517 abandoned by DCausse:

[mediawiki/extensions/CirrusSearch@master] Convert saneitizer metrics to new Stats library

Reason:

squashed

https://gerrit.wikimedia.org/r/1047517

gerritbot added a comment.Jun 24 2024, 4:24 PM

Change #1047518 abandoned by DCausse:

[mediawiki/extensions/CirrusSearch@master] Convert file duplicate metrics to new Stats library

Reason:

squashed

https://gerrit.wikimedia.org/r/1047518

gerritbot added a comment.Jun 24 2024, 4:24 PM

Change #1047519 abandoned by DCausse:

[mediawiki/extensions/CirrusSearch@master] Convert ElasticsearchIntermediary metrics to new Stats library

Reason:

squashed

https://gerrit.wikimedia.org/r/1047519

gerritbot added a comment.Jun 24 2024, 4:24 PM

Change #1047520 abandoned by DCausse:

[mediawiki/extensions/CirrusSearch@master] Convert deepcat sparql timing metrics to new Stats library

Reason:

squashed

https://gerrit.wikimedia.org/r/1047520

dr0ptp4kt subscribed.Jun 24 2024, 6:16 PM
bking mentioned this in T368107: DPE SRE: Increase visibility of Search Platform alerts .Jun 24 2024, 10:53 PM
gerritbot added a comment.Jun 25 2024, 7:42 PM

Change #1047513 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Convert CirrusSearch metrics to new Stats library

https://gerrit.wikimedia.org/r/1047513

ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.12; 2024-07-02).Jun 25 2024, 8:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jun 25 2024, 8:31 PM
gerritbot added a comment.Jul 3 2024, 8:05 AM

Change #1051682 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Fix naming of new prometheus metrics

https://gerrit.wikimedia.org/r/1051682

gerritbot added a project: Patch-For-Review.Jul 3 2024, 8:05 AM
gerritbot added a comment.Jul 8 2024, 5:39 PM

Change #1051682 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Fix naming of new prometheus metrics

https://gerrit.wikimedia.org/r/1051682

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.13; 2024-07-09); removed MW-1.43-notes (1.43.0-wmf.12; 2024-07-02).Jul 8 2024, 6:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jul 9 2024, 7:55 PM
gerritbot added a comment.Jul 12 2024, 2:23 AM

Change #1053825 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@master] Re-add CirrusSearch prefix to statsd metrics

https://gerrit.wikimedia.org/r/1053825

gerritbot added a project: Patch-For-Review.Jul 12 2024, 2:23 AM
gerritbot added a comment.Jul 12 2024, 7:22 AM

Change #1053838 had a related patch set uploaded (by DCausse; author: Ebernhardson):

[mediawiki/extensions/CirrusSearch@wmf/1.43.0-wmf.13] Re-add CirrusSearch prefix to statsd metrics

https://gerrit.wikimedia.org/r/1053838

gerritbot added a comment.Jul 12 2024, 7:50 AM

Change #1053825 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Re-add CirrusSearch prefix to statsd metrics

https://gerrit.wikimedia.org/r/1053825

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.14; 2024-07-16); removed MW-1.43-notes (1.43.0-wmf.13; 2024-07-09).Jul 12 2024, 8:00 AM
gerritbot added a comment.Jul 12 2024, 9:04 AM

Change #1053838 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.43.0-wmf.13] Re-add CirrusSearch prefix to statsd metrics

https://gerrit.wikimedia.org/r/1053838

Stashbot added a comment.Jul 12 2024, 9:10 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-12T09:10:56Z] <dcausse@deploy1002> Started scap sync-world: Backport for [[gerrit:1053838|Re-add CirrusSearch prefix to statsd metrics (T359033)]]

Stashbot added a comment.Jul 12 2024, 9:13 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-12T09:13:27Z] <dcausse@deploy1002> dcausse: Backport for [[gerrit:1053838|Re-add CirrusSearch prefix to statsd metrics (T359033)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Jul 12 2024, 9:20 AM

Mentioned in SAL (#wikimedia-operations) [2024-07-12T09:20:40Z] <dcausse@deploy1002> Finished scap: Backport for [[gerrit:1053838|Re-add CirrusSearch prefix to statsd metrics (T359033)]] (duration: 09m 44s)

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.13; 2024-07-09); removed MW-1.43-notes (1.43.0-wmf.14; 2024-07-16).Jul 12 2024, 10:00 AM
gerritbot added a comment.Jul 15 2024, 11:57 AM

Change #1054317 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/alerts@master] team-search-platform: migrate cirrus_cluster_checks

https://gerrit.wikimedia.org/r/1054317

Gehel added a project: Data-Platform-SRE (2024.07.08 - 2024.07.28).Jul 15 2024, 3:15 PM
gerritbot added a comment.Jul 15 2024, 4:48 PM

Change #1054374 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/alerts@master] team-search-platform: migrate cirrus latencies & mem alert

https://gerrit.wikimedia.org/r/1054374

gerritbot added a comment.Jul 16 2024, 7:01 PM

Change #1054647 had a related patch set uploaded (by Bking; author: Bking):

[operations/puppet@production] elasticsearch: remove obsolete alerts

https://gerrit.wikimedia.org/r/1054647

gerritbot added a comment.Jul 23 2024, 7:45 AM

Change #1054317 merged by jenkins-bot:

[operations/alerts@master] team-search-platform: migrate cirrus_cluster_checks

https://gerrit.wikimedia.org/r/1054317

gerritbot added a comment.Jul 23 2024, 4:54 PM

Change #1054374 merged by jenkins-bot:

[operations/alerts@master] team-search-platform: migrate cirrus latencies & mem alert

https://gerrit.wikimedia.org/r/1054374

gerritbot added a comment.Jul 25 2024, 1:02 PM

Change #1054647 merged by Bking:

[operations/puppet@production] elasticsearch: remove obsolete alerts

https://gerrit.wikimedia.org/r/1054647

Maintenance_bot removed a project: Patch-For-Review.Jul 25 2024, 1:31 PM
bking edited projects, added Data-Platform-SRE (2024.07.29 - 2024.08.16); removed Data-Platform-SRE (2024.07.08 - 2024.07.28).Jul 31 2024, 2:43 PM
Gehel edited projects, added Data-Platform-SRE (2024.08.17 - 2024.09.06); removed Data-Platform-SRE (2024.07.29 - 2024.08.16).Aug 16 2024, 9:45 AM
fgiunchedi mentioned this in T357614: Alert in need of triage: Number of requests triggering circuit breakers due to excessive memory usage (instance graphite1005).Aug 21 2024, 9:11 AM
fgiunchedi mentioned this in T355795: Fix "requests triggering circuit breakers" Elastic alert.Aug 21 2024, 9:44 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2024.08.17 - 2024.09.06) board.Sep 3 2024, 3:17 PM
Gehel edited projects, added Data-Platform-SRE (2024.09.06 - 2024.09.27); removed Data-Platform-SRE (2024.08.17 - 2024.09.06).Sep 6 2024, 9:49 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2024.09.06 - 2024.09.27) board.
bking mentioned this in T372902: Find/replace Graphite metrics with Prometheus metrics.Sep 6 2024, 8:17 PM
Gehel edited projects, added Data-Platform-SRE (2024.09.28 - 2024.10.18); removed Data-Platform-SRE (2024.09.06 - 2024.09.27).Sep 27 2024, 1:00 PM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2024.09.28 - 2024.10.18) board.
BTullis edited projects, added Data-Platform-SRE (2024.10.19 - 2024.11.08); removed Data-Platform-SRE (2024.09.28 - 2024.10.18).Oct 18 2024, 3:09 PM
BTullis moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2024.10.19 - 2024.11.08) board.
Gehel edited projects, added Data-Platform-SRE (2024.11.09 - 2024.11.29); removed Data-Platform-SRE (2024.10.19 - 2024.11.08).Nov 8 2024, 9:48 AM
Gehel moved this task from Backlog - project to Backlog - operations on the Data-Platform-SRE (2024.11.09 - 2024.11.29) board.
Gehel edited projects, added Data-Platform-SRE; removed Data-Platform-SRE (2024.11.09 - 2024.11.29).Nov 12 2024, 4:26 PM
Gehel moved this task from Incoming to Quarterly Goals on the Data-Platform-SRE board.
bking closed subtask T371061: Update CirrusSearch dashboards to use new metrics/refresh dashboards as Resolved.Dec 4 2024, 3:38 PM
dcausse moved this task from Epics to To Be Deployed on the Discovery-Search (Current work) board.Jan 21 2025, 10:46 AM

Starting from MW 1.44.0-wmf.14 CirrusSearch should no longer push any metrics to graphite.

dr0ptp4kt unsubscribed.Jan 21 2025, 6:35 PM
• lmata subscribed.Jan 22 2025, 7:38 PM
In T359033#10478849, @dcausse wrote:

Starting from MW 1.44.0-wmf.14 CirrusSearch should no longer push any metrics to graphite.

\o/ thanks!

Gehel moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Feb 10 2025, 4:28 PM
Gehel edited projects, added Data-Platform-SRE (2025.02.10 - 2025.02.28); removed Data-Platform-SRE.Feb 11 2025, 2:06 PM
Gehel moved this task from Backlog - project to Done on the Data-Platform-SRE (2025.02.10 - 2025.02.28) board.
Gehel edited projects, added Discovery-Search (2025.02.10 - 2025.02.28); removed Discovery-Search (Current work).Feb 11 2025, 2:17 PM
Gehel moved this task from Incoming to Done on the Discovery-Search (2025.02.10 - 2025.02.28) board.
dcausse closed this task as Resolved.Feb 11 2025, 2:36 PM
dcausse claimed this task.
Gehel moved this task from Done to Reported on the Data-Platform-SRE (2025.02.10 - 2025.02.28) board.Feb 14 2025, 8:22 AM
Gehel moved this task from Done to Reported on the Discovery-Search (2025.02.10 - 2025.02.28) board.Feb 14 2025, 8:58 AM
Gehel closed subtask T369148: Replace usage of StatsdDataFactory with StatsFactory as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits