Maniphest T380572

SUP: Reduce Metrics
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	pfischer
	Nov 22 2024, 10:29 AM

Project Tags

Referenced Files

None

Subscribers

Description

We saw out of memory issues related to metrics last week. We want to reduce this cost to ensure stability.

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	cirrus: Update container image	Ebernhardson	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Related Objects

Mentioned In: T399721: SUP: OOM from prometheus

Event Timeline

pfischer created this task.Nov 22 2024, 10:29 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 22 2024, 10:29 AM

Gehel triaged this task as Medium priority.Nov 25 2024, 4:20 PM

Gehel updated the task description. (Show Details)

Gehel edited projects, added: Discovery-Search (Current work); removed: Discovery-Search.Jan 6 2025, 4:19 PM

TJones set the point value for this task to 3.Jan 6 2025, 4:56 PM

TJones moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.

bking subscribed.Jan 13 2025, 4:35 PM

Gehel edited projects, added: Discovery-Search (2025.02.10 - 2025.02.28); removed: Discovery-Search (Current work).Feb 11 2025, 2:19 PM

Gehel moved this task from Incoming to In Progress on the Discovery-Search (2025.02.10 - 2025.02.28) board.

Gehel edited projects, added: Discovery-Search (2025.03.01 - 2025.03.21); removed: Discovery-Search (2025.02.10 - 2025.02.28).Feb 28 2025, 1:37 PM

Gehel moved this task from Incoming to In Progress on the Discovery-Search (2025.03.01 - 2025.03.21) board.

Gehel moved this task from In Progress to Ready for Dev on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 10 2025, 4:37 PM

pfischer claimed this task.Mar 11 2025, 2:47 PM

pfischer moved this task from Ready for Dev to In Progress on the Discovery-Search (2025.03.01 - 2025.03.21) board.

pfischer opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/178

Consumer: reduce HTTP metrics

pfischer moved this task from In Progress to Needs Review on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 12 2025, 4:18 PM

ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/178

Consumer: reduce HTTP metrics

Change #1128505 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):

[operations/deployment-charts@master] cirrus: Update container image

https://gerrit.wikimedia.org/r/1128505

Change #1128505 merged by jenkins-bot:

[operations/deployment-charts@master] cirrus: Update container image

https://gerrit.wikimedia.org/r/1128505

Maintenance_bot removed a project: Patch-For-Review.Mar 17 2025, 7:31 PM

Metrics changes deployed to all consumers (and producer, but it doesn't do anything there afaict). Dashboards for http metrics now show _OTHER as expected.

Thanks! There is no significant effect on task manager memory consumption. CODFW uses almost the same (~2GiB), only EQIAD is now down 100 MiB.

Gehel closed this task as Resolved.Mar 21 2025, 1:57 PM

Gehel moved this task from Done to Reported on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 21 2025, 2:53 PM

dcausse mentioned this in T399721: SUP: OOM from prometheus.Aug 19 2025, 10:02 AM