Page MenuHomePhabricator
Create Task
Maniphest T406926

Investigate potential memory leak
Closed, ResolvedPublicSpike
Actions

Description

What:
We have observed signs of potential memory leak in the Orchestrator. We need to investigate to try and confirm if a leak indeed exists. If so, identify root causes and next steps for mitigation.

Why:

  • memory usage increase over time and has ended frequently in service restarts or OOM kills in Prod logs
  • performance struggles with response delays seemingly correlated with uptime in user experience

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.Cory Massarooperations/deployment-chartsmaster+1 -1
wikifunctions: Upgrade orchestrator from 2025-10-06-225918 to 2025-10-14-194525Jforresteroperations/deployment-chartsmaster+1 -1
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
make sure that heap logs have req-id'srepos/abstract-wiki/wikifunctions/function-orchestrator!466ecarggrace/T406926/add-request-ids-to-heap-logsmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

ecarg created this task.Oct 9 2025, 8:36 PM
Restricted Application changed the subtype of this task from "Task" to "Spike". · View Herald TranscriptOct 9 2025, 8:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ecarg changed the task status from Open to In Progress.Oct 9 2025, 8:37 PM
ecarg triaged this task as High priority.
CodeReviewBot added a project: Patch-For-Review.Oct 9 2025, 9:17 PM

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/466

make sure that heap logs have req-id's

Reedy added a project: Performance Issue.Oct 10 2025, 1:07 AM
CodeReviewBot added a comment.Oct 10 2025, 1:41 PM

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/466

make sure that heap logs have req-id's

Jdforrester-WMF moved this task from Incoming to In Engineering on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.Oct 10 2025, 1:43 PM
Maintenance_bot removed a project: Patch-For-Review.Oct 10 2025, 2:31 PM
gerritbot added a comment.Oct 15 2025, 1:29 PM

Change #1196439 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-06-225918 to 2025-10-14-194525

https://gerrit.wikimedia.org/r/1196439

gerritbot added a project: Patch-For-Review.Oct 15 2025, 1:29 PM
gerritbot added a comment.Oct 15 2025, 2:02 PM

Change #1196439 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-10-06-225918 to 2025-10-14-194525

https://gerrit.wikimedia.org/r/1196439

Maintenance_bot removed a project: Patch-For-Review.Oct 15 2025, 2:31 PM
gerritbot added a comment.Oct 22 2025, 12:25 PM

Change #1198036 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[operations/deployment-charts@master] Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.

https://gerrit.wikimedia.org/r/1198036

gerritbot added a project: Patch-For-Review.Oct 22 2025, 12:25 PM
gerritbot added a comment.Oct 22 2025, 2:13 PM

Change #1198036 merged by jenkins-bot:

[operations/deployment-charts@master] Wikifunctions: Upgrade orchestrator from 2025-10-14-194525 to 2025-10-22-011302.

https://gerrit.wikimedia.org/r/1198036

Maintenance_bot removed a project: Patch-For-Review.Oct 22 2025, 2:31 PM
Jdforrester-WMF removed a project: Epic.Nov 5 2025, 6:54 PM
ecarg closed this task as Resolved.Nov 18 2025, 5:52 AM

Have not found clear evidence of memory leakage, I am resolving this hypothesis so that I can move on to others

ecarg reopened this task as Open.Nov 19 2025, 11:36 PM

reopening as per COW investigation on Orchestrator OOM's today

Jdforrester-WMF changed the task status from Open to In Progress.Nov 20 2025, 8:53 PM
ecarg created subtask T410813: create tool to trace garbage collection.Nov 22 2025, 3:15 AM
ecarg created subtask T410814: create tool to visualize memory usage.Nov 22 2025, 3:18 AM
ecarg changed the status of subtask T410813: create tool to trace garbage collection from Open to In Progress.Nov 25 2025, 9:42 PM
ecarg changed the status of subtask T410814: create tool to visualize memory usage from Open to In Progress.Nov 27 2025, 8:55 AM
ecarg closed subtask T410814: create tool to visualize memory usage as Resolved.Dec 3 2025, 11:48 PM
ecarg closed this task as Resolved.Dec 8 2025, 7:30 PM

Memory leaks detected this quarter and created tools to track Node garbage collection and memory usage using Docker stats.

Jdforrester-WMF moved this task from In Engineering to Needs Sign-off on the Abstract Wikipedia team (26Q2 (Oct–Dec)) board.Dec 8 2025, 7:31 PM
ecarg closed subtask T410813: create tool to trace garbage collection as Resolved.Dec 9 2025, 4:14 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits