Fix recurring error: `FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory`
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	ecarg
	Nov 1 2024, 1:44 AM

Description

What/Why:
We routinely see this error on our dashboard. We have also seen it occur during P0 incidents, we can't be sure how related it is to the last incident, but we need better insight as to why this happens.

How:

Details

Related Changes in Gerrit:

Subject	Repo	Branch	Lines +/-
wikifunctions: Upgrade orchestrator from version: 2025-01-22-203140 to 2025-01-28-144249	operations/deployment-charts	master	+1 -1
wikifunctions: Upgrade orchestrator from 2025-01-08-142250 to 2025-01-15-052609	operations/deployment-charts	master	+1 -1
wikifunctions: Upgrade orchestrator from 2024-11-27-074306 to 2024-12-17-184905	operations/deployment-charts	master	+1 -1

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	preemptive alert/logging on memory issues	repos/abstract-wiki/wikifunctions/function-orchestrator!260	ecarg	grace/T378785/heap-errors	main

Customize query in GitLab

Related Objects

Mentioned In: T383740: [spike] find out what type of requests might trigger NodeJS heap/memory errors
Mentioned Here: T383806: [spike] find out where the delay is coming from

Event Timeline

ecarg created this task.Nov 1 2024, 1:44 AM

Jdforrester-WMF triaged this task as High priority.Nov 6 2024, 5:37 PM

Jdforrester-WMF changed the subtype of this task from "Task" to "Bug Report".

Jdforrester-WMF moved this task from To Triage to 25Q2 (Oct–Dec) on the Abstract Wikipedia team board.

Jdforrester-WMF edited projects, added Abstract Wikipedia team (25Q2 (Oct–Dec)); removed Abstract Wikipedia team.

Jdforrester-WMF moved this task from Incoming to Essential work on the Abstract Wikipedia team (25Q2 (Oct–Dec)) board.Nov 6 2024, 5:47 PM

this is a Node js error saying that memory usage is too much...

perhaps we need to increase heap limit?

...but we should make sure that is absolutely necessary:

monitor garbage collection
monitor memory leakage
monitor heap usage

Observation/notes:

This can be mapped to the spike in CPU usage in Grafana
Around the same time, there is a spike in implementation error count frequency
This Node js built-in error is often prefaced with other built-in errors,"<--- JS stacktrace --->"(used to indicate beginning of a JS stack trace; usu. in error logs for critical issues like memory problems or uncaught exceptions) and/or followed by "<--- Last few GCs --->" logs.

Feeglgeef subscribed.Nov 26 2024, 4:58 PM

This comment was removed by Feeglgeef.

ecarg updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/260

Draft: preemptive alert/logging on memory issues

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/260

preemptive alert/logging on memory issues

ecarg opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/263

Draft: enable Node garbage collection?

Change #1105362 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-11-27-074306 to 2024-12-17-184905

https://gerrit.wikimedia.org/r/1105362

Change #1105362 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2024-11-27-074306 to 2024-12-17-184905

https://gerrit.wikimedia.org/r/1105362

Jdforrester-WMF moved this task from Essential work to In Engineering on the Abstract Wikipedia team (25Q2 (Oct–Dec)) board.Dec 18 2024, 3:37 PM

ecarg updated https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/263

add periodic, more descriptive, logging + refactor

apine merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/263

add periodic, more descriptive, logging + refactor

ecarg mentioned this in T383740: [spike] find out what type of requests might trigger NodeJS heap/memory errors.Jan 14 2025, 9:50 PM

Change #1111626 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-01-08-142250 to 2025-01-15-052609

https://gerrit.wikimedia.org/r/1111626

Jdforrester-WMF moved this task from 25Q2 (Oct–Dec) to 25Q3 (Jan–Mar) on the Abstract Wikipedia team board.Jan 15 2025, 8:11 PM

Jdforrester-WMF edited projects, added Abstract Wikipedia team (25Q3 (Jan–Mar)); removed Abstract Wikipedia team (25Q2 (Oct–Dec)).

Jdforrester-WMF moved this task from Incoming to In Engineering on the Abstract Wikipedia team (25Q3 (Jan–Mar)) board.

Change #1111626 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-01-08-142250 to 2025-01-15-052609

https://gerrit.wikimedia.org/r/1111626

Is there more to do here, or should this be moved to sign-off?

This can be moved to sign-off, thank you! Next steps can be covered via this task https://phabricator.wikimedia.org/T383806

Jdforrester-WMF moved this task from In Engineering to Needs Sign-off on the Abstract Wikipedia team (25Q3 (Jan–Mar)) board.Jan 21 2025, 6:09 PM

Change #1115032 had a related patch set uploaded (by Cory Massaro; author: Cory Massaro):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from version: 2025-01-22-203140 to 2025-01-28-144249

https://gerrit.wikimedia.org/r/1115032

Change #1115032 abandoned by Cory Massaro:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from version: 2025-01-22-203140 to 2025-01-28-144249

Reason:

already done