The orchestrator is crashing very frequently now due to OOM

As discussed in the meeting, SRE and AW teams might have different definitions of what OOM means.

SRE

For an SRE, this usually means that the OOMKiller, a kernel mechanism to protect the system and make an effort at a recovery, has been triggered and has killed the offending application. For the OOMKiller to show up, either the system must be very low on available memory, or, in the case of containers, a container must be trying to go over the configured memory limit.

In absence of other evidence that this is what is being reported here, and with an eye on T398106: fix crashing service, which I 've gone through and found no more evidence than the one @ecarg lists at T398106#10961131, I 've gone ahead and crafted this dashboard, from which we can see that the OOMKiller has only shown up for 1 container for a couple of days the last 90 days and it wasn't for the application itself, but for mcrouter, the sidecar container that helps address memcached. This can be corroborated by the panel regarding the various containers, where we indeed see that mcrouter reached the configured memory limits. From that one we can also infer that in total, OOMKiller was invoked quite probably twice.

It's been a while, so memories might not be very helpful, but the OOM related paste in T398106#10961131 lines up with that time range, so I am thinking that what was observed back then. The rest of that task also has other reasons for failure in there, they appear to be related to the application, I won't be diving into those.

With all of the above, I am feeling pretty confident that application isn't suffering from the kernel denying the application more memory. Whatever issues are being witnessed, are more probably related to the application.

AW

Since @cmassaro mentioned some heap related issues, and from {T398106#10971367}, it's pretty clear that at least some heap related issues have been met, I 've dug a bit on nodejs heap size. service-runner, the abandoned nodejs execution framework in the WMF, allowed to configure the node heap size. wikifunctions apparently utilize service-utils, an effort to revitalize service runner, but does not support, as far as I can tell setting the heap size as of this writing.

I went ahead and refresh my memory on a couple of things.

1. nodejs has the --max-old-space-size parameter which allows to set the old generation size for the heap.
2. In absence of a concretely set heap size, nodejs effectively offloads the task to v8. The comment in the code says that heap size for v8 is 700 MB on 32bit and 1.4GB on 64bit platforms. However, it tries to find the available memory, taking into account Linux cgroups, and trying to inform v8 to via the ConfigureDefaults(total_memory, 0) function. The definition of that function is at https://github.com/v8/v8/blob/main/src/api/api.cc#L554. The code from down there becomes quite complicated quickly, involving figuring out multiplies of the system's pointer size, the configured semi-space, the ratio between young and old generations of heap and the available memory. The TL;DR is that if left to heuristics, it's half the available memory but the minimum is 256MB and the maximum 2GB, on a amd64 system. If the amd64 system has > 16GB RAM, then a provision exists to double that maximum to 4GB. And that's it. That's the most one will get without explicit configuration. This lines up with the experiments that @cmassaro has been doing where approaching 4GB problems start showing up, so that limit is apparently already biting.

Conclusion

With the above out of the way, regardless of how much memory we give a container, heap size is NEVER going to go above 4GB RAM, unless we explicitly configure it to a different number. Even if we give a container 10GB RAM, it won't be used, unless we take explicit action.

My suggestion is to be explicit here and not rely on v8's heuristics. We want to have a value that:

• Is easy to grep in the repo,
• Has an audit trail of changes
• Has an explicit and documented process used to compute it when circumstances require it.

So my suggestion is:

@cmassaro If you have a testbed environment and a dataset that the team would like to see working, please initiate a round of tests passing --max-old-space-size to nodejs, increasing it in steps (of say 500MB) until you hit a number where the dataset is loaded without issues. Please document that process in https://wikitech.wikimedia.org and link to it from this task.

We use that number to configure the heap size of nodejs in the container (a quick look at the image says that the entrypoint will have to be somewhat amended, and then configure the container limit in the pod spec to be that number plus a 500MB leeway for the rest of nodejs's internals to function.

jforrester opened https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/435

Draft: Attempt to explicitly set the Node heap size, to 2 GiB

Change #1189535 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Set ORCHESTRATOR_HEAP_SIZE for later tweaking

https://gerrit.wikimedia.org/r/1189535

jforrester merged https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-orchestrator/-/merge_requests/435

Attempt to explicitly set the Node heap size, to 2 GiB

Change #1191171 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-09-16-134119 to 2025-09-24-180530

https://gerrit.wikimedia.org/r/1191171

Change #1189535 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Set ORCHESTRATOR_HEAP_SIZE for later tweaking

https://gerrit.wikimedia.org/r/1189535

Change #1191171 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Upgrade orchestrator from 2025-09-16-134119 to 2025-09-24-180530

https://gerrit.wikimedia.org/r/1191171