Page MenuHomePhabricator

ORCHESTRATOR_HEAP_SIZE is ignored, and production runs on V8's default heap
Open, In Progress, MediumPublic

Description

The ORCHESTRATOR_HEAP_SIZE environment variable is intended to set the service's V8 old-space limit, but it reaches nothing. The npm start script that consumes it is malformed in two ways and cannot run at all, and the deployed image does not invoke that script in any case, so production runs with whatever heap ceiling V8 infers from the container.

Technical notes

The script in package.json reads NODE_HEAP_SIZE='${ORCHESTRATOR_HEAP_SIZE:-2048}' node --max-old-space-size=NODE_HEAP_SIZE server.js. There are two defects. The single quotes prevent ${ORCHESTRATOR_HEAP_SIZE:-2048} from expanding, so NODE_HEAP_SIZE is set to that literal string. And --max-old-space-size=NODE_HEAP_SIZE names the variable without a $, so node receives a non-numeric value and refuses to start: on node v24.14.1 this gives Error: illegal value for flag --max-old-space-size=NODE_HEAP_SIZE of type size_t. So npm start is broken for local development regardless of the variable.

It is moot for deployments either way. In .pipeline/blubber.yaml both the production and development variants declare entrypoint: [node, server.js], so neither image ever runs an npm script, and no heap flag is applied. Grepping the repository, ORCHESTRATOR_HEAP_SIZE and --max-old-space-size appear only in that one start line, so there is no other path by which the value could take effect. Whether the deployment chart supplies NODE_OPTIONS independently is worth checking, as that is outside this repository.

This matters because the service is demonstrably capable of exhausting a default heap. Job 906413 failed with FATAL ERROR: Ineffective mark-compacts near heap limit after V8 chose a default old-space of roughly 512 MB from its Kubernetes pod, which is a CI job rather than production but shows the workload can reach the ceiling. As things stand there is no supported way to raise that ceiling for a running orchestrator, and the variable that appears to offer one silently does nothing.

Acceptance criteria

  • npm start runs successfully and the resulting process honours ORCHESTRATOR_HEAP_SIZE, defaulting to 2048 MB when it is unset.
  • The deployed image applies an explicit old-space limit derived from the same variable, or the variable is removed and the intended way to tune the heap is documented.

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
blubber: Pass ORCHESTRATOR_HEAP_SIZE to production pod when setrepos/abstract-wiki/wikifunctions/function-orchestrator!777jforresterT433506main
Customize query in GitLab

Event Timeline

Change #1321658 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Lower ORCHESTRATOR_HEAP_SIZE for staging

https://gerrit.wikimedia.org/r/1321658