What/Why:
We are currently focused on finding ways to improve the performance of our `function-orchestrator` service. Since tracking our heap usage through logs, we discovered we are too frequently using over 90% of the NodeJS heap limit. ([[ https://phabricator.wikimedia.org/T384883#10514431 | Ticket and notes here ]]). This week therefore, we increased [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/function-orchestrator/values.yaml#18 | the Orchestrator container k8s limit to 1Gib ]]. Our usage has not change/decreased based on the logs however. We would like to consider ideas including:
- Increasing the Orchestrator's CPU limit to 600ms rather than [[ https://grafana.wikimedia.org/goto/GEDK2xFHg?orgId=1 | the current 400ms ]]? If it's not too much trouble, would be great to try this out.
- Increase k8s memory limit:
```
Current Heap Size Limit:, {\"total_heap_size\":75214848,\"total_heap_size_executable\":1572864,\"total_physical_size\":71139328,\"total_available_size\":2170029904,\"used_heap_size\":26766264,\"heap_size_limit\":2197815296,\"malloced_memory\":540768,\"peak_malloced_memory\":7202840,\"does_zap_garbage\":0,\"number_of_native_contexts\":1,\"number_of_detached_contexts\":0,\"total_global_handles_size\":40960,\"used_global_handles_size\":38816,\"external_memory\":2357124} !!!","service.name":"function-orchestrator"}
```
=> Which means Node has around 2GB heap limit when our k8s container bounds are only up to around 1GB.
How/Next-steps: tbd