Page MenuHomePhabricator

Decide whether to install heapdump by default, or continue to install npm & install on demand
Closed, ResolvedPublic

Description

Service-runner optionally uses the heapdump package to save a heap dump on SIGUSR2. We used to directly require heapdump as a devDependency, but have since removed it & rely on npm to be present on the machines so that we can install it temporarily in case we need to debug a memory issue. The require of the package only happens in the signal handler, so things work fine if the package is not installed.

The reasons for why we no longer require heapdump by default are:

  • It introduced a binary dependency that most third-party users don't need. Even we don't need it normally.
  • Because it interfaces with V8 internals, the heapdump package is fairly specific to a node version, and often fails to work if compiled on a slightly different version of node. Since we don't have systematic building of dependencies & generally rely on doing this on developer laptops this often does not work.
  • We need to install specific versions of the heapdump package for specific node versions. For example, latest heapdump does not work any more with the old node 0.10 version we are running. This means that heapdump often breaks the install for third-party users on different (often more current) node versions by failing to compile.

So, the question is:

  • Should we switch back to installing heapdump by default?

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)
GWicke added a project: service-runner.
GWicke subscribed.
GWicke renamed this task from Decide whether to install heapdump by default, or install npm & install on demand to Decide whether to install heapdump by default, or continue to install npm & install on demand.Apr 8 2015, 3:30 PM
GWicke added projects: acl*sre-team, Parsoid.
GWicke set Security to None.
GWicke edited subscribers, added: mobrovac, ssastry, yuvipanda; removed: Aklapper.

Based on discussions with @GWicke, due to the constraints and dependencies introduced by heapdump, we decided not to include it in service-runner 's package.json. However, in order to still be able to dump the heap in production, there are two options to consider:

  1. Install automatically npm and build-essential on all nodes running Node.js services. This would allow us to then just install heapdump manually and inspect the heap on the spot. The caveat here is that we need to install these packages consistently on all servers.
  2. Not list heapdump in package.json, but still make it part of the deploy repositories, i.e. install it manually there. The upside is that we wouldn't need to install npm and build-essential throughout the system, while still keeping the ability to dump a process' heap. The caveat here is that the host on which the node module is being compiled must have the exact same specs as the production hosts.

I vote for option (2). While it may seem error-prone, we have encountered this problem at least once before with other, less system-dependent node modules, so we'll need a containter-like solution for it anyway.

mobrovac added a project: Services.

The caveat here is that we need to install these packages consistently on all servers.

Why is that necessary?

The caveat here is that the host on which the node module is being compiled

This would depend on T94611: Automate compiling service dependencies using production Jessie libraries being resolved.

The caveat here is that we need to install these packages consistently on all servers.

Why is that necessary?

Ups, my wording seems to be completely messed up today. It should read ...on all servers node.js services are running on.

The caveat here is that the host on which the node module is being compiled

This would depend on T94611: Automate compiling service dependencies using production Jessie libraries being resolved.

Yup. And I, for one, would really like to see that one resolved ASAP.

Ups, my wording seems to be completely messed up today. It should read ...on all servers node.js services are running on.

Ah, I thought you referred to heapdump, not npm / build-dependencies. Makes sense now.

Yup. And I, for one, would really like to see that one resolved ASAP.

Me too, am just a bit sceptical about this happening in an automated manner very soon.

The service-runner PR #30 goes in the direction (2) in that it uses containers to install the right node module dependencies as well as heapdump.

mobrovac claimed this task.
mobrovac added a project: User-mobrovac.

We have been going down the second route for a while now, resolving.