Page MenuHomePhabricator

Fix or remove Blubber's node_modules optimization
Closed, ResolvedPublic

Description

Blubber attempts to optimize the resulting image layers of Node projects, and avoids the need to rebuild all layers, by using multiple steps for `npm install and copying of source files.

  1. Copy package.json to a temp directory
  2. Execute npm install in the temp directory
  3. Copy source files into app directory
  4. Move node_modules from temp to app directory

However, the move is resulting in a substantial additional layer.

IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
970771081d8d        19 hours ago        /bin/sh -c #(nop) ENTRYPOINT ["npm" "start"]    0B
36f35b6b9662        19 hours ago        /bin/sh -c mv /tmp/node-deps/node_modules ./    150MB (<- wtf.)
1e096ee48098        19 hours ago        /bin/sh -c #(nop) COPY dir:af714faee7f3cee...   419kB
ac52dc20b7c8        20 hours ago        /bin/sh -c cd /tmp/node-deps/ && npm install    200MB
7957b9b068dd        20 hours ago        /bin/sh -c #(nop) COPY file:16ecab327f6508...   1.84kB

I wasn't clear to me at first why a move would result in an fs layer of any substance—since moving a file or directory hierarchy is essentially just changing the inode/name entry—but it may be a current limitation of overlayfs and the changes to support more efficient directory tree moves are backwards incompatible. Translation: This is not a reliable optimization as it's currently implemented and won't be for some time.

We should either figure out an alternative way to implement this optimization or get rid of it.

One alternative might be to use a bind mount of node_modules from its original build location to inside the app's directory. It seems convoluted at face value but the solution might have the happy side effect of solving an issue in development where a VOLUME is desired for the app directory over the copying of source files—the bind mount can happen over the shared volume just as it can over the copied directory, hence a higher degree of parity between a dev and staging/prod image.

Event Timeline

dduvall created this task.Jul 25 2017, 6:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 25 2017, 6:10 PM

Could it be that /tmp is a tmpfs and thus moving files under / is actually a copy+delete?

What about running npm install at the root of wherever the modules have to appear?

Could it be that /tmp is a tmpfs and thus moving files under / is actually a copy+delete?

It doesn't appear to be.

runuser@8cadd6104ffc:/srv/service$ mountpoint /tmp
/tmp is not a mountpoint

What about running npm install at the root of wherever the modules have to appear?

Another possibility, but with the current permission/privilege scheme the parent directory (in this particular case /srv/) is owned by root, not the unprivileged user that's effective when executing npm install. We could change that scheme of course but there may be security concerns with executing npm install as root and/or changing ownership of the parent directory without really knowing whether that's appropriate given the current config—changing /srv in this example doesn't seem appropriate.

If we can ensure an empty/absent node_modules in the app directory, can we npm install elsewhere—e.g. /var/local/lib/node_modules, /opt/node_modules or some such directory—and append that location to NODE_PATH?

dduvall triaged this task as High priority.Aug 2 2017, 8:10 PM
dduvall moved this task from Backlog to Doing on the Release Pipeline (Blubber) board.