Page MenuHomePhabricator

[buildservice,nodejs] nodejs buildpack does not take envvars into account
Closed, ResolvedPublic


On my quest to deploy a webservice on the Kubernetes Toolforge cluster, I discovered the new recommended way: using the Toolforge Build Service to build a container image before running it on k8s. This is a nodejs webservice.

Context: I am using pnpm. Problem: Heroku's builder-22 doesn't support any buildpack PNPM compatible. I know that it is planned to support arbitrary buildpacks one day, but this is clearly not a priority and it is only a sub-solution to the real problems.

Indeed, with the help of @bd808, I switched to npm after testing a solution that I will exaplain at the end.

Now, sub-problems met are:

  • If you were using pnpm, you have to delete the node_modules folder, then use npm install, then pnpm install, since we only really need npm for the build process. Only when committing a release, you should use npm i --package-lock-only to update the package-lock.json, instead of checking node_modules and downloading dependencies;
  • Now you have multiple lockfiles found, uhoh. The Heroku/cloud buildpack environement is new to me, but the official documentation notices that:

Node projects have the Yarn package manager or the NPM package manager. If a yarn.lock file is detected in the root of the project, Yarn installs dependencies and runs scripts. Otherwise, npm is used. If you have yarn.lock checked into your project, but want to use npm to build on Heroku, add yarn.lock to your .slugignore file.

So I used a .slugignore with pnpm-lock.yaml in it. It didn't fix the problem:

Multiple lockfiles found: package-lock.json, pnpm-lock.yaml. More than one package manager has created lockfiles for this application. Only one can be used to install dependencies but the buildpack can't determine which when multiple lockfiles are present.

It seems logical if we believe this 10yo old Stack Overflow answer:

Slugignore only controls what ends in the app slug, not what is in the Heroku repo. So if you ran heroku run bash and checked with ls, you would find that the things in .slugignore were not there. Use .gitignore if there's something you don't want to include in the git repo.

For whatever it means...

  • Then I tried something: a dummy secondary branch with only one lock file with the --ref parameter supported by toolforge-build. And it builds! But not in production mode. I have a bunch of TypeScript types errors, I still don't know why. Building a production locally works... but not when building on Toolforge.

Sub-sub-problem now is: it tries to install devDependencies, because envvars are not passed to the buildpack by the Build Service.

All these steps were realized because toolforge build start --envvar USE_NPM_INSTALL=true --envvar NPM_CONFIG_PRODUCTION=true is not working as intended.

And why did I try to use USE_NPM_INSTALL? Because the official documentation states that you can set the package installation process with some envvars. Using npm install means not using npm ci that needs a package-lock.json... the file missing because I wanted to only use pnpm at the start.

I also tried to define these envvars with the envvars Toolforge service. Didn't work. I tried to provoke my luck because I'm having trouble understanding the "build environment context".

For technical aspects, these environnemental variables are indeed read in the environnement by the buildpack:

Here the end of my quest. Big long-text-context because I think it is meaningful for the WMF devs to know the technical problems met and for the posterity.

Event Timeline

Do you have the logs of the build when it did not set the environment variables?
(or the tool/build id)

That would help debugging it.

Lofhi renamed this task from Toolforge Build Service doesn't pass envvars to the buildpack to [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one).Dec 15 2023, 9:23 PM

I can see the preparation script putting the environment variables in the platform directory:

[step-prepare] 2023-12-15T18:39:21.376078717Z --> Creating 'env' directory: /platform/env
[step-prepare] 2023-12-15T18:39:21.382994912Z --> Writing /platform/env/USE_NPM_INSTALL...

It might be related to the newer version of the nodejs buildpack, with the previous one it did pick it up:


I was able to add the execution of env in one of the other buildpacks, and the variables are loaded correctly in the environment:

[step-build] 2023-12-15T21:53:05.161364882Z HOSTNAME=tf-test-buildpacks-pipelinerun-xqlbc-build-from-git-pod
[step-build] 2023-12-15T21:53:05.161372563Z CNB_LAYERS_DIR=/layers/fagiani_apt
[step-build] 2023-12-15T21:53:05.161374218Z NODE_VERBOSE=true  <--- this
[step-build] 2023-12-15T21:53:05.161375315Z PWD=/workspace
[step-build] 2023-12-15T21:53:05.161376200Z CNB_STACK_ID=heroku-22
[step-build] 2023-12-15T21:53:05.161377099Z CNB_BUILDPACK_DIR=/tmp_builder/buildpacks/fagiani_apt/0.2.5
[step-build] 2023-12-15T21:53:05.161378292Z NPM_CONFIG_PRODUCTION=true
[step-build] 2023-12-15T21:53:05.161379231Z HOME=/tekton/home
[step-build] 2023-12-15T21:53:05.161380055Z CNB_PLATFORM_DIR=/platform
[step-build] 2023-12-15T21:53:05.161380926Z SHLVL=1
[step-build] 2023-12-15T21:53:05.161381888Z CNB_BP_PLAN_PATH=/tmp/fagiani_apt-291668211/fagiani_apt/plan.toml
[step-build] 2023-12-15T21:53:05.161382748Z PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[step-build] 2023-12-15T21:53:05.161383807Z USE_NPM_INSTALL=true  <-- this

And the CNB_PLATFORM_DIR is also set correctly. Feels like a bug in the buildpack itself maybe, looking...

By adding a build script to my package.json, I was able to run env as part of the nodejs buildpack, and the environment variables are there:

step-build] 2023-12-15T22:23:14.551954150Z       LIBRARY_PATH=/layers/heroku_nodejs-engine/dist/lib:/layers/fagiani_apt/apt/lib/x86_64-linux-gnu:/layers/fagiani_apt/apt/lib/i386-linux-gnu:/layers/fagiani_apt/apt/lib:/layers/fagiani_apt/apt/usr/lib/x86_64-linux-gnu:/layers/fagiani_apt/apt/usr/lib/i386-linux-gnu:/layers/fagiani_apt/apt/usr/lib
[step-build] 2023-12-15T22:23:14.551963161Z       CPPPATH=/layers/fagiani_apt/apt/usr/include:/layers/fagiani_apt/apt/usr/include/x86_64-linux-gnu
[step-build] 2023-12-15T22:23:14.551964844Z       npm_config_user_agent=npm/10.2.3 node/v20.10.0 linux x64 workspaces/false
[step-build] 2023-12-15T22:23:14.551965764Z       HOSTNAME=tf-test-buildpacks-pipelinerun-grjjx-build-from-git-pod
[step-build] 2023-12-15T22:23:14.551966749Z       npm_node_execpath=/layers/heroku_nodejs-engine/dist/bin/node
[step-build] 2023-12-15T22:23:14.551967689Z       LD_LIBRARY_PATH=/layers/heroku_nodejs-engine/dist/lib:/layers/fagiani_apt/apt/lib/x86_64-linux-gnu:/layers/fagiani_apt/apt/lib/i386-linux-gnu:/layers/fagiani_apt/apt/lib:/layers/fagiani_apt/apt/usr/lib/x86_64-linux-gnu:/layers/fagiani_apt/apt/usr/lib/i386-linux-gnu:/layers/fagiani_apt/apt/usr/lib
[step-build] 2023-12-15T22:23:14.551969101Z       npm_config_noproxy=
[step-build] 2023-12-15T22:23:14.551969974Z       HOME=/tekton/home
[step-build] 2023-12-15T22:23:14.551970924Z       npm_package_json=/workspace/package.json
[step-build] 2023-12-15T22:23:14.551971771Z       CNB_PLATFORM_DIR=/platform
[step-build] 2023-12-15T22:23:14.551972603Z       npm_config_userconfig=/tekton/home/.npmrc
[step-build] 2023-12-15T22:23:14.551973430Z       npm_config_local_prefix=/workspace
[step-build] 2023-12-15T22:23:14.551974334Z       CNB_BUILDPACK_DIR=/tmp_builder/buildpacks/heroku_nodejs-npm-install/2.6.1
[step-build] 2023-12-15T22:23:14.551975332Z       COLOR=0
[step-build] 2023-12-15T22:23:14.551976269Z       CNB_BP_PLAN_PATH=/tmp/heroku_nodejs-npm-install-3857613339/heroku_nodejs-npm-install/plan.toml
[step-build] 2023-12-15T22:23:14.551977121Z       INCLUDE_PATH=/layers/fagiani_apt/apt/usr/include:/layers/fagiani_apt/apt/usr/include/x86_64-linux-gnu
[step-build] 2023-12-15T22:23:14.551978367Z       CNB_LAYERS_DIR=/layers/heroku_nodejs-npm-install
[step-build] 2023-12-15T22:23:14.551979231Z       npm_config_prefix=/layers/heroku_nodejs-engine/dist
[step-build] 2023-12-15T22:23:14.551980055Z       npm_config_npm_version=10.2.3
[step-build] 2023-12-15T22:23:14.551981038Z       PKG_CONFIG_PATH=/layers/fagiani_apt/apt/usr/lib/x86_64-linux-gnu/pkgconfig:/layers/fagiani_apt/apt/usr/lib/i386-linux-gnu/pkgconfig:/layers/fagiani_apt/apt/usr/lib/pkgconfig
[step-build] 2023-12-15T22:23:14.551981921Z       npm_config_cache=/layers/heroku_nodejs-npm-install/npm_cache
[step-build] 2023-12-15T22:23:14.551983170Z       npm_config_node_gyp=/layers/heroku_nodejs-engine/dist/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js
[step-build] 2023-12-15T22:23:14.551985055Z       PATH=/workspace/node_modules/.bin:/node_modules/.bin:/layers/heroku_nodejs-engine/dist/lib/node_modules/npm/node_modules/@npmcli/run-script/lib/node-gyp-bin:/layers/heroku_nodejs-engine/dist/bin:/layers/fagiani_apt/apt/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
[step-build] 2023-12-15T22:23:14.551985943Z       NODE=/layers/heroku_nodejs-engine/dist/bin/node
[step-build] 2023-12-15T22:23:14.551991625Z       npm_package_name=test
[step-build] 2023-12-15T22:23:14.551992510Z       USE_NPM_INSTALL=true            <-- here
[step-build] 2023-12-15T22:23:14.551993349Z       npm_lifecycle_script=env
[step-build] 2023-12-15T22:23:14.551994191Z       NODE_VERBOSE=true               <--- and here
[step-build] 2023-12-15T22:23:14.551995063Z       npm_package_version=0.0.0
[step-build] 2023-12-15T22:23:14.551995916Z       npm_lifecycle_event=build
[step-build] 2023-12-15T22:23:14.551996786Z       npm_config_globalconfig=/layers/heroku_nodejs-engine/dist/etc/npmrc
[step-build] 2023-12-15T22:23:14.551997638Z       npm_config_init_module=/tekton/home/.npm-init.js
[step-build] 2023-12-15T22:23:14.551998773Z       npm_execpath=/layers/heroku_nodejs-engine/dist/lib/node_modules/npm/bin/npm-cli.js
[step-build] 2023-12-15T22:23:14.551999642Z       PWD=/workspace
[step-build] 2023-12-15T22:23:14.552000522Z       npm_config_global_prefix=/layers/heroku_nodejs-engine/dist
[step-build] 2023-12-15T22:23:14.552001375Z       npm_command=run-script
[step-build] 2023-12-15T22:23:14.552003061Z       CPATH=/layers/heroku_nodejs-engine/dist/include:/layers/fagiani_apt/apt/usr/include:/layers/fagiani_apt/apt/usr/include/x86_64-linux-gnu
[step-build] 2023-12-15T22:23:14.552003916Z       CNB_STACK_ID=heroku-22
[step-build] 2023-12-15T22:23:14.552004743Z       INIT_CWD=/workspace
[step-build] 2023-12-15T22:23:14.552005777Z       EDITOR=vi

I will check with the latest versions and open a bug upstream

I get the same behavior using latest heroku/builder:22 builder and the pack tool, the vars are there, but the buildpack still uses npm ci:

vagrant@bullseye:~/wm-lol$ pack build --builder heroku/builder:22 --env NODE_VERBOSE=true --env USE_NPM_INSTALL=true --buildpack heroku/nodejs test

- Installing node modules
  - Using npm version `10.2.3`
  - Creating npm cache
  - Configuring npm cache directory
  - Running `npm ci "--production=false"`

      npm WARN config production Use `--omit=dev` instead.
      npm WARN deprecated graceful-fs@2.0.3: please upgrade to graceful-fs 4 for compatibility with current and future versions of Node.js
      npm WARN deprecated mkdirp@0.3.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
      npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
      npm WARN deprecated mkdirp@0.5.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
      npm WARN deprecated jade@0.26.3: Jade has been renamed to pug, please install the latest version of pug instead of jade
      added 79 packages, and audited 80 packages in 2s
      11 packages are looking for funding
        run `npm fund` for details
      9 vulnerabilities (1 moderate, 5 high, 3 critical)
      Some issues need review, and may require choosing
      a different dependency.
      Run `npm audit` for details.
      npm notice 
      npm notice New patch version of npm available! 10.2.3 -> 10.2.5
      npm notice Changelog: <>
      npm notice Run `npm install -g npm@10.2.5` to update!
      npm notice 
  - Done (1.698s)
- Running scripts
  - Running `npm run build`

      > test@0.0.0 build
      > ls -la $CNB_PLATFORM_DIR/env; env | sort
      total 16
      drwxr-xr-x 1 root root 4096 Dec 15 22:26 .
      drwxr-xr-x 1 root root 4096 Dec 15 22:26 ..
      -rw-r--r-- 1 root root    4 Jan  1  1980 NODE_VERBOSE
      -rw-r--r-- 1 root root    4 Jan  1  1980 USE_NPM_INSTALL
      npm_config_user_agent=npm/10.2.3 node/v20.10.0 linux x64 workspaces/false
      npm_lifecycle_script=ls -la $CNB_PLATFORM_DIR/env; env | sort
dcaro renamed this task from [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one) to [buildservice,nodejs] nodejs buildpack does not take envvars into account.Dec 15 2023, 10:32 PM

A nice xmas-gift I see! Anyway, I can't go any further on the subject until next week, there's no hurry. Thanks for the expertise!

Context: I am using pnpm. Problem: Heroku's builder-22 doesn't support any buildpack PNPM compatible. I know that it is planned to support arbitrary buildpacks one day, but this is clearly not a priority and it is only a sub-solution to the real problems.

It seems like Heroku's node.js buildpack v2.6.0, released yesterday, ships with pnpm support. So I hope we can somehow update to that? (I could not find any documentation on how to do that :()

Context: I am using pnpm. Problem: Heroku's builder-22 doesn't support any buildpack PNPM compatible. I know that it is planned to support arbitrary buildpacks one day, but this is clearly not a priority and it is only a sub-solution to the real problems.

It seems like Heroku's node.js buildpack v2.6.0, released yesterday, ships with pnpm support. So I hope we can somehow update to that? (I could not find any documentation on how to do that :()

There's two things to do, change the version of the nodes build pack when it's injected by the builds-builder, and pull the newer heroku/builder:22 image and push it to harbor (tools and tools beta).

It might upgrade other build packs too so testing some would be wise.

I think I know what happened, heroku deprecated the heroku/builder-classic:22, that uses the old, non-cnb buildpacks, and the supported builder is heroku/builder:22, that instead uses different ones:

In those, the nodejs one does not support all the envvars that the previous one ( did

It seems so complicated. The npm cli or npm install choice is only depending of the major version of npm.

Maybe the buildpack doesn't support envvars, but surely the pack command is supporting passing them. Meaning you can use by example npm config envvars?

I guess with an up-to-date dev environnement you don't really need to chose between the two install options and config settings are enough.

Something I don't understand is: if the Build Service is already using Cloud Native Buildpacks with Builder:22, why using pnpm is not supported? Are we one version behind?

This buildpack's bin/detect will only pass if a pnpm-lock.json exists in the project root. This is done to prevent the buildpack from providing indeterminate and unpredictable dependency trees.

Ugh... Do you mean Toolforge Build Service is already supporting the pnpm buildpack, but the detection is broken, because pnpm use a... YAML lockfile?

It seems that the documentation is just wrong and the YAML lockfile is indeed looked up. Running out of explanations on my side!

Edit: ah yes, forgot the comment about the pnpm buildpack engine shipped with the new release two days ago. At least one problem would be fixed, not sure about the envvars usage changes.

Successfully deployed the (broken) webservice from a built image:
So the envvars buildpack support removed is not truly problematic; at least for this project. Still waiting for the builder update with the pnpm support!

Really glad to hear this! \o/

Something I don't understand is: if the Build Service is already using Cloud Native Buildpacks with Builder:22, why using pnpm is not supported? Are we one version behind?

The image we have currently has the 2.5.0 buildpacks, there's a newer one with the 2.6.1 version, should be easy to update (see T353566: Document how to update heroku-builder:22 container image), a good chance to create docs/cookbook for it and test it :)

04:29 PM ~/Work/repos/per_user/david-caro/wm-lol  (test_aptfile|✔) 
dcaro@urcuchillay$ podman run --rm ls -la /cnb/buildpacks/heroku_nodejs-pnpm-install/  # <- upstream as of today
total 0
drwxr-xr-x. 1 root root 10 Jan  1  1980 .
drwxr-xr-x. 1 root root 24 Jan  2 15:27 ..
drwxr-xr-x. 1 root root 58 Jan  1  1980 2.6.1

04:29 PM ~/Work/repos/per_user/david-caro/wm-lol  (test_aptfile|✔) 
dcaro@urcuchillay$ podman run ls -la /cnb/buildpacks/heroku_nodejs-pnpm-install/  # <- our builder image
total 0
drwxr-xr-x. 1 root root 10 Jan  1  1980 .
drwxr-xr-x. 1 root root 24 Dec 11 12:18 ..
drwxr-xr-x. 1 root root 58 Jan  1  1980 2.5.0

@Lofhi pnpm should be available now (the one here, if you want to try, can you report back how it went?

Gonna try this weekend, thanks for the notice.

So, I tried.

[step-build] 2024-01-07T10:53:05.952465817Z [Heroku Node.js Engine Buildpack]
[step-build] 2024-01-07T10:53:05.952471667Z
[step-build] 2024-01-07T10:53:05.952475907Z [Checking Node.js version]
[step-build] 2024-01-07T10:53:05.981012347Z Detected Node.js version range: >=20.0.0 <21.0.0-0
[step-build] 2024-01-07T10:53:05.981335995Z Resolved Node.js version: 20.10.0
[step-build] 2024-01-07T10:53:05.981368120Z
[step-build] 2024-01-07T10:53:05.981377833Z [Installing Node.js distribution]
[step-build] 2024-01-07T10:53:05.981862711Z Downloading Node.js 20.10.0
[step-build] 2024-01-07T10:53:07.190079798Z Extracting Node.js 20.10.0
[step-build] 2024-01-07T10:53:08.966418093Z Installing Node.js 20.10.0
[step-build] 2024-01-07T10:53:08.990441556Z - Debug info
[step-build] 2024-01-07T10:53:08.992440443Z   - Corepack Requirement Error
[step-build] 2024-01-07T10:53:08.992476068Z
[step-build] 2024-01-07T10:53:08.992484096Z ! A pnpm lockfile (`pnpm-lock.yaml`) was detected, but the
[step-build] 2024-01-07T10:53:08.992491223Z ! version of `pnpm` to install could not be determined.
[step-build] 2024-01-07T10:53:08.992497855Z !
[step-build] 2024-01-07T10:53:08.992507730Z ! `pnpm` may be installed via the `heroku/nodejs-corepack`
[step-build] 2024-01-07T10:53:08.992515002Z ! buildpack. It requires the desired `pnpm` version to be set
[step-build] 2024-01-07T10:53:08.992521717Z ! via the `packageManager` key in `package.json`.
[step-build] 2024-01-07T10:53:08.992528652Z !
[step-build] 2024-01-07T10:53:08.992534596Z ! To set `packageManager` in `package.json` to the latest
[step-build] 2024-01-07T10:53:08.992541530Z ! `pnpm`, run:
[step-build] 2024-01-07T10:53:08.992547877Z !
[step-build] 2024-01-07T10:53:08.992553730Z ! `corepack enable`
[step-build] 2024-01-07T10:53:08.992560377Z ! `corepack use pnpm@*`
[step-build] 2024-01-07T10:53:08.992566540Z !
[step-build] 2024-01-07T10:53:08.992572405Z ! Then commit the result, and try again.
[step-build] 2024-01-07T10:53:08.992578580Z
[step-build] 2024-01-07T10:53:08.994671366Z ERROR: failed to build: exit status 1

A bit weird, with the last... process it was autodetecting it? I guess, no?

Added "packageManager": "^pnpm@8.14.0" in package.json but it failled on the same build step. The engines.node value (20.x) specified and detected was correct, but it seems that it didn't find packageManager.

So, whatever. Replaced the value with just pnpm@8.14.0 instead and it worked.

[step-build] 2024-01-07T11:09:20.629317051Z [Heroku Node.js Engine Buildpack]
[step-build] 2024-01-07T11:09:20.629332035Z
[step-build] 2024-01-07T11:09:20.629369118Z [Checking Node.js version]
[step-build] 2024-01-07T11:09:20.666817833Z Detected Node.js version range: >=20.0.0 <21.0.0-0
[step-build] 2024-01-07T11:09:20.666844642Z Resolved Node.js version: 20.10.0
[step-build] 2024-01-07T11:09:20.666848425Z
[step-build] 2024-01-07T11:09:20.666851322Z [Installing Node.js distribution]
[step-build] 2024-01-07T11:09:20.667716515Z Downloading Node.js 20.10.0
[step-build] 2024-01-07T11:09:22.114132890Z Extracting Node.js 20.10.0
[step-build] 2024-01-07T11:09:23.472700858Z Installing Node.js 20.10.0
[step-build] 2024-01-07T11:09:23.662977837Z
[step-build] 2024-01-07T11:09:23.663018770Z [Installing pnpm 8.14.0 via corepack 0.22.0]
[step-build] 2024-01-07T11:09:23.909339908Z Preparing pnpm@8.14.0...
[step-build] 2024-01-07T11:09:24.939818908Z
[step-build] 2024-01-07T11:09:24.939922229Z [Setting up pnpm dependency store]
[step-build] 2024-01-07T11:09:24.939941306Z Creating new pnpm content-addressable store
[step-build] 2024-01-07T11:09:24.940727314Z Creating pnpm virtual store
[step-build] 2024-01-07T11:09:26.478486882Z
[step-build] 2024-01-07T11:09:26.478570444Z [Installing dependencies]
[step-build] 2024-01-07T11:09:27.129392509Z Lockfile is up to date, resolution step is skipped

But then my build scripts were failling:

[step-build] 2024-01-07T11:09:48.223979865Z [Error: heroku/nodejs-pnpm build script error]
[step-build] 2024-01-07T11:09:48.223988188Z There was an error while attempting to run a build script
[step-build] 2024-01-07T11:09:48.223994369Z from this project's package.json. The command exited with a non-zero exit code.
[step-build] 2024-01-07T11:09:48.224000297Z
[step-build] 2024-01-07T11:09:48.224005749Z Details: Exit code exit status: 1
[step-build] 2024-01-07T11:09:48.224011440Z
[step-build] 2024-01-07T11:09:48.228969931Z ERROR: failed to build: exit status 1

This one seems not related to the Buildservice. I don't know why it was working without @types/node as a devDependency previously... Meh. Added it back.

Everything then was fine. is up. It's really cool to use, but the very basis of the service is hard to get to grips with... Their documentation (not yours) isn't crazy.

Still, I don't know if USE_NPM_INSTALL is needed with the updated buildpack. But I guess it not really my concern since I am using pnpm... Thanks for your work!

dcaro claimed this task.