Page MenuHomePhabricator

[toolforge-envvars.api,toolforge-build.api] Support using custom environment variables at build time
Closed, ResolvedPublic

Description

As an example, Django buildpacks run python mysite/manage.py collectstatic --noinput as part of the build, unless DISABLE_COLLECTSTATIC=1. During the execution of this command, Django needs to access the SECRET_KEY, which in production should not be hard-coded. Until we have deployed the Toolforge 'secret service', there is no other way to supply the secret key, as reading from a file during the build stage isn't possible either.

A solution would be to set a build-time env var, similar how to Heroku apps can use the env vars DYNO and CI to know what environment the app is currently running in.

To implement:

  • Extend the builds-api to support receiving an envvars parameter (hash of string->string) and send it as ENV_VARS parameter to the tekton task (verify that it works)
  • add an --envar NAME=VALUE option (repeatable) to the builds-cli start command to be able to send custom env vars for the build

Event Timeline

The Heroku 'config vars' documentation is somewhat unclear. Based on the available docs, buildpack code and observed behavior, the following seems to be true:

  • Config vars set with the heroku config:set ENV_VAR=VALUE CLI command are in general runtime variables and not available during the build.
  • but... using the Python buildpack, if Django is one of the dependencies, then DEBUG_COLLECTSTATIC and DISABLE_COLLECTSTATIC are used during one of the build steps and can be set via the above method, making them available in the build environment.
  • Most likely, the same is true for other buildpacks: if an env var is needed during the build stage, then it's made available to it; otherwise, config vars are runtime vars only.
  • As far as I've been able to tell, none of this is explicitly documented in the Heroku docs, or in the docs of each buildpack.

Just discovered that this is true also for the heroku clojure buildpack (see https://github.com/heroku/heroku-buildpack-clojure/blob/main/bin/compile#L34) where you can specify envvars to select the version of the tooling to use.

It's not enough to just know which env you are in (build environment or not), it needs the specific env variable set, so this means that we might want to flag envvars to be set for build and or run runtimes.

dcaro renamed this task from [toolforge-build.api] Add an env var to identify the build environment to [toolforge-envvars.api,toolforge-build.api] Support flagging environment variables to be injected at build time .Sep 5 2023, 10:04 AM
dcaro changed the task status from Open to In Progress.Dec 11 2023, 2:42 PM
dcaro claimed this task.
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 02) board.
dcaro renamed this task from [toolforge-envvars.api,toolforge-build.api] Support flagging environment variables to be injected at build time to [toolforge-envvars.api,toolforge-build.api] Support using custom environment variables at build time .Dec 11 2023, 2:45 PM
dcaro updated the task description. (Show Details)

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/154

builds-admission: bump to 0.0.66-20231213102324-01f8d72c

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T10:43:55Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T338142)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T10:44:23Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T338142)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T10:48:01Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-admission (T338142)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T10:48:33Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-admission (T338142)

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/155

builds-api: bump to 0.0.117-20231213110249-3c9d6b72

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T13:26:29Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T338142)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T13:27:01Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T338142)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T13:31:27Z] <wm-bot2> dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T338142)

Mentioned in SAL (#wikimedia-cloud-feed) [2023-12-13T13:32:01Z] <wm-bot2> dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T338142)

dcaro moved this task from In Review to Done on the Toolforge (Toolforge iteration 02) board.