Some findings on how to get this:

Using a custom builder, with a big group definition in it's builder.toml

We would have to maintain a custom builder image, but the buildpacks can be upstream.

We would have to wire (or generate) either:

• A group with all the buildpacks set to optional
• A group for each combination of buildpacks

as the non-optional buildpacks have to all pass the detection step to be applied, and at least one has to match, or the whole group gets discarded.

Example of builder.toml adding python and ruby buildpacks group:

Example of a run:

10:06 AM ~/Work/wikimedia/tbs/cloud-toolforge-buildpacks  (master|✚ 1)
dcaro@vulcanus$ ./make.sh
...
Successfully created builder image toolforge-bullseye0-builder:latest
Tip: Run pack build <image-name> --builder toolforge-bullseye0-builder:latest to use this builder



10:11 AM ~/Work/repos/per_user/david-caro/wm-lol  (upstream_buildpacks|✚ 1)
dcaro@vulcanus$ pack build wm-lol --builder toolforge-bullseye0-builder:latest --verbose
Using project descriptor located at project.toml
...
===> DETECTING
Running the detector on OS linux with:
Container Settings:
  Args: /cnb/lifecycle/detector -app /workspace -log-level debug
  System Envs: CNB_PLATFORM_API=0.9
  Image: pack.local/builder/646b70626578786c6272:latest
  User:
  Labels: map[author:pack]
Host Settings:
  Binds: pack-layers-foerfguzfb:/layers pack-app-bkwsdpeoxk:/workspace
  Network Mode:
...
[detector] 6 of 9 buildpacks participating
[detector] paketo-buildpacks/ca-certificates 3.5.0
[detector] paketo-buildpacks/cpython         1.8.0
[detector] paketo-buildpacks/pip             0.16.1
[detector] paketo-buildpacks/pip-install     0.5.7
[detector] paketo-buildpacks/python-start    0.14.0
[detector] paketo-buildpacks/procfile        5.4.0
...
[exporter] Layer 'cache.sbom' SHA: sha256:aa3f96e2b104d486c89d54f5000faf56c5c6f7db7180c4127769ee1ae623ea8e
Successfully built image wm-lol

Specifying in the source code with project.toml

Another option is forcing the buildpacks you want to use by adding a project descriptor project.yaml file with the buildpacks you want:

dcaro@vulcanus$ cat project.toml
[tool.isort]
profile="black"

[[build.buildpacks]]
uri = "paketo-buildpacks/python"

[[build.buildpacks]]
uri = "paketo-buildpacks/ruby"

Then it will force those buildpacks, but they will be added as "mandatory" so they must not fail, so becomes a bit trickier.

Unfortunately, this is not implemented yet on tekton (see https://github.com/buildpacks/tekton-integration/issues/33, the solution is https://github.com/buildpacks/rfcs/pull/238 )

Did a quick test following the https://phabricator.wikimedia.org/T325799#8486313 solution, rebuilding the paketo-buildpacks builder with support for all the official buildpacks it comes with as optional, that makes the detect step be way longer as it has to check every combination:

  step-detect:
    Container ID:  docker://b172bc5f2f14682513b0414e3334984a014e2fc890ae601bb587d5c97a9e0ec1
    Image:         quay.io/dcaro/tests:latest
...
      Started:      Fri, 23 Dec 2022 13:34:13 +0100
      Finished:     Fri, 23 Dec 2022 13:36:46 +0100

but it finishes correctly, and with multiple stacks (in this case ruby + python):

[build-from-git : detect] 10 of 22 buildpacks participating
[build-from-git : detect] paketo-buildpacks/ca-certificates 3.5.1
[build-from-git : detect] paketo-buildpacks/mri             0.10.1
[build-from-git : detect] paketo-buildpacks/bundler         0.6.2
[build-from-git : detect] paketo-buildpacks/bundle-install  0.6.0
[build-from-git : detect] paketo-buildpacks/rackup          0.4.12
[build-from-git : detect] paketo-buildpacks/procfile        5.5.0
[build-from-git : detect] paketo-buildpacks/cpython         1.8.0
[build-from-git : detect] paketo-buildpacks/pip             0.16.1
[build-from-git : detect] paketo-buildpacks/pip-install     0.5.7
[build-from-git : detect] paketo-buildpacks/python-start    0.14.0

Mention again that it requires that your code is shaped in a way that the buildpacks detect it correctly, there's no way to force a buildpack to run if it does not pass the detection.

The documentation at https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app states that Heroku tools (which are presumably using Heroku build packs) can create a custom stack by configuring additional builders to run on their repos manually.

I found these docs from https://github.com/heroku/heroku-buildpack-apt which is a buildpack that I think we are also interested in allowing from outside of the current builder-classic-22 stack.

In T325799#8777279, @bd808 wrote:

The documentation at https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app states that Heroku tools (which are presumably using Heroku build packs) can create a custom stack by configuring additional builders to run on their repos manually.

Yep, they do, unfortunately, that's because they have their own platform implementation (that afaik it's not open-source), that allows them to do that on the fly. For us that live with tekton, we have to either do ourselves (with things like tweaking the prepare script, or more core changes like https://github.com/buildpacks/rfcs/pull/238) or come with it before-hand (in the builder, that's what I was testing in the previous comments)

I found these docs from https://github.com/heroku/heroku-buildpack-apt which is a buildpack that I think we are also interested in allowing from outside of the current builder-classic-22 stack.

Yes, and the only way we have to do that currently is to create our own builder image, and include it in the bulider.toml, specifically, you have to make sure that there's a order.group that has the combination of buildpacks you want (it will use the first group that matches all the buildpacks that passed the detect stage, see https://phabricator.wikimedia.org/T325799#8486313 for an example with a group with python and ruby at the same time).
Note that the pack cli is able to inject buildpacks, but tekton as it is it's not.

An option there would be to replace the whole tekton buildpack integration with packer somehow, that requires though access to docker or similar to be able to start containers on demand. Or kpack, that is similar on top of k8s (not sure if that one supports the buildpack injection though).

In T325799#8778592, @dcaro wrote:

In T325799#8777279, @bd808 wrote:

I found these docs from https://github.com/heroku/heroku-buildpack-apt which is a buildpack that I think we are also interested in allowing from outside of the current builder-classic-22 stack.

Yes, and the only way we have to do that currently is to create our own builder image, and include it in the bulider.toml, specifically, you have to make sure that there's a order.group that has the combination of buildpacks you want (it will use the first group that matches all the buildpacks that passed the detect stage, see https://phabricator.wikimedia.org/T325799#8486313 for an example with a group with python and ruby at the same time).
Note that the pack cli is able to inject buildpacks, but tekton as it is it's not.

A couple of non-expert questions:

• Can we create a Tekton Pipeline/Task/Whatever to build a new builder image?
• Can we create a Tekton Pipeline/Task/Whatever that allows selecting the builder image to use at runtime?

My thought here is would it be possible to build our way around the lack of support for injecting buildpacks by instead enabling the creation and use of custom builders?

Imagine a world where you can put a .toolforge/builder.toml file in your repo. When submitted to the build service that file would be seen by a pre-builder inspection script, the script would kick off a builder build based on the config, and eventually tell Tekton to run that new builder against the repo to produce the desired container image. I think I mean doing the functional equivalent of:

$ git clone $REPO && cd $REPO
$ pack builder create $TOOL/builder:latest --config .toolforge/builder.toml
$ pack build $TOOL/$TOOL:latest --builder $TOOL/builder:latest

This is a lot of complexity and degrees of freedom to burden most users with, so if we could do this then we could also make things a bit simpler by generating the builder.toml file from the less complex subset of project.toml that @dcaro showed in T325799#8488391.

In T325799#8780655, @bd808 wrote:

In T325799#8778592, @dcaro wrote:

In T325799#8777279, @bd808 wrote:

I found these docs from https://github.com/heroku/heroku-buildpack-apt which is a buildpack that I think we are also interested in allowing from outside of the current builder-classic-22 stack.

Yes, and the only way we have to do that currently is to create our own builder image, and include it in the bulider.toml, specifically, you have to make sure that there's a order.group that has the combination of buildpacks you want (it will use the first group that matches all the buildpacks that passed the detect stage, see https://phabricator.wikimedia.org/T325799#8486313 for an example with a group with python and ruby at the same time).
Note that the pack cli is able to inject buildpacks, but tekton as it is it's not.

A couple of non-expert questions:

• Can we create a Tekton Pipeline/Task/Whatever to build a new builder image?

I guess it would be possible, though that would be aside from buildpacks, and would have to be supported with packer builder create, that means somehow enabling docker inside k8s itself or similar, would have to investigate.

• Can we create a Tekton Pipeline/Task/Whatever that allows selecting the builder image to use at runtime?

That would be possible yes, currently there's a parameter to toolforge build, --builder-image, that allows specifying the builder to use at runtime. The buildpack-admission-controller only allows heroku for now though.

My thought here is would it be possible to build our way around the lack of support for injecting buildpacks by instead enabling the creation and use of custom builders?

That would be possible, but that also opens the door to being able to hijack the bulid pipeline, as now the builder image is a custom one, that builder can do essentially whatever you want at any step of the build, for example, calling an external webservice passing through any env vars and secrets.

Imagine a world where you can put a .toolforge/builder.toml file in your repo. When submitted to the build service that file would be seen by a pre-builder inspection script, the script would kick off a builder build based on the config, and eventually tell Tekton to run that new builder against the repo to produce the desired container image. I think I mean doing the functional equivalent of:

$ git clone $REPO && cd $REPO
$ pack builder create $TOOL/builder:latest --config .toolforge/builder.toml
$ pack build $TOOL/$TOOL:latest --builder $TOOL/builder:latest

This is a lot of complexity and degrees of freedom to burden most users with, so if we could do this then we could also make things a bit simpler by generating the builder.toml file from the less complex subset of project.toml that @dcaro showed in T325799#8488391.

I think this approach of generating the toml file might be safer (I think we could be able to inject it without having to rebuild the builder), and still allow users to define extra buildpacks if they want, it would essentially be a partial custom implementation of https://github.com/buildpacks/rfcs/pull/238. Note that the key difference is leaving the control of the builder image to us, instead of letting the user control it, but allowing the user to specify buildpacks (that we might be able to verify), that will run only at build time, in the analyze and build steps (will not run at the export step, where we have the credentials we don't want to share).

Now that we have apt buildpack support, this task is just to validate a couple flows:

• NPM build (buildpack) + python backend (apt)
• NPM build (apt) + python backend (buildpack)
• Php-composer (buildpack) + python (apt)
• php(apt) + python-pip (buildpack)

And add some docs/hints on how to do it, if it's easy enough we can document this and fix it with the apt buildpack.

Scenario: I can use multiple stacks on my app at build time (ex. compiling react static assets and serving with a python web service)

I'm wondering if this specific example is something we should implement/document, as opposed to having users building static assets locally (at least until we have proper object storage)?

Heroku uses the start and build commands from the package.json to build the assets when using the nodejs buildpack, e.g:

"scripts": {
  "build": "nitro build", // or `nuxt build` if using nuxt
  "start": "node .output/server/index.mjs"
}

This behavior may be part of the buildpack itself, although I haven't investigated yet. If it is, then this should be straightforward, i.e. inject python via Apt buildback, build/run the app via the package.json scripts.

In T325799#9163716, @Slst2020 wrote:

This behavior may be part of the buildpack itself, although I haven't investigated yet. If it is, then this should be straightforward, i.e. inject python via Apt buildback, build/run the app via the package.json scripts.

Hmm, just having Python is not enough though as we'd also need to pip install from requirements.txt. How would that work?

Having played around with pack, combining nodejs (for the purpose of building frontend assets) with a backend language that serves the application is unfortunately not that straightforward. For instance, installing python and pip via the apt buildpack still requires installing the dependencies at some step of the process. This could in theory be done by leveraging the scripts section in the package.json but is a bit tricky.

Another thing is that as long as there is any "trace" of Python (e.g a manage.py file) in the top-level project directory when running the detect stage of the lifecycle, the script will exit complaining that it couldn't find requirements.txt and skip over the remaining detection steps.

@dcaro The injected apt buildpack is currently always last in the execution order, as it's appended to group.toml (vs. prepended). Is this intentional?

In T325799#9210048, @Slst2020 wrote:

@dcaro The injected apt buildpack is currently always last in the execution order, as it's appended to group.toml (vs. prepended). Is this intentional?

We discussed this and it's not. Buildpacks run in the order listed in group.toml, so the apt buildpack should be prepended in case it installs dependencies needed by downstream buildpacks T347985: [tbs][buildpacks] ensure apt buildpack runs before others

@Slst2020 I think we can close this, and create a task for each stack we need when we need it instead.