Which leads to the question of what is the current best practice for testing build pack based images?

Short answer, there's no current best practice, that is not new with buildpacks/bulids-api though, as there never has been a supported flow for it.

It is also not a trivial problem, so it will take some effort and time to be able to get a "well paved" flow going (specially with the current lack of resources in the team).

Some things that could help in the short-term (from the top of my head):

• Pin your python version to the one supported by the current buildpacks
• Have a 'dev/test' tool you deploy on first

Some things that can be explored in the mid-long term:

• Creating a builder image so it can be run locally or on CI systems (might not be able to run on gitlab.w.o though, would have to double-check with the team that maintains it)
  • This requires an initial effort to package and publish the custom/modified buildpacks, for which we had no easy way before (now gitlab supports packages, so that might be an option)
• Get more resources so we can keep up with updates (currently the team is -3 members down and -1 manager for more than two quarters)
• Figure out some custom way to run the same build process that toolforge does for CI runs
  • The first point would help with this too, having a builder image
• Maybe allow using the latest built image to run on CI somehow
  • The images are already public, so at least on your CI you can reuse the last build and run your new code/tests in it
  • Will not re-pull dependencies though, just use the same ones that were there, so not the full test env specially if the deps changed, for python-pip it might be possible to run pip install/ugrade, but buildpacks are not built for it, so might need some tweaks
• Move away from buildpacks
  • Blubber might be an option (that's how prod images are built), though it is not as featureful/easy to use afaik (ex. changing runtimes, etc., at least that I know of, it has been some time since I explored the current features)

Will have to investigate and think of more solutions, feel free to propose more, though unfortunately, we are very understaffed and will be for the next couple quarters at least.

It is also not a trivial problem, so it will take some effort and time to be able to get a "well paved" flow going

Indeed and that is fine, what we have works in broad strokes, so this is more about the future. The primary consuming service is still in beta anyway.

Get more resources so we can keep up with updates (currently the team is -3 members down and -1 manager for more than two quarters)

I see there is an open position for at least a manager and some OKR type things to expand/productionise tools

Creating a builder image so it can be run locally or on CI systems

I think this would be a good option if the ecosystem stays in the realm of build packs

Creating a builder image so it can be run locally or on CI systems / Move away from buildpacks

The other option would be to allow external registries, there are some good reasons not to (image size), but given what we have today there is little security added by restricting the image builds; if scripts can be executed then any random binary can find its way into the env, actually that's how some things have been deployed for years, even back with grid engine.

Pin your python version to the one supported by the current buildpacks

This is done by forcing the project version constraint so the install fails on newer versions, it leaves the PR open in a failed state but adding exclusions was a bit more gross. It works, but is overhead.

Have a 'dev/test' tool you deploy on first

I have this for 1 tool, now everything is under components it would be reasonably easy to duplicate every tool, though that just makes the deployment fail on a different tool (with the current restrictions/speed deploying pre-merge for each change is not very viable)

Will have to investigate and think of more solutions, feel free to propose more

I'll come back if any random thoughts come to mind