(The examples in the task description are all from tools-harbor.wmcloud.org/tool-lucaswerkmeister-test/tool-lucaswerkmeister-test:latest, as built from this wd-shex-infer commit. But don’t expect that image tag to be stable. And since I may amend the commit later as well, and GitLab may garbage collect the original, I’ve put the Aptfile and Procfile into a GitHub gist for the record as far as this task is concerned.)

This is kind of a strange task (hence the bleh title). I suspect several of the issues mentioned in the task description aren’t really fixable without fundamentally changing the way the buildpack works (so it wouldn’t make much sense to break them out into individual subtasks). But at the same time… I assume the apt buildpack isn’t built that way just to make my life difficult or anything: maybe installing arbitrary apt packages is just actually not well supported in the build pack system in general, and the “extract in /layers/fagiani_apt/apt/” approach is (close to) the best thing that’s available and somewhat works? :/

There's a strong restriction on the buildpacks side that the build steps are not run as root, so yes, the apt-buildpack is a bit of a hack in the sense that it's not really installing things on the root system, and instead it tries to do so in a user directory.

This is an intentional restriction of the system and will not be easily changed.

As such, also some python scripts that have shebangs with paths to python hardcoded in them will also fail to run inside the buildpack environment. An alternative is adding those packages to the build and run images (the base images used during the build process, and the generated image), but that affects all the users and can't be done at runtime (has to be done before hand).

On the jq side, I'm looking, it's interesting that it says it's being reinstalled, as it's not there in the run image, might be in the build image instead, that might be a deeper issue (and yep, an issue with the buildpack not pulling dependencies into the run image when they exist in the build image).

Besides all that, from the code on your app I see that you are trying to run jsub from within the application, that is not supported (no grid packages are installed in the image, nor the configuration needed).

I'd recommend trying to create an image with the java dependencies using the java buildpack, to use for the jobs, and then one with the python code to use as the webservice.

From within the webservice you might be able to use the toolforge cli libraries (https://pypi.org/project/toolforge-jobs-framework-cli/) to start/stop jobs and such. Note that this flow has not been yet tested properly, but it's something we want to tackle sooner than later :), so I'm happy to help you try to set it up.

Yep, jq is installed only on the build image (see https://github.com/heroku/cnb-builder-images):

dcaro@urcuchillay$ podman run --rm -ti --entrypoint bash heroku/heroku:22-cnb-build
✔ docker.io/heroku/heroku:22-cnb-build
Trying to pull docker.io/heroku/heroku:22-cnb-build...
Getting image source signatures
Copying blob 2b01e5e338df done   | 
Copying blob 5e8117c0bd28 done   | 
Copying blob 6a25730225da done   | 
Copying blob dfa98ed79808 done   | 
Copying blob ae8ce49fd359 done   | 
Copying blob 2ee497db45ce done   | 
Copying blob fbeee558462f done   | 
Copying config 7ff3be6cff done   | 
Writing manifest to image destination
heroku@bc887373cde2:/$ jq
jq - commandline JSON processor [version 1.6]

Usage:  jq [options] <jq filter> [file...]
        jq [options] --args <jq filter> [strings...]
        jq [options] --jsonargs <jq filter> [JSON_TEXTS...]

jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.

The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).

For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq

Example:

        $ echo '{"foo": 0}' | jq .
        {
                "foo": 0
        }

For a listing of options, use jq --help.

I don't find a better-maintained replacement for the apt buildpack, and there's no alternative on the paketo world either :/

We might have to bite the bullet and fork + maintain it ourselves.

I have added support for recursive dependency checks to the apt buildpack, it's a bit of a hack, but will do for now.

Tested with python3 + jq:

toolsbeta.tf-test@lima-bookworm:~$ toolforge build start https://gitlab.wikimedia.org/toolforge-repos/wm-lol --ref with_everything
...
[step-build] 2024-01-16T15:53:32.729768466Z -----> Fetching .debs for python3
[step-build] 2024-01-16T15:53:34.637377242Z        Get:1 http://archive.ubuntu.com/ubuntu jammy-security/main amd64 python3 amd64 3.10.6-1~22.04 [22.8 kB]
[step-build] 2024-01-16T15:53:34.745148009Z        Fetched 22.8 kB in 1s (17.3 kB/s)
[step-build] 2024-01-16T15:53:34.775246180Z -----> Fetching .debs for jq
[step-build] 2024-01-16T15:53:35.634483537Z        Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB]
[step-build] 2024-01-16T15:53:35.838028176Z        Fetched 52.5 kB in 0s (123 kB/s)
[step-build] 2024-01-16T15:53:35.863746950Z -----> Fetching .debs for dependency libjq1 (pulled by jq)
[step-build] 2024-01-16T15:53:36.697750464Z        Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB]
[step-build] 2024-01-16T15:53:37.045216801Z        Fetched 133 kB in 1s (247 kB/s)
...
...
[step-results] 2024-01-16T15:55:37.695601620Z Built image 172.19.0.1/tool-tf-test/tool-tf-test:latest@sha256:3e7c8b1ecc3d0b6b7264f19ad53f5c3444acd8616e2fda5fb4b84254d620d669

dcaro@lima-bookworm:/home/dcaro/Work/wikimedia/lima-kilo$ docker run --rm -ti --entrypoint launcher 172.19.0.1/tool-tf-test/tool-tf-test:latest@sha256:3e7c8b1ecc3d0b6b7264f19ad53f5c3444acd8616e2fda5fb4b84254d620d669 bash
Enabled locales: de_DE it_IT pl_PL ja_JP en_GB fr_FR zh-hans_CN zh-hans_SG zh-hant_TW zh-hant_HK ru_RU es_ES nl_BE pt_PT
heroku@5af8f266232f:/workspace$ jq
jq - commandline JSON processor [version 1.6]

Usage:  jq [options] <jq filter> [file...]
        jq [options] --args <jq filter> [strings...]
        jq [options] --jsonargs <jq filter> [JSON_TEXTS...]

jq is a tool for processing JSON inputs, applying the given filter to
its JSON text inputs and producing the filter's results as JSON on
standard output.

The simplest filter is ., which copies jq's input to its output
unmodified (except for formatting, but note that IEEE754 is used
for number representation internally, with all that that implies).

For more advanced filters see the jq(1) manpage ("man jq")
and/or https://stedolan.github.io/jq

Example:

        $ echo '{"foo": 0}' | jq .
        {
                "foo": 0
        }

For a listing of options, use jq --help.

Thanks, but this doesn’t resolve all of my problems, just the one you retitled it to :/ as far as I can tell:

• /layers/fagiani_apt/apt/.profile.d/000_apt.sh is still not automatically sourced
• alternatives still aren’t set up
• there are still broken symlinks to absolute paths (even in wm-lol --ref with_everything, e.g. “/layers/fagiani_apt/apt/usr/bin/pygettext3: broken symbolic link to pygettext3.10”)

Should I create more issues for these? I guess T353847#9429736 and T353847#9437551 means that we’re committed to this approach and are going to work around whatever issues we find with it (because there is no alternative that would fix these issues at a more fundamental level)?

In T353847#9463709, @LucasWerkmeister wrote:

Thanks, but this doesn’t resolve all of my problems, just the one you retitled it to :/ as far as I can tell:

• /layers/fagiani_apt/apt/.profile.d/000_apt.sh is still not automatically sourced
• alternatives still aren’t set up
• there are still broken symlinks to absolute paths (even in wm-lol --ref with_everything, e.g. “/layers/fagiani_apt/apt/usr/bin/pygettext3: broken symbolic link to pygettext3.10”)

Should I create more issues for these?

I'll create them no problem, but feel free to add any extra info/tests/etc. that you find out, that always helps.

I guess T353847#9429736 and T353847#9437551 means that we’re committed to this approach and are going to work around whatever issues we find with it (because there is no alternative that would fix these issues at a more fundamental level)?

Yep, the alternative is to hack into the core of the buildpack system, or replace them completely, but there's many benefits that we would need to reimplement/work around if we do so, so we think that for now the effort to use buildpacks as the underlying technology overcomes the effort of maintaining something else (anything is doable given money and time, and we don't have too much of any xd).