Page MenuHomePhabricator
Create Task
Maniphest T422384

[builds-builder] incompatibility of fagiani/apt and builder stack "24"
Closed, ResolvedPublicBUG REPORT
Actions

Description

Steps to replicate the issue (include links if applicable):
Run buildservice for Rust

What happens?:
observed in tool mix-n-match but probably others too:

...
[step-inject-buildpacks] 2026-04-06T14:04:55.215898794Z
[step-inject-buildpacks] 2026-04-06T14:04:55.215901654Z     [[order.group]]
[step-inject-buildpacks] 2026-04-06T14:04:55.215904495Z     id = "heroku/nodejs"
[step-inject-buildpacks] 2026-04-06T14:04:55.215907339Z     version = "5.5.3"
[step-inject-buildpacks] 2026-04-06T14:04:55.215910159Z     optional = false
[step-inject-buildpacks] 2026-04-06T14:04:55.215912942Z     [[order.group]]
[step-inject-buildpacks] 2026-04-06T14:04:55.215915851Z     id = "heroku/procfile"
[step-inject-buildpacks] 2026-04-06T14:04:55.215918662Z     version = "4.2.1"
[step-inject-buildpacks] 2026-04-06T14:04:55.215921508Z     optional = true
[step-set-stack-toml] 2026-04-06T14:04:43.204845904Z 2026/04/06 14:04:43 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-set-stack-toml] 2026-04-06T14:04:55.245343137Z Using run image: tools-harbor.wmcloud.org/toolforge/heroku-runner:24_0.21.5
[step-analyze] 2026-04-06T14:04:43.750811400Z 2026/04/06 14:04:43 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-analyze] 2026-04-06T14:04:57.321599898Z Restoring data for SBOM from previous image
[step-detect] 2026-04-06T14:04:58.244059530Z target distro name/version labels not found, reading /etc/os-release file
[step-detect] 2026-04-06T14:04:58.287662246Z 2 of 5 buildpacks participating
[step-detect] 2026-04-06T14:04:58.287729779Z fagiani/apt 0.2.5
[step-detect] 2026-04-06T14:04:58.287760228Z emk/rust    0.1
[step-restore] 2026-04-06T14:04:45.062254566Z 2026/04/06 14:04:45 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-restore] 2026-04-06T14:04:59.133048368Z Restoring metadata for "fagiani/apt:imagemagick_fix" from app image
[step-build] 2026-04-06T14:04:59.596716299Z target distro name/version labels not found, reading /etc/os-release file
[step-build] 2026-04-06T14:04:59.721018503Z -----> Detected Aptfile changes, flushing cache
[step-build] 2026-04-06T14:04:59.768844610Z -----> Updating apt caches
[step-build] 2026-04-06T14:04:59.836918828Z        Reading package lists...
[step-build] 2026-04-06T14:04:59.855343648Z -----> Fetching .debs for php
[step-build] 2026-04-06T14:04:59.987474659Z ERROR: failed to build: exit status 1
[step-fix-nested-procfile-launcher] 2026-04-06T14:05:00.765900438Z 2026/04/06 14:05:00 Skipping step because a previous step failed
[step-fix-imagemagick-config] 2026-04-06T14:05:00.979952909Z 2026/04/06 14:05:00 Skipping step because a previous step failed
[step-fix-permissions] 2026-04-06T14:05:01.221464034Z 2026/04/06 14:05:01 Skipping step because a previous step failed
[step-export] 2026-04-06T14:04:46.650576293Z 2026/04/06 14:04:46 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-export] 2026-04-06T14:05:01.663624100Z 2026/04/06 14:05:01 Skipping step because a previous step failed
[step-results] 2026-04-06T14:05:01.872295594Z 2026/04/06 14:05:01 Skipping step because a previous step failed

What should have happened instead?:
Rust build complete

Related Objects
Search...

Event Timeline

magnusmanske created this task.Apr 6 2026, 2:20 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptApr 6 2026, 2:20 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
bd808 mentioned this in T387141: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack.Apr 6 2026, 10:42 PM
bd808 added subscribers: fnegri, bd808.Apr 6 2026, 10:44 PM

It looks to me like the failure is:

[step-build] 2026-04-06T14:04:59.855343648Z -----> Fetching .debs for php
[step-build] 2026-04-06T14:04:59.987474659Z ERROR: failed to build: exit status 1

My guess is that the Buildpack environment version upgrade has caused this problem. As I reported in T394466: [build-service] remove legacy fagiani/apt 0.2.5 builder from `--use-latest-versions` stack the fagiani/apt builder dies with this message when applied within the new "24" builder stack. I had suggested in T394466 that the legacy fagiani/apt builder be removed. It seems that did not happen before or after my bug report was closed as a duplicate of @fnegri's T387141: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack.

https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images#heroku/deb-packages_(Ubuntu_2024.04_stack,_with_--use-latest-versions) describes a local fix you will need to apply @magnusmanske in order to migrate to the newer heroku/deb-packages builder which does work with the "24" builder stack.

bd808 renamed this task from Buildservice for Rust fails to Buildservice for Rust fails due to fagiani/apt and builder stack "24" mismatch.Apr 6 2026, 10:44 PM
bd808 added a parent task: T380127: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble.Apr 6 2026, 10:56 PM
dcaro subscribed.Apr 7 2026, 8:43 AM

That would be the future-proof way to go yep (url changed https://wikitech.wikimedia.org/wiki/Help:Toolforge/Building_container_images#heroku/deb-packages_(Ubuntu_2024.04_stack,_current_default). I'll give it a quick look, but if it gives too much trouble I might just disable the old apt buildpack.

magnusmanske added a comment.Apr 7 2026, 8:52 AM

I have added the project.toml file to both my repo root and the tool root. Same thing.

dcaro added a comment.Apr 7 2026, 8:57 AM
In T422384#11792774, @magnusmanske wrote:

I have added the project.toml file to both my repo root and the tool root. Same thing.

You have to remove the Aptfile too, otherwise it detects it anyhow as fagiani/apt buildpack.

dcaro renamed this task from Buildservice for Rust fails due to fagiani/apt and builder stack "24" mismatch to [builds-builder] incompatibility of fagiani/apt and builder stack "24".Apr 7 2026, 9:44 AM
dcaro claimed this task.
dcaro triaged this task as High priority.
dcaro added a project: tools-platform-team.
dcaro moved this task from Backlog to In progress on the tools-platform-team board.
Sean.hoyland subscribed.Apr 7 2026, 11:46 AM

I had a similar build error with a python tool that used an Aptfile to get noto font packages for a plotting library.
Replacing the Aptfile with the project.toml resolved the build error.
I needed to update the paths
from
NOTO_CORE_PATH = "/layers/fagiani_apt/apt/usr/share/fonts/truetype/noto"
NOTO_CJK_PATH = "/layers/fagiani_apt/apt/usr/share/fonts/opentype/noto"
to
NOTO_CORE_PATH = "/layers/heroku_deb-packages/packages/usr/share/fonts/truetype/noto"
NOTO_CJK_PATH = "/layers/heroku_deb-packages/packages/usr/share/fonts/opentype/noto"

dcaro moved this task from In progress to In review on the tools-platform-team board.Apr 7 2026, 1:15 PM

Got a fix for this, waiting for reviews, should be merged soon-ish: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/86

dcaro closed this task as Resolved.Apr 7 2026, 3:37 PM
In T422384#11793745, @dcaro wrote:

Got a fix for this, waiting for reviews, should be merged soon-ish: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/86

Deployed, this is fixed now in production, I'll close the task, but feel free to reopen if you still find the same issue (or open a new one if the issue is a different one).

dcaro moved this task from In review to Done on the tools-platform-team board.Apr 7 2026, 4:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits