Page MenuHomePhabricator

[builds-api,builds-builder] php buildpack is broken on current default buidlpacks and `--use-latest-versions` ones
Closed, ResolvedPublic

Description

Sometime in the last day or two, trying to rebuild a Toolforge tool started to fail; the following part of the output is where it all seems to go wrong:

[step-set-stack-toml] 2026-03-19T00:14:09.453544792Z 2026/03/19 00:14:09 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-set-stack-toml] 2026-03-19T00:14:21.471606585Z Using run image: tools-harbor.wmcloud.org/toolforge/heroku-runner:22-cnb
[step-analyze] 2026-03-19T00:14:09.885656849Z 2026/03/19 00:14:09 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-analyze] 2026-03-19T00:14:23.556450974Z Restoring data for SBOM from previous image
[step-detect] 2026-03-19T00:14:24.631407358Z 2 of 4 buildpacks participating
[step-detect] 2026-03-19T00:14:24.631488524Z heroku/php      0.1.2
[step-detect] 2026-03-19T00:14:24.631503117Z heroku/procfile 2.0.2
[step-restore] 2026-03-19T00:14:10.704376023Z 2026/03/19 00:14:10 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-restore] 2026-03-19T00:14:24.755830992Z Restoring metadata for "heroku/procfile:nested_launcher_fix" from app image
[step-build] 2026-03-19T00:14:25.183101241Z
[step-build] 2026-03-19T00:14:25.183183437Z [Bootstrapping]
[step-build] 2026-03-19T00:14:26.000316705Z
[step-build] 2026-03-19T00:14:26.000379960Z [Error: A download error occurred during bootstrapping]
[step-build] 2026-03-19T00:14:26.000405583Z https://codeload.github.com/heroku/heroku-buildpack-php/tar.gz/refs/heads/cnb-installer: status code 404 (redirected from https://github.com/heroku/heroku-buildpack-php/archive/refs/heads/cnb-installer.tar.gz)
[step-build] 2026-03-19T00:14:26.000416935Z
[step-build] 2026-03-19T00:14:26.000427359Z This is most likely a transient internal error; you should re-try the operation.
[step-build] 2026-03-19T00:14:26.000436073Z
[step-build] 2026-03-19T00:14:26.000444997Z Please make sure that you are not running an outdated version of the buildpack.
[step-build] 2026-03-19T00:14:26.000454837Z
[step-build] 2026-03-19T00:14:26.000465007Z If the problem persists, open a support ticket and include the full log output of this build.
[step-build] 2026-03-19T00:14:26.002601395Z ERROR: failed to build: exit status 1

I have two different Toolforge tools and they both have this problem, so I don't think it's due to some specific configuration. Is anyone else experiencing this?

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

@Yaron_Koren can you show the toolforge build list output? that will give some info on what code you used and what tool is affected

I have been able to reproduce with the sample php app:

tools.sample-php-buildpack-app@tools-bastion-15:~$ toolforge build start https://gitlab.wikimedia.org/toolforge-repos/sample-php-buildpack-app
...
[step-build] 2026-03-19T11:35:20.304099834Z [Bootstrapping]
[step-build] 2026-03-19T11:35:21.243962133Z 
[step-build] 2026-03-19T11:35:21.244046206Z [Error: A download error occurred during bootstrapping]
[step-build] 2026-03-19T11:35:21.244070188Z https://codeload.github.com/heroku/heroku-buildpack-php/tar.gz/refs/heads/cnb-installer: status code 404 (redirected from https://github.com/heroku/heroku-buildpack-php/archive/refs/heads/cnb-installer.tar.gz)
[step-build] 2026-03-19T11:35:21.244087493Z 
[step-build] 2026-03-19T11:35:21.244098850Z This is most likely a transient internal error; you should re-try the operation.
[step-build] 2026-03-19T11:35:21.244109415Z 
[step-build] 2026-03-19T11:35:21.244120140Z Please make sure that you are not running an outdated version of the buildpack.
[step-build] 2026-03-19T11:35:21.244129471Z 
[step-build] 2026-03-19T11:35:21.244140161Z If the problem persists, open a support ticket and include the full log output of this build.
[step-build] 2026-03-19T11:35:21.246770933Z ERROR: failed to build: exit status 1
[step-fix-nested-procfile-launcher] 2026-03-19T11:35:21.543194317Z 2026/03/19 11:35:21 Skipping step because a previous step failed
[step-fix-imagemagick-config] 2026-03-19T11:35:21.764367107Z 2026/03/19 11:35:21 Skipping step because a previous step failed
[step-fix-permissions] 2026-03-19T11:35:21.998371723Z 2026/03/19 11:35:21 Skipping step because a previous step failed
[step-export] 2026-03-19T11:35:06.466913943Z 2026/03/19 11:35:06 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-export] 2026-03-19T11:35:22.485501022Z 2026/03/19 11:35:22 Skipping step because a previous step failed
[step-results] 2026-03-19T11:35:22.703491544Z 2026/03/19 11:35:22 Skipping step because a previous step failed

Looking, might be related to the buildpack being used being too old

Using --latest-versions also fails:

tools.sample-php-buildpack-app@tools-bastion-15:~$ toolforge build start https://gitlab.wikimedia.org/toolforge-repos/sample-php-buildpack-app --use-latest-versions
...
[step-build] 2026-03-19T11:36:44.282068973Z target distro name/version labels not found, reading /etc/os-release file
[step-build] 2026-03-19T11:36:44.293157910Z 
[step-build] 2026-03-19T11:36:44.293198137Z [Bootstrapping]
[step-build] 2026-03-19T11:36:45.108410255Z 
[step-build] 2026-03-19T11:36:45.108470865Z [Error: A download error occurred during bootstrapping]
[step-build] 2026-03-19T11:36:45.108479288Z https://codeload.github.com/heroku/heroku-buildpack-php/tar.gz/refs/heads/cnb-installer: status code 404 (redirected from https://github.com/heroku/heroku-buildpack-php/archive/refs/heads/cnb-installer.tar.gz)
[step-build] 2026-03-19T11:36:45.108485792Z 
[step-build] 2026-03-19T11:36:45.108491166Z This is most likely a transient internal error; you should re-try the operation.
[step-build] 2026-03-19T11:36:45.108495972Z 
[step-build] 2026-03-19T11:36:45.108500540Z Please make sure that you are not running an outdated version of the buildpack.
[step-build] 2026-03-19T11:36:45.108505077Z 
[step-build] 2026-03-19T11:36:45.108510196Z If the problem persists, open a support ticket and include the full log output of this build.
[step-build] 2026-03-19T11:36:45.110614046Z ERROR: failed to build: exit status 1
[step-fix-nested-procfile-launcher] 2026-03-19T11:36:45.440863767Z 2026/03/19 11:36:45 Skipping step because a previous step failed
[step-fix-imagemagick-config] 2026-03-19T11:36:45.678531986Z 2026/03/19 11:36:45 Skipping step because a previous step failed
[step-fix-permissions] 2026-03-19T11:36:45.930768829Z 2026/03/19 11:36:45 Skipping step because a previous step failed
[step-export] 2026-03-19T11:36:32.368081684Z 2026/03/19 11:36:32 warning: unsuccessful cred copy: ".docker" from "/tekton/creds" to "/tekton/home": unable to open destination: open /tekton/home/.docker/config.json: permission denied
[step-export] 2026-03-19T11:36:46.381494204Z 2026/03/19 11:36:46 Skipping step because a previous step failed
[step-results] 2026-03-19T11:36:46.878015539Z 2026/03/19 11:36:46 Skipping step because a previous step failed

Using the soon to be 'latest' buildpacks does work correctly:

[step-build] 2026-03-19T11:48:02.353045996Z ## Heroku PHP Buildpack
[step-build] 2026-03-19T11:48:02.353050653Z 
[step-build] 2026-03-19T11:48:02.353052705Z - Bootstrapping
[step-build] 2026-03-19T11:48:04.526777231Z - Preparing platform packages installation
[step-build] 2026-03-19T11:48:04.527390000Z - Installing platform packages
[step-build] 2026-03-19T11:48:07.201679592Z   - php (8.5.3)
[step-build] 2026-03-19T11:48:07.685756348Z   - composer (2.9.3)
[step-build] 2026-03-19T11:48:07.754431933Z - Installing web servers
[step-build] 2026-03-19T11:48:09.620923582Z   - nginx (1.28.1)
[step-build] 2026-03-19T11:48:09.632540237Z   - apache (2.4.66)
[step-build] 2026-03-19T11:48:09.691352562Z   - boot-scripts (1.276.0)
[step-build] 2026-03-19T11:48:09.718512773Z - Installing dependencies
[step-build] 2026-03-19T11:48:09.851111641Z   Installing dependencies from lock file
[step-build] 2026-03-19T11:48:09.851152993Z   Verifying lock file contents can be installed on current platform.
[step-build] 2026-03-19T11:48:09.864533200Z   Nothing to install, update or remove
[step-build] 2026-03-19T11:48:09.865977130Z   Generating optimized autoload files
[step-build] 2026-03-19T11:48:09.871366551Z - Preparing Composer runtime environment
[step-build] 2026-03-19T11:48:09.946832316Z - Done (finished in 7.5s)
[step-build] 2026-03-19T11:48:09.951473249Z

Maybe we should update directly the latest builders with the newer ones? php is going to be broken even with the current latest ones :/

dcaro renamed this task from "toolforge build start" fails due to Heroku download error? to [builds-api,builds-builder] php buildpack is broken on current default buidlpacks and `--use-latest-versions` ones.Mar 19 2026, 11:54 AM
dcaro claimed this task.
dcaro triaged this task as High priority.
dcaro edited projects, added: Toolforge (Toolforge iteration 26); removed: Toolforge.
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 26) board.

The current list of tools using the latest builders (or that used them recently) is:

dcaro@tools-bastion-15:~$ kubectl-sudo get pipelineruns -n image-build  -o "custom-columns=TOOL:.metadata.labels.user,BUILDER:.spec.params[0].value" | grep :24 | awk '{print $1}'| sort | uniq 
agpb
attribution-configurator
bd808-buildpack
bd808-test
best-of
bs-map-editor
campwiz
campwiz-backend
campwiz-backend-beta
campwiz-backend-dev
campwiz-beta
campwiz-bot
campwiz-dev
cluebot-syncer
cluebot3
cluebotng
cluebotng-editsets
cluebotng-monitoring
cluebotng-review
cluebotng-staging
cluebotng-trainer
commonsdb-registry
containers
curator
dev-agpb
entityvalidator-backend
github-pr-closer
gitlab-content
lexeme-forms
lucaswerkmeister-test
openstack-browser
osmviews
pagepile-visual-filter
phabricator-bug-status
pulac
pywikibot
quickcategories
ranker
sample-php-buildpack-app
spacemedia
sparql-rc2-backend
speedpatrolling
techdoc-dashboard
techdoc-linter
toolforge-functional-runner
translate-link
unwatchlist
video2commons-test
wd-image-positions
wdactle
wdtmcollab
wdtmcollab-api
whopaintedthis
wikilink-archive
wm-lol

I'll try to test locally see if there would be many errors or not.

Maybe we should update directly the latest builders with the newer ones?

+1 for updating "latest" sooner than planned, to provide a fix for this issue.

The fix has been deployed, for php, you have to use the latest versions toolforge build start --use-latest-versions https://gitlab.wikimedia.org/....

Thank you! It works with "--use-latest-versions".

dcaro moved this task from In Progress to Done on the Toolforge (Toolforge iteration 26) board.