fnegri opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/88

README: Clarify process for updating Heroku images

I would support a policy where the unversioned interface (without --use-latest-versions|--use-deprecated-versions) follows an upstream major version (currently 24, until recently 22). Minor version bumps should be applied to this stack as they are released. When a new major version is released (next 26), --use-latest-versions should be enabled for a reasonable window to allow controlled migration (3-6 months) and again follow upstream minor version bumps while active. After the controlled migration window ends, the prior default version should move to --use-deprecated-versions for a reasonable hard deprecation period (1 month?). The upstream stack major versions follow Ubuntu LTS cycles, so the upgrade cycle should typically only come once every 2 years.

We were really quite slow on the 22->24 transition having just completed it as Ubuntu 26.04 was released with the Heroku-26 derived stack likely to follow in the next few months. To reduce maintainer fatigue we should maybe hold on to 26 in the --use-latest-versions a bit longer, but really if 26 drops in June/July 2026 a 6 month transition window would last until nearly the end of calendar year 2026 which is probably fine.

@bd808 thanks for the feedback! I think we can wait for @dcaro to be back before deciding on a policy, but we can start collecting ideas.

Your proposal looks good to me, one caveat is that "minor version bumps" are a bit confusing. The upstream images don't seem to have a "minor version" tag, they simply force-push the 22, 24 and 26 tags (see the heroku/cnb-builder-images repo). On our copy of the images, right now we add a suffix referring to the version of lifecycle that is included in the runner image. Currently we have the default builds using heroku-runner:24_0.21.5 and --use-latest-versions using heroku-runner:24_0.21.7. I think this was done out of caution, to avoid the risk that updating to a newer version of heroku-runner:24 including a newer lifecycle version might break some builds.

Ideally I would like to propose that we stop using this type of suffix, and we instead add a date suffix referring to the day we last pulled the upstream image, e.g. heroku-runner:24_20260427. Then we could set up a process (manual or automated) to pull a new version every x months, or more frequently in case of security bugs. This assumes that major breakages are unlikely if we stay within the same "stack" (e.g. heroku:24) even if new versions of the stack are released with updates to the included lifecycle and to the included buildpacks.

I would like to test this assumption with this patch that updates our config to use the latest available version of heroku:24, both for default builds and for builds using --use-latest-versions.

fnegri updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225

builds-api: update heroku builder and runner images

In T424362#11861392, @fnegri wrote:

Your proposal looks good to me, one caveat is that "minor version bumps" are a bit confusing. The upstream images don't seem to have a "minor version" tag, they simply force-push the 22, 24 and 26 tags (see the heroku/cnb-builder-images repo).

My understanding of the upstream is that each "stack" is the unit of management for user facing version control. I may be wrong, but my mental model is that it is lot like a Debian release in that bug fixes and new features are added within the buildpacks for a given stack, and that breaking changes are reserved for crossing the 22->24->26 stack boundaries.

fnegri merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-builder/-/merge_requests/88

README: Clarify process for updating Heroku images

group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1248

builds-builder: bump to 0.0.149-20260511095946-3ab2c1ce

fnegri closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1248

builds-builder: bump to 0.0.149-20260511095946-3ab2c1ce

group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255

builds-builder: bump to 0.0.150-20260514144411-83a0f413

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1255

builds-builder: bump to 0.0.150-20260514144411-83a0f413

Side-note: Heroku automatically restarts all containers every 24 hours, so any update to the stacks are rolled out automatically:

you don’t need to redeploy your app to include these changes. We automatically restart any running dynos as we roll out the new base images for each stack

fnegri merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1225

builds-api: update heroku builder and runner images

I would like to test this assumption with this patch that updates our config to use the latest available version of heroku:24, both for default builds and for builds using --use-latest-versions.

The patch was merged today, we are now using the same exact image (heroku:24 fetched on 2026-05-11) for default builds and for builds with --use-latest-versions. This new image contains a few version upgrades (new lifecycle version, new buildpack versions), including one change which caused a compatibility issue in lima-kilo (T426016). So far it seems to be running fine in production.

Side-note: the new image is double-tagged in Harbor as 24_0.21.8 and 24_20260511 because we need to have a different tag for --use-latest-versions. The next major version heroku:26 is already available upstream, but has not been officially released yet: https://github.com/heroku/roadmap/issues/355. When it is officially released, we can start offering heroku:26 to users building with --use-latest-versions.

We discussed this during the last Toolforge Monthly Meeting (meeting notes). We decided to adopt the following process:

• support only ONE major version from upstream (e.g. we currently support only heroku-24)
• pull the latest images weekly from upstream, but only offer them to users specifying --use-latest-versions
• users who do not specify --use-latest-version will get a "stable" version of the image, that we will update once a month
• in case of an urgent security fix, we can instead bump both the latest and the stable images at the same time
• for major version bumps (e.g. heroku-24 to heroku-26), we will follow the same process as last time (T380127#11696394)

One topic that was not discussed is how we should tag the images. I propose we use tags like 24_20260511, indicating the major version, and the date the image was pulled from upstream.

I will create follow-up tasks for automating the weekly image pulls, the monthly "stable" update, and for doing the major-version bump from heroku-24 to heroku-26.