Page MenuHomePhabricator

Support pre-built images on components-api
Open, HighPublicFeature

Description

Feature summary (what you would like to be able to do and where):

Currently there are some 'standard' images provided in toolforge:

tools.cluebotng-review@tools-bastion-15:~$ toolforge jobs images
+------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------+
|                               Short name                               |                                              image                                              |
+------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------+
|                                bookworm                                |                docker-registry.tools.wmflabs.org/toolforge-bookworm-sssd:latest                 |
|                                bullseye                                |                docker-registry.tools.wmflabs.org/toolforge-bullseye-sssd:latest                 |
|                                 jdk17                                  |               docker-registry.tools.wmflabs.org/toolforge-jdk17-sssd-base:latest                |
|                                 jdk21                                  |               docker-registry.svc.toolforge.org/toolforge-jdk21-sssd-base:latest                |
|                                mariadb                                 |              docker-registry.tools.wmflabs.org/toolforge-mariadb-sssd-base:latest               |
|                                mono6.12                                |              docker-registry.tools.wmflabs.org/toolforge-mono612-sssd-base:latest               |
|                                mono6.8                                 |               docker-registry.tools.wmflabs.org/toolforge-mono68-sssd-base:latest               |
|                                 node16                                 |               docker-registry.tools.wmflabs.org/toolforge-node16-sssd-base:latest               |
|                                 node18                                 |               docker-registry.tools.wmflabs.org/toolforge-node18-sssd-base:latest               |
|                                 node20                                 |               docker-registry.svc.toolforge.org/toolforge-node20-sssd-base:latest               |
|                                perl5.32                                |              docker-registry.tools.wmflabs.org/toolforge-perl532-sssd-base:latest               |
|                                perl5.36                                |              docker-registry.tools.wmflabs.org/toolforge-perl536-sssd-base:latest               |
|                                perl5.40                                |              docker-registry.svc.toolforge.org/toolforge-perl540-sssd-base:latest               |
|                                 php7.4                                 |               docker-registry.tools.wmflabs.org/toolforge-php74-sssd-base:latest                |
|                                 php8.2                                 |               docker-registry.tools.wmflabs.org/toolforge-php82-sssd-base:latest                |
|                                 php8.4                                 |               docker-registry.svc.toolforge.org/toolforge-php84-sssd-base:latest                |
|                               python3.11                               |             docker-registry.tools.wmflabs.org/toolforge-python311-sssd-base:latest              |
|                               python3.13                               |             docker-registry.svc.toolforge.org/toolforge-python313-sssd-base:latest              |
|                               python3.9                                |              docker-registry.tools.wmflabs.org/toolforge-python39-sssd-base:latest              |
|                                ruby2.1                                 |               docker-registry.tools.wmflabs.org/toolforge-ruby21-sssd-base:latest               |
|                                ruby2.7                                 |               docker-registry.tools.wmflabs.org/toolforge-ruby27-sssd-base:latest               |
|                                ruby3.1                                 |               docker-registry.tools.wmflabs.org/toolforge-ruby31-sssd-base:latest               |
|                                ruby3.3                                 |               docker-registry.svc.toolforge.org/toolforge-ruby33-sssd-base:latest               |
|                                 tcl8.6                                 |               docker-registry.tools.wmflabs.org/toolforge-tcl86-sssd-base:latest                |
|                                 trixie                                 |                 docker-registry.svc.toolforge.org/toolforge-trixie-sssd:latest                  |
+------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------+

These all require mounting NFS and are not supported via components.

It would be quite useful to be able to execute these images via components and not have to execute them on NFS workers (which appear to have the worst availability).

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

  • Cron jobs, which execute basic commands such as curl
  • Database dumps executed on a schedule (require NFS, but are limited to jobs currently)

Benefits (why should this be implemented?):

  • Image reuse - no need for every tool to build their own meta images, saving resources in the registry and on the worker nodes
  • Consistency across apis - no need for maintainers to manage both jobs and components
  • Reduce NFS usage - no nfs by default

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
taavi renamed this task from Build standard images under pack / support execution via components-api to Support pre-built images on components-api.Sep 29 2025, 8:29 AM

On a first glance this is a duplicate of T362076: [components-api] Add support for pre-built images (ex. python3.11, to refine), although that claims this is already implemented but I don't see any mention of that on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Deploy_your_tool.

It was done, and then scraped out for the beta

For now I have https://github.com/cluebotng/external-utilities which basically covers mariadb (curl actually is done in the report container now, but was in bookworm).

That works and perhaps is a good long term solution as it decouples maintenance/breakage from toolforge admins onto maintainers, but it also means I hit the build limits/quota more and generally things use more storage.... so in general low priority from my side, just throwing it out there.

taavi triaged this task as High priority.Oct 1 2025, 2:07 PM
taavi updated the task description. (Show Details)