Page MenuHomePhabricator
Create Task
Maniphest T405262

[components-api] Support pre-built images on components-api
Open, MediumPublicFeature
Actions

Description

Feature summary (what you would like to be able to do and where):

Currently there are some 'standard' images provided in toolforge:

tools.cluebotng-review@tools-bastion-15:~$ toolforge jobs images
+------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------+
|                               Short name                               |                                              image                                              |
+------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------+
|                                bookworm                                |                docker-registry.tools.wmflabs.org/toolforge-bookworm-sssd:latest                 |
|                                bullseye                                |                docker-registry.tools.wmflabs.org/toolforge-bullseye-sssd:latest                 |
|                                 jdk17                                  |               docker-registry.tools.wmflabs.org/toolforge-jdk17-sssd-base:latest                |
|                                 jdk21                                  |               docker-registry.svc.toolforge.org/toolforge-jdk21-sssd-base:latest                |
|                                mariadb                                 |              docker-registry.tools.wmflabs.org/toolforge-mariadb-sssd-base:latest               |
|                                mono6.12                                |              docker-registry.tools.wmflabs.org/toolforge-mono612-sssd-base:latest               |
|                                mono6.8                                 |               docker-registry.tools.wmflabs.org/toolforge-mono68-sssd-base:latest               |
|                                 node16                                 |               docker-registry.tools.wmflabs.org/toolforge-node16-sssd-base:latest               |
|                                 node18                                 |               docker-registry.tools.wmflabs.org/toolforge-node18-sssd-base:latest               |
|                                 node20                                 |               docker-registry.svc.toolforge.org/toolforge-node20-sssd-base:latest               |
|                                perl5.32                                |              docker-registry.tools.wmflabs.org/toolforge-perl532-sssd-base:latest               |
|                                perl5.36                                |              docker-registry.tools.wmflabs.org/toolforge-perl536-sssd-base:latest               |
|                                perl5.40                                |              docker-registry.svc.toolforge.org/toolforge-perl540-sssd-base:latest               |
|                                 php7.4                                 |               docker-registry.tools.wmflabs.org/toolforge-php74-sssd-base:latest                |
|                                 php8.2                                 |               docker-registry.tools.wmflabs.org/toolforge-php82-sssd-base:latest                |
|                                 php8.4                                 |               docker-registry.svc.toolforge.org/toolforge-php84-sssd-base:latest                |
|                               python3.11                               |             docker-registry.tools.wmflabs.org/toolforge-python311-sssd-base:latest              |
|                               python3.13                               |             docker-registry.svc.toolforge.org/toolforge-python313-sssd-base:latest              |
|                               python3.9                                |              docker-registry.tools.wmflabs.org/toolforge-python39-sssd-base:latest              |
|                                ruby2.1                                 |               docker-registry.tools.wmflabs.org/toolforge-ruby21-sssd-base:latest               |
|                                ruby2.7                                 |               docker-registry.tools.wmflabs.org/toolforge-ruby27-sssd-base:latest               |
|                                ruby3.1                                 |               docker-registry.tools.wmflabs.org/toolforge-ruby31-sssd-base:latest               |
|                                ruby3.3                                 |               docker-registry.svc.toolforge.org/toolforge-ruby33-sssd-base:latest               |
|                                 tcl8.6                                 |               docker-registry.tools.wmflabs.org/toolforge-tcl86-sssd-base:latest                |
|                                 trixie                                 |                 docker-registry.svc.toolforge.org/toolforge-trixie-sssd:latest                  |
+------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------+

These all require mounting NFS and are not supported via components.

It would be quite useful to be able to execute these images via components and not have to execute them on NFS workers (which appear to have the worst availability).

Use case(s) (list the steps that you performed to discover that problem, and describe the actual underlying problem which you want to solve. Do not describe only a solution):

  • Cron jobs, which execute basic commands such as curl
  • Database dumps executed on a schedule (require NFS, but are limited to jobs currently)

Benefits (why should this be implemented?):

  • Image reuse - no need for every tool to build their own meta images, saving resources in the registry and on the worker nodes
  • Consistency across apis - no need for maintainers to manage both jobs and components
  • Reduce NFS usage - no nfs by default

Related Objects

Event Timeline

DamianZaremba created this task.Sep 22 2025, 5:52 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptSep 22 2025, 5:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• taavi subscribed.Sep 29 2025, 8:28 AM

On a first glance this is a duplicate of T362076: [components-api] Add support for pre-built images (ex. python3.11, to refine), although that claims this is already implemented but I don't see any mention of that on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Deploy_your_tool.

• taavi renamed this task from Build standard images under pack / support execution via components-api to Support pre-built images on components-api.Sep 29 2025, 8:29 AM
dcaro subscribed.Sep 29 2025, 8:30 AM
In T405262#11223516, @taavi wrote:

On a first glance this is a duplicate of T362076: [components-api] Add support for pre-built images (ex. python3.11, to refine), although that claims this is already implemented but I don't see any mention of that on https://wikitech.wikimedia.org/wiki/Help:Toolforge/Deploy_your_tool.

It was done, and then scraped out for the beta

DamianZaremba added a comment.Sep 29 2025, 8:34 AM

For now I have https://github.com/cluebotng/external-utilities which basically covers mariadb (curl actually is done in the report container now, but was in bookworm).

That works and perhaps is a good long term solution as it decouples maintenance/breakage from toolforge admins onto maintainers, but it also means I hit the build limits/quota more and generally things use more storage.... so in general low priority from my side, just throwing it out there.

• taavi triaged this task as High priority.Oct 1 2025, 2:07 PM
• taavi updated the task description. (Show Details)
fnegri edited projects, added: cloud-services-team (FY2025/2026-Q3-Q4); removed: cloud-services-team.Jan 13 2026, 4:04 PM
aputhin lowered the priority of this task from High to Medium.Apr 8 2026, 2:19 PM
aputhin subscribed.

Changing to medium prio after refinement. We need a bit more clarity on impact of the use cases for this feature. @dcaro to list the use cases we already identified.

aputhin added a project: tools-platform-team.Apr 8 2026, 2:23 PM
• bd808 subscribed.Apr 8 2026, 8:47 PM

It would be quite useful to be able to execute these images via components and not have to execute them on NFS workers (which appear to have the worst availability).

I don't think many of these images are useful without NFS as a source of application and data. Even the mariadb container we made to help folks do data dumps is pretty useless without NFS as a place to store the dumped data at least temporarily.

Image reuse - no need for every tool to build their own meta images, saving resources in the registry and on the worker nodes

This is legitimate need. I made the containers tool as a partial solution for sharable containers designed for various use cases. As far as I understand it the components API does not yet support the general functionality of providing a URI for an existing container as a substitute for a build definition. That does feel like something that can and should be improved.

dcaro added a comment.Apr 9 2026, 11:04 AM
In T405262#11799849, @aputhin wrote:

Changing to medium prio after refinement. We need a bit more clarity on impact of the use cases for this feature. @dcaro to list the use cases we already identified.

Some use cases (repeating the ones in the desc + new ones):

  • Cron jobs, which execute basic commands such as curl
  • Database dumps executed on a schedule (require NFS, but are limited to jobs currently)
  • Pywikibot oneliners
  • logrotate cronjobs

This is legitimate need. I made the containers tool as a partial solution for sharable containers designed for various use cases. As far as I understand it the components API does not yet support the general functionality of providing a URI for an existing container as a substitute for a build definition. That does feel like something that can and should be improved.

I thought there was a task for that, but can't find it, will create a separated one.

aputhin moved this task from Backlog to Todos on the tools-platform-team board.Apr 29 2026, 2:18 PM
aputhin edited projects, added: Toolforge (Push-to-Deploy); removed: Toolforge.Jun 3 2026, 10:15 AM
aputhin moved this task from Backlog to Stabilization phase on the Toolforge (Push-to-Deploy) board.Jul 8 2026, 2:52 PM
fnegri removed a project: cloud-services-team (FY2025/2026-Q3-Q4).Jul 13 2026, 3:25 PM
dcaro renamed this task from Support pre-built images on components-api to [components-api] Support pre-built images on components-api.Jul 14 2026, 5:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits