Page MenuHomePhabricator
Create Task
Maniphest T428449

[jobs-api] is not listing all container images for a tool
Closed, ResolvedPublicBUG REPORT
Actions

Description

A build was triggered from a deployment, completed successfully, but then was not found by jobs-api

tools.cluebotng-review@tools-bastion-15:~$ toolforge components deployment show
Deployment ID: 20260608-130414-oqi9cstmgr
Created: 20260608-130414
Status: failed
Long status: 
  Got exception: Failed run for component irc-relay: 400 Client Error: Bad Request for url: https://api.svc.tools.eqiad1.wikimedia.cloud:30003/jobs/v1/tool/cluebotng-review/jobs/ (400): No such image 'tools-harbor.wmcloud.org/tool-cluebotng-review/irc-relay:latest@sha256:7a95172b4ad1797b1341502e4edf508fa3c02b9f558c43bebdb2c2557aaa8f60'

Builds:
  irc-relay(successful): id:cluebotng-review-buildpacks-pipelinerun-jxlcx You can see the logs with `toolforge build logs cluebotng-review-buildpacks-pipelinerun-jxlcx`

Builds api reports it as there

tools.cluebotng-review@tools-bastion-15:~$ toolforge build show cluebotng-review-buildpacks-pipelinerun-jxlcx
Build ID: cluebotng-review-buildpacks-pipelinerun-jxlcx
Start Time: 2026-06-08T13:04:00Z
End Time: 2026-06-08T13:04:51Z
Status: ok
Message: Tasks Completed: 1 (Failed: 0, Cancelled 0), Skipped: 0
Parameters:
    Source URL: https://github.com/cluebotng/irc-relay.git
    Ref: refs/tags/v2.0.81
    Envvars: N/A
Use latest versions: False
Use deprecated versions: False
Destination Image: tools-harbor.wmcloud.org/tool-cluebotng-review/irc-relay:latest@sha256:7a95172b4ad1797b1341502e4edf508fa3c02b9f558c43bebdb2c2557aaa8f60

Jobs api reports it as not

tools.cluebotng-review@tools-bastion-15:~$ toolforge jobs images
+----------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
|                     Short name                     |                                                                        image                                                                        |
+----------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
|                      bookworm                      |                                        docker-registry.tools.wmflabs.org/toolforge-bookworm-web-sssd:latest                                         |
|                      bullseye                      |                                          docker-registry.tools.wmflabs.org/toolforge-bullseye-sssd:latest                                           |
|                       jdk17                        |                                          docker-registry.tools.wmflabs.org/toolforge-jdk17-sssd-web:latest                                          |
|                       jdk21                        |                                          docker-registry.svc.toolforge.org/toolforge-jdk21-sssd-web:latest                                          |
|                      mariadb                       |                                        docker-registry.tools.wmflabs.org/toolforge-mariadb-sssd-base:latest                                         |
|                      mono6.12                      |                                        docker-registry.tools.wmflabs.org/toolforge-mono612-sssd-base:latest                                         |
|                      mono6.8                       |                                         docker-registry.tools.wmflabs.org/toolforge-mono68-sssd-base:latest                                         |
|                       node16                       |                                         docker-registry.tools.wmflabs.org/toolforge-node16-sssd-web:latest                                          |
|                       node18                       |                                         docker-registry.tools.wmflabs.org/toolforge-node18-sssd-web:latest                                          |
|                       node20                       |                                         docker-registry.svc.toolforge.org/toolforge-node20-sssd-web:latest                                          |
|                      perl5.32                      |                                         docker-registry.tools.wmflabs.org/toolforge-perl532-sssd-web:latest                                         |
|                      perl5.36                      |                                         docker-registry.tools.wmflabs.org/toolforge-perl536-sssd-web:latest                                         |
|                      perl5.40                      |                                         docker-registry.svc.toolforge.org/toolforge-perl540-sssd-web:latest                                         |
|                       php7.4                       |                                          docker-registry.tools.wmflabs.org/toolforge-php74-sssd-web:latest                                          |
|                       php8.2                       |                                          docker-registry.tools.wmflabs.org/toolforge-php82-sssd-web:latest                                          |
|                       php8.4                       |                                          docker-registry.svc.toolforge.org/toolforge-php84-sssd-web:latest                                          |
|                     python3.11                     |                                        docker-registry.tools.wmflabs.org/toolforge-python311-sssd-web:latest                                        |
|                     python3.13                     |                                        docker-registry.svc.toolforge.org/toolforge-python313-sssd-web:latest                                        |
|                     python3.9                      |                                        docker-registry.tools.wmflabs.org/toolforge-python39-sssd-web:latest                                         |
|                      ruby2.1                       |                                         docker-registry.tools.wmflabs.org/toolforge-ruby21-sssd-web:latest                                          |
|                      ruby2.7                       |                                         docker-registry.tools.wmflabs.org/toolforge-ruby27-sssd-web:latest                                          |
|                      ruby3.1                       |                                         docker-registry.tools.wmflabs.org/toolforge-ruby31-sssd-web:latest                                          |
|                      ruby3.3                       |                                         docker-registry.svc.toolforge.org/toolforge-ruby33-sssd-web:latest                                          |
|                       tcl8.6                       |                                          docker-registry.tools.wmflabs.org/toolforge-tcl86-sssd-web:latest                                          |
| tool-cluebotng-review/backup-database-daily:latest | tools-harbor.wmcloud.org/tool-cluebotng-review/backup-database-daily:latest@sha256:d7af79ad145224e6a48e828ecaa7b9f89f90b8d50b7d5031cd3b6aea2e4467a1 |
|    tool-cluebotng-review/backup-database:latest    |    tools-harbor.wmcloud.org/tool-cluebotng-review/backup-database:latest@sha256:d007d7b01bb018583d421ac65841bf1e02d681b34ec1ee00c85e78ac1ce532e0    |
|  tool-cluebotng-review/cluebotng-reviewer:latest   |  tools-harbor.wmcloud.org/tool-cluebotng-review/cluebotng-reviewer:latest@sha256:9307314140b4e61f944217d72ea503db0b8657087c598ecf84bb2ca0f4dc9681   |
|         tool-cluebotng-review/core:latest          |         tools-harbor.wmcloud.org/tool-cluebotng-review/core:latest@sha256:21c3c364e8dd8cfd98d326b4e69161a594a13b998ac83c514d528c1d6a01ddcf          |
|     tool-cluebotng-review/grafana-alloy:latest     |     tools-harbor.wmcloud.org/tool-cluebotng-review/grafana-alloy:latest@sha256:51cbe87dc15b87be7744694724f88972852db7f55db2b2b032957372981285a7     |
|      tool-cluebotng-review/pushgateway:latest      |      tools-harbor.wmcloud.org/tool-cluebotng-review/pushgateway:latest@sha256:9a48f6e003984c48e0df59b62e38405c09c260b41e61273b9a10bc4dacccf435      |
|         tool-cluebotng-review/redis:latest         |         tools-harbor.wmcloud.org/tool-cluebotng-review/redis:latest@sha256:b038d7883b36f0c7c319cd856394403346ea3243689da2dd0b0829ae9dc3d5ac         |
|                       trixie                       |                                         docker-registry.svc.toolforge.org/toolforge-trixie-web-sssd:latest                                          |
+----------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+

Either the image is not being persisted into harbor or jobs-api is returning some cached result

This appears to be occurring regularly and in combination with T428214 is making deployments very difficult.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
core/images.py: fetch all images for a toolrepos/cloud/toolforge/jobs-api!309fnegriT428449main
Customize query in GitLab

Related Objects

Event Timeline

DamianZaremba created this task.Jun 8 2026, 1:10 PM
Restricted Application added a project: cloud-services-team. · View Herald TranscriptJun 8 2026, 1:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
DamianZaremba updated the task description. (Show Details)Jun 8 2026, 1:12 PM
aputhin edited projects, added: tools-platform-team; removed: cloud-services-team.Jun 10 2026, 7:44 AM
fnegri subscribed.EditedJun 10 2026, 8:18 AM

The irc-relay image in Harbor matches the sha of the one shown in toolforge build list:

tools.cluebotng-review@tools-bastion-15:~$ toolforge build list |grep irc-relay
cluebotng-review-buildpacks-pipelinerun-c89t4  ok        2026-06-09T17:56:24Z  2026-06-09T17:57:17Z  https://github.com/cluebotng/irc-relay.git               refs/tags/v2.0.81  N/A        False                  False                      tools-harbor.wmcloud.org/tool-cluebotng-review/irc-relay:latest@sha256:90ed1fe3af0f6198fcae4c6ee5cc402f8b0798d89947c2ba04e7715832ce20b5

But toolforge jobs list does not find it, so it looks like a bug in jobs-api.

fnegri added a comment.EditedJun 10 2026, 8:44 AM

Found the issue, we're only fetching the first 25 images for a given tool from Harbor, and cluebotng-review has 27, so irc-relay is never fetched.

fnegri renamed this task from [builds-api] images are not persisted / [jobs-api] is incorrectly caching tool container images to [jobs-api] is not listing all container images for a tool.Jun 10 2026, 8:47 AM
CodeReviewBot added a project: Patch-For-Review.Jun 10 2026, 8:50 AM

fnegri opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/309

core/images.py: fetch up to 50 images for a tool

fnegri claimed this task.Jun 10 2026, 4:04 PM
fnegri moved this task from Backlog to In review on the tools-platform-team board.
aputhin triaged this task as Medium priority.Jun 10 2026, 4:12 PM
Raymond_Ndibe subscribed.Jun 10 2026, 4:26 PM

I also left on comment on https://phabricator.wikimedia.org/T428214 for you @DamianZaremba . I was unable to reproduce the issue you are describing on the other task. Could you provide steps to reproduce the issue for that task too ?

CodeReviewBot added a comment.Jun 11 2026, 12:47 PM

fnegri merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/309

core/images.py: fetch all images for a tool

CodeReviewBot added a comment.Jun 11 2026, 12:51 PM

group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291

jobs-api: bump to 0.0.502-20260611124739-a22e5bf8

Stashbot added a comment.Jun 11 2026, 12:55 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-11T12:55:49Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T428449)

Stashbot added a comment.Jun 11 2026, 1:08 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-11T13:08:14Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T428449)

Stashbot added a comment.Jun 11 2026, 1:18 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-11T13:18:15Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component jobs-api (T428449)

fnegri added a comment.Jun 11 2026, 1:20 PM

@DamianZaremba the patch was deployed and I can now see the image in toolforge jobs images:

tools.cluebotng-review@tools-bastion-15:~$ toolforge jobs images |grep irc-relay
|       tool-cluebotng-review/irc-relay:latest       |       tools-harbor.wmcloud.org/tool-cluebotng-review/irc-relay:latest@sha256:182bf2230c7d727947288cfb2c904ae940a2b1827638c3ec46d8d9ecaea3a33a       |

It should work in components-api as well.

DamianZaremba added a comment.Jun 11 2026, 1:25 PM

Deployment is now succeeding as expected;

m00m00:component-configs damian$ TARGET_USER=cluebotng-review fab deploy
Applying to cluebotng-review: NetworkPolicy(name=allow-irc-relay-access-from-all-tool-jobs)
Applying to cluebotng-review: NetworkPolicy(name=restrict-grafana-alloy)
Applying to cluebotng-review: NetworkPolicy(name=restrict-pushgateway)
Applying to cluebotng-review: NetworkPolicy(name=restrict-core)
Applying to cluebotng-review: NetworkPolicy(name=restrict-monitoring-toolforge)
Applying config for cluebotng-review
Configuration for cluebotng-review updated successfully.
Warning: You are using a beta feature of Toolforge.
Started deployment: 20260611-132151-xyer06k07m
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment has finished successfully
Applying to cluebotng-review: HttpRoute(backend=cluebotng-reviewer, port=8000)
Writing trove-mysql.sh to /data/project/cluebotng-review/trove-mysql (750)
Stashbot added a comment.Jun 11 2026, 1:31 PM

Mentioned in SAL (#wikimedia-cloud-feed) [2026-06-11T13:31:11Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api (T428449)

CodeReviewBot added a comment.Jun 11 2026, 1:32 PM

fnegri merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1291

jobs-api: bump to 0.0.502-20260611124739-a22e5bf8

fnegri closed this task as Resolved.Jun 11 2026, 2:41 PM
fnegri moved this task from In review to Done on the tools-platform-team board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits