[components-api] updated image is not restarted
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	DamianZaremba
	Feb 15 2026, 1:35 PM

Project Tags

Referenced Files

None

Subscribers

Description

Steps to replicate the issue (include links if applicable):

Create a new deployment, the image is build and the job run

tools.cluebot3@tools-bastion-15:~$ toolforge components deployment show
Deployment ID: 20260215-132223-9slqnf335x
Created: 20260215-132223
Status: successful
Long status: 
  Finished at 2026-02-15 13:23:06.180435

Builds:
  cluebot3(successful): id:cluebot3-buildpacks-pipelinerun-w9dkj You can see the logs with `toolforge build logs cluebot3-buildpacks-pipelinerun-w9dkj`

Runs:
  cluebot3(successful): created or updated job cluebot3, [info](Job cluebot3 updated)

Tool config:
  components:
    cluebot3:
      build:
        ref: refs/tags/v1.4.0
        repository: https://github.com/cluebotng/cluebot3.git
        use_latest_versions: true
      run:
        command: run-bot
        cpu: '3'
        health_check_script: health-check
        memory: 1Gi

What happens?:

jobs-api reports that the job was updated, which causes components-api to skip explicitly restarting it (https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/blob/main/components/runtime/toolforge.py?ref_type=heads#L406)

However the job has not been updated/restarted by jobs-api:

tools.cluebot3@tools-bastion-15:~$ kubectl get deployment cluebot3
NAME       READY   UP-TO-DATE   AVAILABLE   AGE
cluebot3   1/1     1            1           93d

tools.cluebot3@tools-bastion-15:~$ kubectl get pods cluebot3-5584db5cf9-zh5td 
NAME                        READY   STATUS    RESTARTS       AGE
cluebot3-5584db5cf9-zh5td   1/1     Running   277 (8h ago)   31d

What should have happened instead?:

The deployment should be updated causing the hash to change causing the pod to be restarted.

Ideally the image should be set to that returned by builds-api

tools.cluebot3@tools-bastion-15:~$ kubectl get deployment cluebot3 -o json | jq '.spec.template.spec.containers[0].image'
"tools-harbor.wmcloud.org/tool-cluebot3/cluebot3:latest"

tools.cluebot3@tools-bastion-15:~$ toolforge build show --json | jq .build.destination_image
"tools-harbor.wmcloud.org/tool-cluebot3/cluebot3:latest@sha256:198dd4eea7e29db57aec7022d751c8d4898a4ac62891d0ed4166231641295fd6"

This seems to be a recurrence of the job having some junk in the diff causing job_changed to be true, but without a meaningful change.

Event Timeline

DamianZaremba created this task.Feb 15 2026, 1:35 PM

Restricted Application added a project: cloud-services-team. · View Herald TranscriptFeb 15 2026, 1:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

There seems to be striping of the digest happening also

tools.cluebot3@tools-bastion-15:~$ cat tmp.yaml
- command: run-bot
  continuous: true
  cpu: '3.0'
  health-check-script: health-check
  image: tools-harbor.wmcloud.org/tool-cluebot3/cluebot3:latest@sha256:198dd4eea7e29db57aec7022d751c8d4898a4ac62891d0ed4166231641295fd6
  mem: 1.0Gi
  name: cluebot3

tools.cluebot3@tools-bastion-15:~$ toolforge jobs load tmp.yaml
INFO: loading job 'cluebot3'...
Job cluebot3 updated
INFO: 1 job(s) loaded successfully

tools.cluebot3@tools-bastion-15:~$ toolforge jobs dump
WARNING: unexpected key 'job_type' in job, omitting it from dump
- command: run-bot
  continuous: true
  cpu: '3.0'
  health-check-script: health-check
  image: tool-cluebot3/cluebot3:latest
  mem: 1.0Gi
  name: cluebot3

tools.cluebot3@tools-bastion-15:~$ kubectl get deployment cluebot3 -o json | jq '.spec.template.spec.containers[0].image'
"tools-harbor.wmcloud.org/tool-cluebot3/cluebot3:latest"

Workaround for now is to explicitly restart the job, which forces a template change

It seems like when an image with digest (e.g. '192.168.5.15/tool-tf-test/cluebot3:latest@sha256:b43fe64ac24365bd7cf3731f010e08020b6ce7304dd7852cd689600318ff270d') is provided while attempting to create job, what ends up in k8s is something like 192.168.5.15/tool-tf-test/cluebot3:latest, the digest is being dropped.
This is happening because we are running Image.from_url_or_name again here https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/main/tjf/runtimes/k8s/runtime.py?ref_type=heads#L191, which drops the digest.
I'm not sure why we are doing that again in that line of code, since we already ran it in the api/models.py. Will just remove it and see what happens.