As a workaround you can read /var/run/secrets/kubernetes.io/serviceaccount/namespace and strip off the tool- prefix from inside the pod.

Verified this was an oversight when running a clean (remove all objects) deployment for T403735

First alert received at 05:57 (cest)

T403735 fixed the literal cause of this, but I'll leave it open as the pending MR handles the more general case of failing to load things from runtime, which is worthwhile.

Before;

tools.cluebotng@tools-bastion-15:~$ kubectl get pod bot-57f4859f94-4rdc8 -o json | jq '.spec.containers[0].livenessProbe.exec.command'
[
  "/bin/sh",
  "-c",
  "health-check"
]

In T432592#12140722, @Danysan1 wrote:

That is exactly the bug this task covers.

In T432586#12136256, @taavi wrote:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/lib/venv/lib/python3.13/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.scope, self.receive, self.send
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/lib/venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/venv/lib/python3.13/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/opt/lib/venv/lib/python3.13/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 177, in __call__
    raise exc
  File "/opt/lib/venv/lib/python3.13/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 175, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/routing.py", line 716, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/routing.py", line 736, in app
    await route.handle(scope, receive, send)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/routing.py", line 290, in handle
    await self.app(scope, receive, send)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/routing.py", line 78, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/routing.py", line 75, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/opt/lib/venv/lib/python3.13/site-packages/fastapi/routing.py", line 302, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
    )
    ^
  File "/opt/lib/venv/lib/python3.13/site-packages/fastapi/routing.py", line 215, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/venv/lib/python3.13/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
    return await anyio.to_thread.run_sync(func)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lib/venv/lib/python3.13/site-packages/anyio/to_thread.py", line 63, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        func, args, abandon_on_cancel=abandon_on_cancel, limiter=limiter
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/lib/venv/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 2596, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/opt/lib/venv/lib/python3.13/site-packages/anyio/_backends/_asyncio.py", line 1029, in run
    result = context.run(func, *args)
  File "/app/tjf/api/jobs.py", line 73, in api_get_jobs
    user_jobs = current_app(request).core.get_jobs(tool_name=tool_name)
  File "/app/tjf/core/core.py", line 260, in get_jobs
    runtime_job = self.runtime.get_continuous_job(
        job_name=storage_job.job_name, tool_name=tool_name
    )
  File "/app/tjf/runtimes/k8s/runtime.py", line 190, in get_continuous_job
    job = get_continuous_job_from_k8s_object(
        k8s_object=k8s_obj,
        default_cpu_limit=self.default_cpu_limit,
        tool_name=tool_name,
    )
  File "/app/tjf/runtimes/k8s/jobs.py", line 667, in get_continuous_job_from_k8s_object
    script = container_spec["startupProbe"]["exec"]["command"][2]
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Haha, ok so that is from me editing the deployment object to verify T403735 works.

Actually jobs-cli just seems generally broken;

tools.cluebotng@tools-bastion-15:~$ toolforge jobs show bot
ERROR: TjfCliError: Unknown error (list index out of range)
ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu

@dcaro could you tag with needs review when you have chance please

I noticed this again on T430578, which simplifies the behaviour for the main command.

In T432262#12125900, @bd808 wrote:

In T432262#12125843, @DamianZaremba wrote:

I think this could be fixed with rel=noopener on the toolhub side also (I assume the protection is there to stop "remote" access to toolsadmin (which had elevated permissions in the form of account management etc).

The docs at https://developer.mozilla.org/en-US/docs/Web/HTML/Reference/Attributes/rel/noopener say that target="_blank" does the same thing as rel=noopener. In some quick local testing adding rel=noopener explicitly doesn't seem to change anything.

In T423549#12125257, @komla wrote:

We want to know if an average of 300 SSH logins are seen a day over time and we are now seeing only 30 a day, we can tell activity level has significantly dropped.

@bd808 you're probably right.

In T432097#12121462, @DamianZaremba wrote:

Thanks @dcaro, I'll push some releases tomorrow (eu) late morning/afternoon and let you know if anything comes up.

In T432115#12120872, @dcaro wrote:

This is sorted out, I'll create a task for the pods getting stuck in waiting for 5h+ and see how we surface that issue.

Thanks @dcaro, I'll push some releases tomorrow (eu) late morning/afternoon and let you know if anything comes up.

FWIW that job was re-created (https://phabricator.wikimedia.org/T430578#12118143) earlier today prior to the above, so it was in a pretty 'clean' state.

Signals are now working as expected;

tools.cluebotng@tools-bastion-15:~$ kubectl logs -f bot-74d9c5f44f-vhcdb &
[1] 3169809
tools.cluebotng@tools-bastion-15:~$ toolforge jobs restart bot
[2026-07-14T09:48:06.554690+01:00] cluebotng.INFO: Skipping: Below threshold {"revision_id":1364072116,"score":"0.010758"} []
[2026-07-14T09:48:06.963547+01:00] cluebotng.INFO: Received shutdown signal 15, beginning graceful shutdown [] []
[2026-07-14T09:48:06.963735+01:00] cluebotng.INFO: HttpFeed shutting down, no longer processing new events [] []
[2026-07-14T09:48:06.966920+01:00] cluebotng.INFO: Persisted last event id on shutdown: [{"offset":-1,"partition":0,"topic":"codfw.mediawiki.recentchange"},{"topic":"eqiad.mediawiki.recentchange","partition":0,"timestamp":1784018886882}] [] []
[2026-07-14T09:48:06.976330+01:00] cluebotng.INFO: EventStream stopped [] []
[2026-07-14T09:48:06.976508+01:00] cluebotng.INFO: Waiting for 0 child process(es) to finish [] []
[2026-07-14T09:48:06.976594+01:00] cluebotng.INFO: Shutdown complete, exiting [] []

Manually fixed with

tools.cluebotng@tools-bastion-15:~$ toolforge jobs delete bot
tools.cluebotng@tools-bastion-15:~$ toolforge components deployment create
tools.cluebotng@tools-bastion-15:~$ toolforge components deployment show
Deployment ID: 20260714-084623-lup2bz3d76
Created: 20260714-084623
Status: successful
Long status: 
  Finished at 2026-07-14 08:47:10.272403

Working as expected;

tools.cluebotng@tools-bastion-15:~$ kubectl get pod -l app.kubernetes.io/name=bot -o json | jq '.items[0].spec.containers[0].command, .items[0].spec.containers[0].livenessProbe.exec.command'
[
  "/bin/sh",
  "-c",
  "--",
  "launcher run-cbng"
]
[
  "/bin/sh",
  "-c",
  "health-check"
]

Should be fixed by T402568

In T430578#12076220, @aputhin wrote:

@DamianZaremba thank you for working on this. Please ping us when the MR is ready and we can help test it out.

I didn't managed to grab the logs. I suspect my connectivity to be part of the issue (specifically for pulling docker images) - connecting to a VPN seemed to help things along.

Eventually;

damian@lima-kilo:~$ kubectl get pods -A
NAMESPACE            NAME                                              READY   STATUS    RESTARTS   AGE
foxtrot-ldap         foxtrot-ldap-0                                    1/1     Running   0          13m
kube-system          coredns-668d6bf9bc-gqfc7                          1/1     Running   0          38m
kube-system          coredns-668d6bf9bc-wqvkv                          1/1     Running   0          38m
kube-system          etcd-toolforge-control-plane                      1/1     Running   1          40m
kube-system          kindnet-6p899                                     1/1     Running   0          4m45s
kube-system          kube-apiserver-toolforge-control-plane            1/1     Running   1          40m
kube-system          kube-controller-manager-toolforge-control-plane   1/1     Running   1          40m
kube-system          kube-proxy-tsnqp                                  1/1     Running   1          38m
kube-system          kube-scheduler-toolforge-control-plane            1/1     Running   1          40m
local-path-storage   local-path-provisioner-567f868bf9-lwh5n           1/1     Running   0          38m

It would be useful if the kind/k8s setup role checked the cluster is healthy before finishing.

Kubernetes is not ready;

damian@lima-kilo:~$ kubectl get pods -A
NAMESPACE            NAME                                              READY   STATUS             RESTARTS   AGE
foxtrot-ldap         foxtrot-ldap-0                                    0/1     Pending            0          5m50s
kube-system          coredns-668d6bf9bc-gqfc7                          0/1     Pending            0          30m
kube-system          coredns-668d6bf9bc-wqvkv                          0/1     Pending            0          30m
kube-system          etcd-toolforge-control-plane                      1/1     Running            1          33m
kube-system          kindnet-k89vf                                     0/1     ImagePullBackOff   0          30m
kube-system          kube-apiserver-toolforge-control-plane            1/1     Running            1          33m
kube-system          kube-controller-manager-toolforge-control-plane   1/1     Running            1          33m
kube-system          kube-proxy-tsnqp                                  1/1     Running            1          30m
kube-system          kube-scheduler-toolforge-control-plane            1/1     Running            1          33m
local-path-storage   local-path-provisioner-567f868bf9-lwh5n           0/1     Pending            0          30m

This would be a significant improvement, as a data point comparing a (Python based) build (yesterday) that uses the Toolforge image vs the current upstream, trivy reports 21 high severity vulnerabilities in the Toolforge image and 0 in the upstream image. Frankly I don't have time to regularly review 20+ reports across 20+ repos, to determine if they are actually impactful, so having them just removed from the runtime would be of beneficial to probably all maintainers.

In T429638#12045147, @Ladsgroup wrote:

Cross referencing: https://github.com/cluebotng/bot/issues/58

Initial proposal for un-breaking this (specifically for buildpack images), minimising the amount of exploding the rest of the world - https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333

Interestingly the tests in jobs-api are verifying behaviour when command + args is used, not when the wrapper is used =\ I haven't gone all the way back through the history, but I think that comes from before jobs-api was a thing.

Example when command is not wrapped in sh;

tools.cluebotng@tools-bastion-15:~$ kubectl get deployment broken-signals -o json | jq .spec.template.spec.containers[0].command
[
  "launcher",
  "run-logic"
]

In T417225#12065949, @aputhin wrote:

@DamianZaremba for what you are trying to do for longer than 10 mins, what would be a good timeout that doesn't disrupt it?

We could look into increasing the config as a quick fix at least, but want to understand what is reasonable for users' needs (e.g. if we shoot it to 30 mins, would it help or not enough?)

Re-opening as this is happening again.

In T417225#12062324, @DamianZaremba wrote:

I've hit this multiple times today while tailing logs, so defiantly still happening.

I've hit this multiple times today while tailing logs, so defiantly still happening.

In T429229#12052371, @Buxtang wrote:

However I don't seem to have a contributor role to the build-cli repo so I can't make a branch. How should I seek access to this "builds-cli" repo and "components-cli" for the second issue?

There is a similar issue with components deployment list where it exits with a failure code (2) when there are no deployments, due to the API response, which would also be a simple thing to fix.

@aputhin it would be good to get to the bottom of the transient errors as it causes hassle for maintainers every time it happens, which goes through phases of being quite regularly.

Is not happening currently (logs are being returned), so there appears to be some transient error that was present overnight

This might also explain why some deployments are restarting jobs that have no changes e.g.

Deployment ID: 20260616-165737-i4ix4154s0

Yes, I also noticed that with another account yesterday that doesn't emit the warning.

Updated cluebotng-review to keep the daily backup on nfs and the intra-daily in object storage with a short retention, nfs usage should now be below 15G.

I can't replicate this at the moment on any tool, so it appears to have resolved it's self.

Deployment is now succeeding as expected;

m00m00:component-configs damian$ TARGET_USER=cluebotng-review fab deploy
Applying to cluebotng-review: NetworkPolicy(name=allow-irc-relay-access-from-all-tool-jobs)
Applying to cluebotng-review: NetworkPolicy(name=restrict-grafana-alloy)
Applying to cluebotng-review: NetworkPolicy(name=restrict-pushgateway)
Applying to cluebotng-review: NetworkPolicy(name=restrict-core)
Applying to cluebotng-review: NetworkPolicy(name=restrict-monitoring-toolforge)
Applying config for cluebotng-review
Configuration for cluebotng-review updated successfully.
Warning: You are using a beta feature of Toolforge.
Started deployment: 20260611-132151-xyer06k07m
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment is pending or in progress
Deployment has finished successfully
Applying to cluebotng-review: HttpRoute(backend=cluebotng-reviewer, port=8000)
Writing trove-mysql.sh to /data/project/cluebotng-review/trove-mysql (750)

Added the same workaround to cluebot3 which is also failing building (https://github.com/cluebotng/cluebot3/commit/71ad1fa819e07a8cf6a5541cb0f551fad836ce9e)

Once T428214 is resolved I can adjust the retention, this has over the last months with ~6.5k edits added last week.

Also getting failures when interacting with the build api directly;

tools.cluebotng@tools-bastion-15:~$ toolforge build start --ref main -i core https://github.com/cluebotng/external-core.git
ReadTimeout: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=20)
Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu

This is still happening, so doesn't appear to be a transient issue.

Reduced cpu request @ https://github.com/cluebotng/component-configs/commit/874d7f6f407fc9a3995f52f40312cd7d3a712176.

In the last few weeks this has caused an outage on ClueBot NG twice (as dependency updates happen).

Workaround is https://github.com/cluebotng/bot/commit/468143e5eb21d0b3b620623294e0f9a454932abb which prevents the default web entry.

This seems to break when there is not a web process.

In T424191#11852377, @Raymond_Ndibe wrote:

Hello Damian what does your config look like?

I've tried this with with a bare executable (script) @ https://github.com/cluebotng/bot/blob/v2.0.4/Procfile rather than using args @ https://github.com/cluebotng/bot/blob/v2.0.3/Procfile and it fails in the same way.

Confirming my tools are now working (with http health checking) as expected.

The culprit seems to be
https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/blob/main/openapi/openapi.yaml?ref_type=heads&plain=1#L784

I was going to try and re-produce this on staging (removing the health check), but it now seems to be stuck waiting for builds to start (20260409-180511-p2a131uce6) =\

That was changed in https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/commit/1b5c88374b0986e9b2a260b7ed55a633e113897a#line_d9ffab664_A154 and the test in https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/commit/1b5c88374b0986e9b2a260b7ed55a633e113897a#line_46ecef6a9_A432, which I guess was based on https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/commit/1b5c88374b0986e9b2a260b7ed55a633e113897a#line_e66c439c6_A387 (generated) changing.

In T422046#11778804, @dcaro wrote:

Pack will not be able to reproduce the build currently, you'll need also the runner and the whole environment that builds-api sets up (ex. custom scripts, injected buildpacks, etc.).

This would work for you right now maybe, but will break for others :/, maybe a better path would be to try to create our own builder/runner combo with all that stuff in in, so it can be used with pack. Might be possible in the mid term (we are getting some new hires soonish).

Something like https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/157 to enable something like https://github.com/InfraBits/ci-pack/pull/1

All of the cluebot tools are running with the latest builder image (also in CI), the only small issue that came up was opcache wanting an existing path with the newer version of php, otherwise everything is working as expected.

Checked my deployments yesterday, all appear to work as expected with the digest.

Workaround for now is to explicitly restart the job, which forces a template change

There seems to be striping of the digest happening also

tools.cluebot3@tools-bastion-15:~$ cat tmp.yaml
- command: run-bot
  continuous: true
  cpu: '3.0'
  health-check-script: health-check
  image: tools-harbor.wmcloud.org/tool-cluebot3/cluebot3:latest@sha256:198dd4eea7e29db57aec7022d751c8d4898a4ac62891d0ed4166231641295fd6
  mem: 1.0Gi
  name: cluebot3