Page MenuHomePhabricator

[jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • Construct an application that uses a signal handler to shutdown cleanly
  • Have a fixed entrypoint (as supported by buildpack)

What happens?:

The fixed entrypoint is wrapped in sh which does not pass signals down to the main process.

This is historical logic to support filelogging, but is wrapping all buildpack with no filelogging in sh (no bash logic).

What should have happened instead?:

The fixed entrypoint should be run, with all signals passed from the runtime.

Other information (browser name/version, screenshots, etc.):

Example reproduction;

  1. Build and run application
`
$ toolforge build start --image-name broken-signals https://github.com/cluebotng/toolforge-broken-signal-handling.git
$ toolforge jobs run --image tool-cluebotng/broken-signals --command run-logic --continuous broken-signals
`
  1. Observe the launcher has been wrapped
tools.cluebotng@tools-bastion-15:~$ kubectl get deployment broken-signals -o json | jq .spec.template.spec.containers[0].command
[
  "/bin/sh",
  "-c",
  "--",
  "launcher run"
]
  1. Verify the signal handler functions as expected from within the container
tools.cluebotng@tools-bastion-15:~$ kubectl exec broken-signals-98799f87f-697s6 -- bash -c 'ps | grep -E "php$" | awk "{print \$1}" | xargs kill'
tools.cluebotng@tools-bastion-15:~$ kubectl logs broken-signals-98799f87f-697s6
Recieved signal: 15
  1. Verify the signal is not propagated from pid 1
tools.cluebotng@tools-bastion-15:~$ kubectl exec broken-signals-98799f87f-697s6 -- bash -c 'kill 1'
tools.cluebotng@tools-bastion-15:~$ kubectl logs broken-signals-98799f87f-697s6
Recieved signal: 15
  1. Verify no signal propagated from pod termination (as expected, since pid 1 eats it);
tools.cluebotng@tools-bastion-15:~$ kubectl logs broken-signals-98799f87f-697s6 -f &
[1] 245803
tools.cluebotng@tools-bastion-15:~$ Recieved signal: 15

tools.cluebotng@tools-bastion-15:~$ kubectl delete pod broken-signals-98799f87f-697s6
pod "broken-signals-98799f87f-697s6" deleted
[1]+  Done                    kubectl logs broken-signals-98799f87f-697s6 -f
tools.cluebotng@tools-bastion-15:~$

This has annoying side effects such as long running bots being able to save persistence on restart.

The file logging logic should not wrap in a shell unless it is needed - though that is easier said than done as the side effect is command supports shellisms - build pack is documented to not be supported in doing this (at least in terms of the launcher env), so that should be a smaller amount of surface area to blow up.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
functional-tests - builds-api - wrap commandrepos/cloud/toolforge/toolforge-deploy!1327damianfeature/update-buildpack-jobsmain
command - selectively wrap commandrepos/cloud/toolforge/jobs-api!333damianfeature/do-not-wrap-in-shell-unless-neededmain
Customize query in GitLab

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
DamianZaremba renamed this task from [builds-api] `step-fix-procfile-args-passing` causes signals (e.g. SIGTERM) to not be seen by application to [jobs-api] `get_command_for_k8s` causes signals (e.g. SIGTERM) to not be seen by application.Jun 29 2026, 5:48 PM
DamianZaremba updated the task description. (Show Details)

Example when command is not wrapped in sh;

tools.cluebotng@tools-bastion-15:~$ kubectl get deployment broken-signals -o json | jq .spec.template.spec.containers[0].command
[
  "launcher",
  "run-logic"
]

tools.cluebotng@tools-bastion-15:~$ kubectl logs broken-signals-f7c46b584-rf5sk -f &
[1] 259592

tools.cluebotng@tools-bastion-15:~$ kubectl delete pod broken-signals-f7c46b584-rf5sk
pod "broken-signals-f7c46b584-rf5sk" deleted
Recieved signal: 15
[1]+  Done                    kubectl logs broken-signals-f7c46b584-rf5sk -f

Interestingly the tests in jobs-api are verifying behaviour when command + args is used, not when the wrapper is used =\ I haven't gone all the way back through the history, but I think that comes from before jobs-api was a thing.

Initial proposal for un-breaking this (specifically for buildpack images), minimising the amount of exploding the rest of the world - https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/333

Will try and get lima-kilo working tomorrow to do some functional testing.

aputhin triaged this task as High priority.Jul 1 2026, 1:57 PM
aputhin subscribed.

@DamianZaremba thank you for working on this. Please ping us when the MR is ready and we can help test it out.

@DamianZaremba thank you for working on this. Please ping us when the MR is ready and we can help test it out.

Should be good to look at now.

group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1334

jobs-api: bump to 0.0.539-20260713213805-22bde880

Raymond_Ndibe moved this task from In review to Done on the tools-platform-team board.

Working as expected;

tools.cluebotng@tools-bastion-15:~$ kubectl get pod -l app.kubernetes.io/name=bot -o json | jq '.items[0].spec.containers[0].command, .items[0].spec.containers[0].livenessProbe.exec.command'
[
  "/bin/sh",
  "-c",
  "--",
  "launcher run-cbng"
]
[
  "/bin/sh",
  "-c",
  "health-check"
]

tools.cluebotng@tools-bastion-15:~$ toolforge jobs delete bot

tools.cluebotng@tools-bastion-15:~$ toolforge components deployment create
Deployment for cluebotng created successfully.
Deployment ID: 20260714-081506-7ykqooi6qs
Created: 20260714-081506
[....]

tools.cluebotng@tools-bastion-15:~$ kubectl get pod -l app.kubernetes.io/name=bot -o json | jq '.items[0].spec.containers[0].command, .items[0].spec.containers[0].livenessProbe.exec.command'
[
  "launcher",
  "run-cbng"
]
[
  "/bin/sh",
  "-c",
  "health-check"
]

We could also drop the sh from the liveness/startup probes, but that is less impactful

Signals are now working as expected;

tools.cluebotng@tools-bastion-15:~$ kubectl logs -f bot-74d9c5f44f-vhcdb &
[1] 3169809
tools.cluebotng@tools-bastion-15:~$ toolforge jobs restart bot
[2026-07-14T09:48:06.554690+01:00] cluebotng.INFO: Skipping: Below threshold {"revision_id":1364072116,"score":"0.010758"} []
[2026-07-14T09:48:06.963547+01:00] cluebotng.INFO: Received shutdown signal 15, beginning graceful shutdown [] []
[2026-07-14T09:48:06.963735+01:00] cluebotng.INFO: HttpFeed shutting down, no longer processing new events [] []
[2026-07-14T09:48:06.966920+01:00] cluebotng.INFO: Persisted last event id on shutdown: [{"offset":-1,"partition":0,"topic":"codfw.mediawiki.recentchange"},{"topic":"eqiad.mediawiki.recentchange","partition":0,"timestamp":1784018886882}] [] []
[2026-07-14T09:48:06.976330+01:00] cluebotng.INFO: EventStream stopped [] []
[2026-07-14T09:48:06.976508+01:00] cluebotng.INFO: Waiting for 0 child process(es) to finish [] []
[2026-07-14T09:48:06.976594+01:00] cluebotng.INFO: Shutdown complete, exiting [] []