Page MenuHomePhabricator
Create Task
Maniphest T382865

[jobs-api,jobs-cli] increased exit code 137 rate since 2024-12-14 due to low ephemeral-storage
Closed, ResolvedPublic
Actions

Description

I have received 33 emails for job failures with one of the two below reasons for jobs that should have ended successfully (i.e., no application error terminating the pod).

  • Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
  • Exit code was '137'. With reason 'Error'.

Usually, I have only seen 137 when a k8s node is drained/restarted.

* Job 'dfy-tagger' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'dfy-tagger-28902840-q7h44'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28904197-btkm8'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28904197-btkm8'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28904147-hl5dd'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-15T07:47:01Z. Finish timestamp 2024-12-16T03:36:57Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdwl' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdwl-28905157-qp8tm'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-16T00:37:02Z. Finish timestamp 2024-12-16T19:43:20Z. Exit code was '137'. With reason 'Error'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28907077-h9mxt'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28907077-h9mxt'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'admin-activity-early' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'admin-activity-early-28909447-lm5wt'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'admin-activity-early' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'admin-activity-early-28909447-lm5wt'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'massmessage-list-updater' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'massmessage-list-updater-28910893-vjstw'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-20T00:13:18Z. Finish timestamp 2024-12-20T00:14:08Z. Exit code was '137'. With reason 'Error'.
* Job 'massmessage-list-updater' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'massmessage-list-updater-28910893-vjstw'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-20T00:13:18Z. Finish timestamp 2024-12-20T00:14:08Z. Exit code was '137'. With reason 'Error'
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28913897-frfjt'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28913897-frfjt'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28916657-gzdgd'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28916657-gzdgd'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'massmessage-list-updater' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'massmessage-list-updater-28918093-r9hkb'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-25T00:13:23Z. Finish timestamp 2024-12-25T00:14:45Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28920737-gqdhp'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-26T20:17:18Z. Finish timestamp 2024-12-26T20:19:41Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28920737-gqdhp'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-26T20:17:18Z. Finish timestamp 2024-12-26T20:19:41Z. Exit code was '137'. With reason 'Error'.
* Job 'admin-activity-early' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'admin-activity-early-28920967-wncl4'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'admin-activity-early' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'admin-activity-early-28920967-wncl4'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28922117-5tfs7'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-27T19:17:20Z. Finish timestamp 2024-12-27T19:20:48Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28922117-5tfs7'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-27T19:17:20Z. Finish timestamp 2024-12-27T19:20:48Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28922417-dqn7r'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28922417-dqn7r'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'all-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'all-vios-28923268-sfx8b'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-28T14:28:14Z. Finish timestamp 2024-12-28T14:40:36Z. Exit code was '137'. With reason 'Error'.
* Job 'purge-weekly' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-weekly-28925367-n28t6'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2024-12-30T01:27:17Z. Finish timestamp 2024-12-30T01:35:58Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28925747-rt9cg'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28925747-rt9cg'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28928197-r887w'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-01T00:37:18Z. Finish timestamp 2025-01-01T01:12:43Z. Exit code was '137'. With reason 'Error'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28928197-r887w'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-01T00:37:18Z. Finish timestamp 2025-01-01T01:12:43Z. Exit code was '137'. With reason 'Error'.
* Job 'purge-dup-args' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-dup-args-28929393-d65lf'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'purge-dup-args' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-dup-args-28929393-d65lf'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'admin-activity-early' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'admin-activity-early-28932487-8hjbx'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'commons-potd-importer' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'commons-potd-importer-28935367-gc2mq'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28936397-7m5lm'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28940677-pbwj9'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28943207-lfxgd'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28943807-gjv4h'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28944977-92t6t'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'purge-dup-args' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-dup-args-28944993-9jz8g'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28944617-4fjtq'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28944977-92t6t'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28945127-cwfv2'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'purge-cat-redir' (normal) (emails: onfailure) had 1 events:
  -- Pod 'purge-cat-redir-k7ztx'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-03T02:06:27Z. Finish timestamp 2025-01-12T22:32:30Z. Exit code was '137'. With reason 'Error'.
* Job 'bsicons-replacer' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'bsicons-replacer-28945867-wqzkv'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-13T07:07:05Z. Finish timestamp 2025-01-13T11:38:28Z. Exit code was '137'. With reason 'Error'.
* Job 'purge-cat-redir' (normal) (emails: onfailure) had 1 events:
  -- Pod 'purge-cat-redir-jgq6t'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-12T23:24:31Z. Finish timestamp 2025-01-13T11:38:28Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28946327-jlg6n'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-13T14:47:06Z. Finish timestamp 2025-01-13T14:59:59Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28946447-sc5sz'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-13T16:47:13Z. Finish timestamp 2025-01-13T17:39:28Z. Exit code was '137'. With reason 'Error'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28946437-bg4f8'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-13T16:37:23Z. Finish timestamp 2025-01-13T17:39:44Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28947797-ngcf8'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-14T15:17:20Z. Finish timestamp 2025-01-14T15:36:16Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28949957-bhcjv'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-16T03:17:11Z. Finish timestamp 2025-01-16T03:33:17Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28951217-cxt47'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'purge-dup-args' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-dup-args-28951293-hsthf'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'purge-dup-args' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-dup-args-28952673-r92dm'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-18T00:33:14Z. Finish timestamp 2025-01-18T00:56:01Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28952897-mq529'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-18T04:17:07Z. Finish timestamp 2025-01-18T04:27:31Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28953197-g7qlh'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-18T09:17:05Z. Finish timestamp 2025-01-18T09:32:36Z. Exit code was '137'. With reason 'Error'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28953497-5lv26'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-18T14:17:21Z. Finish timestamp 2025-01-18T14:29:38Z. Exit code was '137'. With reason 'Error
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28954357-4c4rr'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'purge-daily' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-daily-28955580-dg494'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'recent-vios' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'recent-vios-28955557-4gwzn'. Phase: 'failed'. Container state: 'terminated'. Exit code was '137'. With reason 'ContainerStatusUnknown'. With message: 'The container could not be located when the pod was terminated'.
* Job 'cfdw' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'cfdw-28955837-tjhvp'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-20T05:21:34Z. Finish timestamp 2025-01-20T05:29:58Z. Exit code was '137'. With reason 'Error'.
* Job 'purge-weekly' (cronjob) (emails: onfailure) had 1 events:
  -- Pod 'purge-weekly-28955607-4tr8h'. Phase: 'failed'. Container state: 'terminated'. Start timestamp 2025-01-20T01:27:12Z. Finish timestamp 2025-01-20T17:15:58Z. Exit code was '137'. With reason 'Error'.

Related Objects
Search...

Event Timeline

JJMC89 created this task.Jan 2 2025, 12:24 AM
Restricted Application added a subscriber: • Aklapper. · View Herald TranscriptJan 2 2025, 12:24 AM
JJMC89 renamed this task from Toolforge joibs: increased exit code 137 rate since 2024-12-14 to Toolforge jobs: increased exit code 137 rate since 2024-12-14.Jan 2 2025, 12:25 AM
JJMC89 updated the task description. (Show Details)Jan 4 2025, 12:15 AM
JJMC89 updated the task description. (Show Details)Jan 6 2025, 12:14 AM
dcaro subscribed.Jan 6 2025, 9:47 AM

I see no logs on the 14th (https://sal.toolforge.org/tools?d=2024-12-14), the whole team was on an offsite during that week, so there were no big interventions.

Can you update if it's still happening?

JJMC89 added a comment.Jan 6 2025, 4:01 PM

Yes, its still happening.

dcaro triaged this task as Medium priority.Jan 6 2025, 4:58 PM
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.
JJMC89 updated the task description. (Show Details)Jan 6 2025, 5:27 PM
JJMC89 updated the task description. (Show Details)Jan 9 2025, 4:44 PM
JJMC89 updated the task description. (Show Details)Jan 11 2025, 5:28 PM
JJMC89 updated the task description. (Show Details)Jan 11 2025, 9:06 PM
JJMC89 updated the task description. (Show Details)Jan 12 2025, 5:36 PM
JJMC89 updated the task description. (Show Details)Jan 12 2025, 6:56 PM
JJMC89 updated the task description. (Show Details)Jan 12 2025, 11:21 PM
JJMC89 updated the task description. (Show Details)Jan 13 2025, 4:19 PM
JJMC89 updated the task description. (Show Details)Jan 13 2025, 5:53 PM
JJMC89 updated the task description. (Show Details)Jan 14 2025, 3:56 PM
JJMC89 updated the task description. (Show Details)Jan 16 2025, 3:41 AM
JJMC89 updated the task description. (Show Details)Jan 17 2025, 12:29 AM
JJMC89 updated the task description. (Show Details)Jan 17 2025, 1:55 AM
JJMC89 updated the task description. (Show Details)Jan 18 2025, 1:36 AM
JJMC89 updated the task description. (Show Details)Jan 18 2025, 4:42 AM
JJMC89 updated the task description. (Show Details)Jan 18 2025, 9:44 AM
JJMC89 updated the task description. (Show Details)Jan 18 2025, 5:38 PM
JJMC89 updated the task description. (Show Details)Jan 19 2025, 4:46 AM
JJMC89 updated the task description. (Show Details)Jan 20 2025, 1:11 AM
JJMC89 updated the task description. (Show Details)
JJMC89 updated the task description. (Show Details)Jan 20 2025, 5:38 AM
dcaro added a comment.Jan 20 2025, 9:37 AM

I have not had time yet to take a look, but usually 137 might happen when the process is killed due to OOM (killed by the system), if you did not change anything it might just be that the workers are a bit more loaded than before and there's less space to go over the requests the pods made.

JJMC89 updated the task description. (Show Details)Jan 20 2025, 5:38 PM
JJMC89 added a comment.Jan 20 2025, 5:48 PM

If they're being OOM killed, it is not due to needing to exceed the requested resources. Also, I think the email will say that it was OOM killed when that happens.

purge-weekly-28955607-4tr8h, for example, was at 47.5/256 MiB leading up to the exit, which was also the average. (grafana)

dcaro added a comment.Jan 20 2025, 6:06 PM
In T382865#10477423, @JJMC89 wrote:

If they're being OOM killed, it is not due to needing to exceed the requested resources. Also, I think the email will say that it was OOM killed when that happens.

purge-weekly-28955607-4tr8h, for example, was at 47.5/256 MiB leading up to the exit, which was also the average. (grafana)

It seems it's a different quota that's being reached:

message: 'The node was low on resource: ephemeral-storage. Threshold quantity: 2096143596,
  available: 1962580Ki. Container job was using 592Ki, request is 0, has larger
  consumption of ephemeral-storage. '

I had not seen this before, for reference it was running on tools-k8s-worker-nfs-7 when it died.

Noting for myself (on my way out now) to open a task about the logs too, as toolforge jobs logs on it fails:

[18:05:02]tools.jjmc89-bot@tools-bastion-13:~$ toolforge jobs logs purge-weekly
ERROR: TjfCliError: Unknown error (Unknown string format: unable)
ERROR: Please report this issue to the Toolforge admins if it persists: https://w.wiki/6Zuu

So I had to use kubectl get pods -o yaml purge-weekly-28955607-4tr8h to see the messages.

dcaro renamed this task from Toolforge jobs: increased exit code 137 rate since 2024-12-14 to [jobs-api,jobs-cli] increased exit code 137 rate since 2024-12-14.Jan 21 2025, 9:10 AM
dcaro claimed this task.
dcaro edited projects, added: Toolforge (Toolforge iteration 17); removed: Toolforge.
dcaro closed subtask T384250: [infra,k8s] Workers are not rotating the /var/log/acctount/pacct log and it's growing as Resolved.Jan 21 2025, 5:09 PM
dcaro added a comment.Jan 21 2025, 5:12 PM

@JJMC89 I have cleanup some space on all the workers (also added alerts + automation to avoid it from getting there), can you update in a couple days if you are still getting this same issue? (you have been updating the task often - thanks!, just making sure that if you don't update it is because you are not getting the more errors, not because you got tired xd)

JJMC89 renamed this task from [jobs-api,jobs-cli] increased exit code 137 rate since 2024-12-14 to [jobs-api,jobs-cli] increased exit code 137 rate since 2024-12-14 due to low ephemeral-storage.Jan 23 2025, 3:29 AM
JJMC89 closed this task as Resolved.

I haven't seen any new 137 errors since I last updated the task.

dcaro moved this task from Next Up to Done on the Toolforge (Toolforge iteration 17) board.Jan 23 2025, 9:44 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits