Currently toolforge workers have 16G of RAM each, this means that 16G is a hard upper limit. Said that, the scheduling will have a really hard time finding a 16G free slot in the fleet (as workers tend to be running something all the time), so I would recommend using 8G maximum for now and seeing if that's enough and can be scheduled easy enough.

We can look into adding bigger workers (will take some time to roll out), though it's an inherent problem of big pods to be hard to schedule, so I'd recommend as much as possible to lower the ram usage (using tempfiles for example).

So I propose:

• Increase the limit to 8G
• See if you can improve the RAM usage of that job (split it in more than one, use tempfiles, ...), feel free to ask for help on this, though I can't commit to anything right now

If that does not work, and you still need more RAM:

• Open a task to get bigger workers (I can do that)
• Eventually raise the quota to 16G (will still be hard to schedule though, but possible)

Does that work for you?

@dcaro Let's start from 8 or, better, 12 GB, if you could set 12. I don't want to cause you any unnecessary inconvenience and will try to stay within these limits.

After quote increase, should I use this in my jobs.yaml on certain job?
mem: 12Gi

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/728

maintain-harbor: increase memory quota for mbh

In T389733#10678994, @MBH wrote:

@dcaro Let's start from 8 or, better, 12 GB, if you could set 12. I don't want to cause you any unnecessary inconvenience and will try to stay within these limits.

After quote increase, should I use this in my jobs.yaml on certain job?
mem: 12Gi

That should do it yes, note that it might become really hard for your jobs to find a worker to run on, so keep an eye on that, I'm also guessing that you only want to run one job at that high mem quota, so I'm bumping your overall quota to 16 (sum of all jobs), but let me know if you want to run more than one (you'll need more overall quota).

Waiting for a review and then it will be applied :)

Also, request T359061 isn't relevant now, you could lower number of continuous jobs on mbh account to default.

We raised the defaults not very long ago, and they are higher now that what you requested there, so you are good to go there 👍, thanks for the note though

it might become really hard for your jobs to find a worker to run on

I know almost nothing about k8s, so I didn't think such a problem was possible (i don't even know what "worker" is). What happens if my process doesn't find the worker it needs? It will not run, or it will try to run in next hours?

In T389733#10684245, @MBH wrote:

it might become really hard for your jobs to find a worker to run on

I know almost nothing about k8s, so I didn't think such a problem was possible (i don't even know what "worker" is). What happens if my process doesn't find the worker it needs? It will not run, or it will try to run in next hours?

Sorry, no problem, let me elaborate :)

A worker is a virtual machine that runs the kubernetes agent of sorts, it's where you job will be running. That's why the maximum size of your job is limited to the size of one of those worker VMs.

The thing is that those worker VMs are usually already running something or other, so they will have only some RAM free at any given time, so when you try to run your job, it will search for a worker VM that has enough RAM free for your job, and run it there.
If you job requires a big amount of RAM, finding a worker that has that amount of RAM free becomes harder and harder (as the moments those worker VMs are idle are not that many), so it will hang waiting for the opportunity, sometimes for long periods (depending on the load of the cluster).

Fore example, for 12G of RAM job, you will find only 1 workers VM right now able to take that job (tools-k8s-worker-nfs-57):

The average free memory for a worker in the last 7 days was between 2-4G of RAM, and in the last 7 days (taking out today that we rebooted the nodes), there's been less than 1h total where any worker had >12G free (or even >8G free), most of the time the maximum free memory was between 4 and 6G.

Essentially, this means that if you would have wanted to run anything >6G in the last week, you would have been able to run it only 3 times in ~10min periods, and would have been waiting the rest of the time.

This is actually a very good rationale to increase the worker sizes, I'll open a task with this info :)

Created T390228: [toolforge] increase worker sizes in tools to increase worker sizes :)

In the meantime, once the quota is increased, you can try using a smaller limit if you see that your job is not getting scheduled.

I thought, limits works not like

you can run a job only when it's possible to allocate you 12 GB right now

but

you can run a job in any time, if your job will request only 2 GB in certain run, it will be successfully completed even if jobs.yaml requires 12 GB for this job, but when your job will ask to allocate 12 GB + 1 byte of RAM, it will be killed by process manager

I doesn't thought that situation "server can't allocate you 12 GB of RAM" is possible at all, I thought Toolforge is a server system with terabytes of RAM in common use.

If my job reqiures high RAM limit in jobs file, but actually uses less RAM, will it work? Or process manager can run my job only if it can allocate all 12 GB at start of job?

Another question: in last month many of my jobs disappeared from my processes (and wasn't successfully completed) without any output to .err files. Is it how process manager kills outlimited processes, or such killing will create a record in .err file?

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-28T09:30:30Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component maintain-harbor (T389733)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-28T09:42:41Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-harbor (T389733)

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-28T10:00:00Z] <fnegri@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component maintain-kubeusers (T389733)

fnegri merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/728

maintain-kubeusers: increase memory quota for mbh

Mentioned in SAL (#wikimedia-cloud-feed) [2025-03-28T10:10:57Z] <fnegri@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component maintain-kubeusers (T389733)

@dcaro is it possible to run one-off job with a high limit? I tried to add --mem:12Gi into toolforge jobs run request, but in doesn't work.

In T389733#10686326, @MBH wrote:

If my job reqiures high RAM limit in jobs file, but actually uses less RAM, will it work? Or process manager can run my job only if it can allocate all 12 GB at start of job?

If you say it requires high RAM, it will wait to have enough RAM free to start it yep.

Another question: in last month many of my jobs disappeared from my processes (and wasn't successfully completed) without any output to .err files. Is it how process manager kills outlimited processes, or such killing will create a record in .err file?

When the pods are killed by the system due to getting out of memory, scheduled and continuous jobs should not disappear, one-offs might (if you used the retry, they should respawn). So if you had one-off jobs, they might seem like they disappeared.
Note also that one-offs will get stopped also if we have to evacuate the worker VM they are running in (we have done many of those lately :/, had a couple outages), I would guess that is most probable, if you have a specific job I can try looking a bit more into it.
You can also enable emails (--emails option) to get notified if/when the job gets stopped.

@dcaro is it possible to run one-off job with a high limit? I tried to add --mem:12Gi into toolforge jobs run request, but in doesn't work.

Should be possible yes, what's the error you get? (if you can copy-paste the command you run and the output it would help debug)

In T389733#10686326, @MBH wrote:

If my job reqiures high RAM limit in jobs file, but actually uses less RAM, will it work? Or process manager can run my job only if it can allocate all 12 GB at start of job?

If you say it requires high RAM, it will wait to have enough RAM free to start it yep.

Another question: in last month many of my jobs disappeared from my processes (and wasn't successfully completed) without any output to .err files. Is it how process manager kills outlimited processes, or such killing will create a record in .err file?

When the pods are killed by the system due to getting out of memory, scheduled and continuous jobs should not disappear, one-offs might (if you used the retry, they should respawn). So if you had one-off jobs, they might seem like they disappeared.
Note also that one-offs will get stopped also if we have to evacuate the worker VM they are running in (we have done many of those lately :/, had a couple outages), I would guess that is most probable, if you have a specific job I can try looking a bit more into it.
You can also enable emails to get notified if/when the job gets stopped.

@dcaro is it possible to run one-off job with a high limit? I tried to add --mem:12Gi into toolforge jobs run request, but in doesn't work.

Should be possible, what's the error you get? (might take long enough to start that it times out)