+1

@dcaro do we have a way to automatically handle requests like this?

In T357209#9536704, @Raymond_Ndibe wrote:

@dcaro do we have a way to automatically handle requests like this?

Not yet no, feel free to try to create a cookbook :), though it's managed through commits to the maintain-kubeusers repo:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Quota_management

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/197

maintain-kubeusers: increase quota for wd-shex-infer

The limit also isn’t working properly yet; from kubectl get events:

15s         Warning   FailedCreate        job/wd-shex-infer-101               Error creating: pods "wd-shex-infer-101-b25wv" is forbidden: maximum memory usage per Container is 6Gi, but limit is 8G

(I think I’ll just kubectl edit this job to unstuck it and test that the rest of T320140 works, but it would be nice to have this working in general.)

Meh, doesn’t work, Kubernetes complained that the memory limit is immutable (if I understood the error message correctly).

I guess I also need the limitrange increased? At least I can see a 6Gi max there.

tools.wd-shex-infer@tools-sgebastion-10:~/www/python/src$ kubectl describe limitrange
Name:       tool-wd-shex-infer
Namespace:  tool-wd-shex-infer
Type        Resource  Min    Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---    ---  ---------------  -------------  -----------------------
Container   cpu       50m    3    250m             500m           -
Container   memory    100Mi  6Gi  256Mi            512Mi          -

(I’m leaving the job alive for now, by the way, and hope that it can successfully run once the limitrange has been increased.)

In T357209#9550918, @dcaro wrote:

Done:

root@tools-k8s-control-6:~# kubectl -n tool-wd-shex-infer get resourcequotas tool-wd-shex-infer -o json | jq '.spec.hard."limits.memory"'
"10Gi"

https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/197 was never merged. How did this happen?

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/197

maintain-kubeusers: increase quota for wd-shex-infer

Just merged it (forgot to), deployed it from the branch as usual,.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/198

wd-shex-infer: update also the limitrange

In T357209#9553241, @LucasWerkmeister wrote:

I guess I also need the limitrange increased? At least I can see a 6Gi max there.

tools.wd-shex-infer@tools-sgebastion-10:~/www/python/src$ kubectl describe limitrange
Name:       tool-wd-shex-infer
Namespace:  tool-wd-shex-infer
Type        Resource  Min    Max  Default Request  Default Limit  Max Limit/Request Ratio
----        --------  ---    ---  ---------------  -------------  -----------------------
Container   cpu       50m    3    250m             500m           -
Container   memory    100Mi  6Gi  256Mi            512Mi          -

Yep, sorry about that, we don't usually increase the limit range (though maybe we should :/, feels weird limiting).

For the request.memory value, we currently set it to half the memory, I'll have to change our quota managment scripts to allow setting it to something different.

dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/198

wd-shex-infer: update also the limitrange

Updated the limitrange:

root@tools-k8s-control-6:~# kubectl -n tool-wd-shex-infer get limitrange tool-wd-shex-infer -o json | jq '.spec.limits[].max.memory'
"10Gi"

For the requests, you can work-around the default by passing the --mem/--cpu when creating the jobs.

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/12

quota: allow overriding the requests.cpu and memory

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/199

maintain-kubeusers: bump to 0.0.121-20240219092902-759465a7

dcaro closed https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/199

maintain-kubeusers: bump to 0.0.121-20240219092902-759465a7

Thanks, the updated limitrange seems to be working!

@LucasWerkmeister can this task be resolved, or is there anything missing?

Mentioned in SAL (#wikimedia-cloud) [2024-03-02T12:06:32Z] <wmbot~lucaswerkmeister@tools-sgebastion-10> update config.yaml: increase job limits.memory from 6G to 8G, should be possible now (T357209)

It’s mostly working for me, but I’d still like to be able to set requests.memory higher than at the moment (which is blocked on T357881 if I understand correctly).

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/218

maintain-kubeusers: Bump memory for wd-shex-infer

taavi closed https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/12

quota: allow overriding the requests.cpu and memory

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/218

maintain-kubeusers: Bump memory for wd-shex-infer

Mentioned in SAL (#wikimedia-cloud) [2024-03-08T15:15:54Z] <wmbot~lucaswerkmeister@tools-sgebastion-10> bump requests.memory to 8G (T357209 / T320140)