A good chunk of tools' current workload is PAWS pods. I'd like to get PAWS cluster to about 10 nodes, so we've plenty of room to maneuver :)
Description
Description
Event Timeline
Comment Actions
To be clear, I don't actually care too much about the size of Tools' k8s cluster :) I only want about 8-10 nodes for the PAWS cluster :) Am happy to let toolforge admins decide if they wanna down size tools' cluster
Comment Actions
It's at 5 nodes now, and I've run out of quota in tools now.
@bd808 @chasemp what do you think of me depooling & deleting 5 k8s tools nodes and creating 5 paws ones? That would mean that the tools k8s cluster would've totally lost 5 nodes, and the PAWS cluster would be at 10 nodes. That should be able to stably sustain everything for a while!
Another option is to just increase tools quota. Either is fine by me!
Comment Actions
Added 40G of ram quota:
$ nova quota-show --tenant tools +-----------------------------+---------+ | Quota | Limit | +-----------------------------+---------+ | instances | 150 | | cores | 596 | | ram | 1128964 | | floating_ips | 53 | | fixed_ips | 200 | | metadata_items | 128 | | injected_files | 5 | | injected_file_content_bytes | 10240 | | injected_file_path_bytes | 255 | | key_pairs | 100 | | security_groups | 50 | | security_group_rules | 20 | | server_groups | 10 | | server_group_members | 10 | +-----------------------------+---------+ $ nova quota-update tools --ram 1169924 $ nova quota-show --tenant tools +-----------------------------+---------+ | Quota | Limit | +-----------------------------+---------+ | instances | 150 | | cores | 596 | | ram | 1169924 | | floating_ips | 53 | | fixed_ips | 200 | | metadata_items | 128 | | injected_files | 5 | | injected_file_content_bytes | 10240 | | injected_file_path_bytes | 255 | | key_pairs | 100 | | security_groups | 50 | | security_group_rules | 20 | | server_groups | 10 | | server_group_members | 10 | +-----------------------------+---------+ $ nova-manage project quota --project tools No handlers could be found for logger "oslo_config.cfg" Quota Limit In Use Reserved metadata_items 128 0 0 injected_file_content_bytes 10240 0 0 server_group_members 10 0 0 server_groups 10 0 0 ram 1169924 1122308 0 floating_ips 53 53 0 security_group_rules 20 0 0 instances 150 142 0 key_pairs 100 0 0 injected_files 5 0 0 cores 596 548 0 fixed_ips 200 147 0 injected_file_path_bytes 255 0 0 security_groups 50 16 0
Comment Actions
tvym, @bd808!
Since labs in general hates it when I try to create new instances, it took me almost 20 tries with lots of instances failing, but I now have ten nodes! Hopefully I'll not have to touch instance creation again for a long time. It's quite frustrating... :(