The ToolforgeKubernetesCapacity alert has been flapping for the lastl few days, and kubectl sudo top node is indeed showing that most nodes are using most of their allocated RAM.
It's probably time to add more nodes to both the NFS and non-NFS worker pools.
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-26T07:44:53Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster (T384790)
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-26T07:56:50Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster (T384790)
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-26T08:06:45Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T384790)
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-26T08:16:44Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T384790)
Mentioned in SAL (#wikimedia-cloud-feed) [2025-01-26T08:27:38Z] <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster (T384790)