At 2020-02-10T21:27Z the 2020 Kubernetes cluster contains 21 worker nodes with 222 active pods across 128 namespaces. These pods are consuming 29% of available RAM and 8% of CPU. Another 646 Pods are still running on the legacy Kubernetes cluster. All of these are expected to be migrated to the 2020 cluster with a final load of approximately 4x the current load.
It would be nice to grow the 2020 cluster to at least 35 worker nodes (+14 nodes) before attempting to place another 600+ Pods on it.
The legacy cluster is both heavily oversubscribed and weakly instrumented for determining active load. This makes it difficult to determine how many worker nodes from the legacy cluster we can reclaim without impacting the current workloads running there.
We do seem to have some extra capacity in the grid engine's webservice cluster which we can reclaim and repurpose for 2020 Kubernetes workers. After doing that we can evaluate how many net new resources we need to finish this round of 2020 Kubernetes build-out.
