Page MenuHomePhabricator

Scale up 2020 Kubernetes cluster for final migration of legacy cluster workloads
Closed, ResolvedPublic

Description

At 2020-02-10T21:27Z the 2020 Kubernetes cluster contains 21 worker nodes with 222 active pods across 128 namespaces. These pods are consuming 29% of available RAM and 8% of CPU. Another 646 Pods are still running on the legacy Kubernetes cluster. All of these are expected to be migrated to the 2020 cluster with a final load of approximately 4x the current load.

It would be nice to grow the 2020 cluster to at least 35 worker nodes (+14 nodes) before attempting to place another 600+ Pods on it.

The legacy cluster is both heavily oversubscribed and weakly instrumented for determining active load. This makes it difficult to determine how many worker nodes from the legacy cluster we can reclaim without impacting the current workloads running there.

We do seem to have some extra capacity in the grid engine's webservice cluster which we can reclaim and repurpose for 2020 Kubernetes workers. After doing that we can evaluate how many net new resources we need to finish this round of 2020 Kubernetes build-out.

Event Timeline

bd808 created this task.Feb 10 2020, 9:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 10 2020, 9:45 PM

Mentioned in SAL (#wikimedia-cloud) [2020-02-10T21:47:13Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0928 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-10T22:01:16Z] <bd808> Manually starting webservices for tools that were running on tools-sgewebgrid-lighttpd-0928 (T244791)

Bstorm triaged this task as High priority.Feb 11 2020, 4:08 PM

Mentioned in SAL (#wikimedia-cloud) [2020-02-11T23:38:04Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0927 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-11T23:56:30Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0926 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-11T23:58:26Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0925 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-12T00:05:24Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0924 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-12T00:05:43Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0923 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-12T00:09:14Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0922 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-12T00:14:04Z] <bd808> Depooling tools-sgewebgrid-lighttpd-0921 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-12T00:19:41Z] <bd808> Depooling tools-sgewebgrid-generic-0904 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-12T00:20:47Z] <bd808> Depooling tools-sgewebgrid-generic-0903 (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-13T21:33:13Z] <bd808> Removed tools-sgewebgrid-lighttpd-092{1,2,3,4,5,6,7,8} & tools-sgewebgrid-generic-090{3,4} from grid engine config (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-13T21:35:09Z] <bd808> Deleted tools-sgewebgrid-lighttpd-092{1,2,3,4,5,6,7,8} & tools-sgewebgrid-generic-090{3,4} (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-13T23:38:13Z] <bd808> Added tools-k8s-worker-22 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-13T23:50:13Z] <bd808> Added tools-k8s-worker-23 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-13T23:53:38Z] <bd808> Added tools-k8s-worker-24 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:03:28Z] <bd808> Added tools-k8s-worker-25 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:07:17Z] <bd808> Added tools-k8s-worker-26 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:13:30Z] <bd808> Added tools-k8s-worker-27 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:15:16Z] <bd808> Added tools-k8s-worker-28 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:17:25Z] <bd808> Added tools-k8s-worker-29 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:25:36Z] <bd808> Added tools-k8s-worker-30 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:25:42Z] <bd808> Added tools-k8s-worker-31 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:29:38Z] <bd808> Added tools-k8s-worker-32 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:32:08Z] <bd808> Added tools-k8s-worker-33 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:34:00Z] <bd808> Added tools-k8s-worker-34 to 2020 Kubernetes cluster (T244791)

Mentioned in SAL (#wikimedia-cloud) [2020-02-14T00:38:13Z] <bd808> Added tools-k8s-worker-35 to 2020 Kubernetes cluster (T244791)