This is annoying. Raising priority.

For some reason, the metrics-server doesn't work on the tools project cluster.

I think this is what we are looking for in this ticket:

My first impulse was to look at https://github.com/kubernetes/kube-state-metrics instead, but I'm not sure at this point if they provide the same or are competing options, or what.

@Bstorm please give this a final review.

Waiting for the folks in charge of the repo to rename the directory before enabling the mirror.

I just gave this another tested: 3 pods with 20k ingress objects. It works pretty well. I'm used that as the replica factor for now.

First step I think would be to have a proper routing_source_ip address in codfw1dev.

@Andrew is the clinic-duty person this week. Assigning this task to him.

Saw the patch, thanks!

I think the upgraded k8s cluster in toolbeta has been up and running stable for some time now. Resolving this task with the hope we can better focus on the several subtasks we have previous to the final operations in the tools project.

Triaging this as low priority, since this would be required for k8s 1.16. We are targeting 1.15 for the initial migration.

This should be done now. Please reopen if required.

In T237749#5715108, @Andrew wrote:

codfw1-dev is running Ocata, and I've scheduled an upgrade window for eqiad1. Approximate steps are...

Doing apt-get dist-upgrade is technically possible, but we don't encourage it becuase we would lost track of what the base operating system is. This is something that could be improved in our side but that won't happen in the short term I'm afraid.

I deleted all the VMs and the project itself. Thanks!

BTW, the reload time with 20k ingress objects with 2 pods is about ~90s:

good news, donwscaling the nginx-ingress to 2 pod replicas and creating again 20k ingress objects and the issue above didn't show up again. Our magic number could be 2 perhaps.

Oh, some more news. I scaled the deployment to 5 pods and created 20k ingress objects and the whole thing crashed.
I was not able to curl the test tool we have and I was unable to fetch ingress-nginx logs. Pods crashed and were re-created by k8s.

Regarding the number of nginx-ingress pods, I have some comments. We may not need any autoscaling mechanism. Let's assume we use the magic static number of 5 nginx-ingress pods.

• If we have very little usage on the cluster, this is no problem. We are not paying anything per use. We can afford having 5 pods doing nothing. No need to downscale. The magic number works.
• If we have suddenly have very high load in the cluster. This is indicative of other kind of problems. We may need to manually scale the cluster itself (workers), not just the number of nginx-ingress pods. Adjusting the magic number is just a very minor step in that scaling workflow. So the magic number works in this case too.
• We have currently 30 worker nodes in the legacy cluster. A magic number of 5 nginx-ingress pods mean 1 pod per 6 worker nodes. I think nginx can handle this.

With an actual destination svc/pod, we get similar reload times. I can confirm there is no downtime for current traffic while the reload is taking place, ie:

Some initial simple tests, with a single nginx-ingress pod, a fake ingress object that points to nowhere (non existent svc or endpoint):

• with 100 ingress objects, config reload takes about ~0.5s
• with 1k ingress objects, config reload takes about ~2s
• with 5k ingress objects, config reload takes about ~10s
• with 10k ingress objects, config reload takes about ~20s

Joined the 3 new VMs into the cluster and updated the docs https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Deploying_k8s to mention what to do in case kubeadm token is expired.

We might lack some of the things described in the task description, but for the record, I'm sharing here some grafana dashboard based on prometheus metrics we have for NFS:

I declare this is mostly done, at least until we start to have real traffic in the service and see where we lack metrics.

Removing the cloud-services-team (Kanban) tag since I don't think we the SREs in the WMCS team have a lot of action items here.

I don't think the diff is such a big deal. Closing task now.

We are working on a new kubernetes cluster for Toolforge. This cluster has better user quota management, see T234702: Review and establish configurable quotas for users in the new Kubernetes cluster for reference.
Closing this task now.

We are working on a new kubernetes cluster for Toolforge. This cluster has better user quota management, see T234702: Review and establish configurable quotas for users in the new Kubernetes cluster for reference.
Closing this task now.

Created a couple of grafana dashboards:

• this one is for haproxy in front of the apiserver and nginx-ingress:

https://grafana-labs.wikimedia.org/d/5O3YKfbWz/toolforge-k8s-haproxy

• this one aggregates metrics for all the ingress path:

https://grafana-labs.wikimedia.org/d/R7BPaEbWk/toolforge-ingress?refresh=1m&orgId=1

In the WMCS team meeting, we decided to rename these servers to better reflect what they do and to avoid naming clashes:

• from labmon1001 to cloudmetrics1001 and
• from labmon1002 to cloudmetrics1002

This seems solved! Please reopen if required.

This seems to be solved by using puppet cert SANs to include all the other servers. This can be done easily via hiera.

This seems solved. Thanks @bd808 and @Bstorm !! Closing task now, please reopen if required.