Page MenuHomePhabricator

Setup Kubernetes Masters in a HA setup
Closed, ResolvedPublic

Description

Since we have instances randomly freezing, and it could happen to the kubernetes master too, let's make sure it's got a HA setup going.

Need to follow http://kubernetes.io/docs/admin/high-availability/#replicated-api-servers

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a project: Cloud-Services. · View Herald TranscriptAug 12 2016, 8:15 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Change 304503 had a related patch set uploaded (by Yuvipanda):
k8s: Make controller-manager & scheduler be HA

https://gerrit.wikimedia.org/r/304503

Change 304504 had a related patch set uploaded (by Yuvipanda):
tools: Allow multiple k8s master to access etcd

https://gerrit.wikimedia.org/r/304504

This ran into a bump - we have kube-maintainusers, which is used to populate token auth of all the masters. This should run in only one place, however, and push updates to all the places.

To do this, I am going to do the following:

  1. Move maintain-kubeusers to a centralized location (puppetmaster maybe?)
  2. Setup some way for it to push config to all the masters and restart them only when it's sure it has propogated everywhere.

Change 304503 merged by Yuvipanda:
k8s: Make controller-manager & scheduler be HA

https://gerrit.wikimedia.org/r/304503

Change 304504 merged by Yuvipanda:
tools: Allow multiple k8s master to access etcd

https://gerrit.wikimedia.org/r/304504

scfc triaged this task as Medium priority.Feb 16 2017, 8:14 PM
scfc moved this task from Triage to Backlog on the Toolforge board.
yuvipanda removed yuvipanda as the assignee of this task.Mar 22 2017, 10:31 PM
aborrero closed this task as Resolved.Jul 4 2019, 4:42 PM
aborrero claimed this task.
aborrero added a subscriber: aborrero.

We know how to do this now.

In T215531: Deploy upgraded Kubernetes to toolsbeta we are developing a new k8s cluster which is deployed by using kubeadm. This new mechanism takes care of building the multi master setup for us.

The next version of the toolforge k8s service should contain a fix for this.

Closing task now. Feel free to reopen if required.