I am not so sure we actually do scratch that memory limit now. Looking at kubemaster2001 last week

and the other 3 kubernetes masters

So, at the very least we got ~50% of the VMs memory capacity before hitting problems again.

Upper memory that we can handle nicely in Ganeti is by experience ~16GB btw. But this is mostly because most applications after than number tend to both consume a lot of memory AND alter the contents of the memory faster than the migration algorithm can catch up with ending with either stuck or at least very long running migrations.

CPU usage also fell to ~20% now so we got some room to spare and think how and when we want to tackle this.

In T353464#9412761, @akosiaris wrote:

I am not so sure we actually do scratch that memory limit now. Looking at kubemaster2001 last week

I wasn't saying we do. There are still quite a number of nodes to come, even with those I won't suspect us hitting 16 or 12GB. But with the IOPS bottlenecks we saw with etcd on ganeti we probably need to move etcd servers to hardware and in that case it does not make sense to not move control-planes as well IMHO.

Forgive me for the drive-by comment, but would it be possible to create high IOPS tiers for Ganeti (RAID-0?) I'd recommend deploying in conjunction with non-DRDB VMs for services that have their own HA (such as Kubernetes control plane). I bring it up as I feel like Ganeti is an underused resource, and using it helps to avoid some of the management overhead associated with physical machines.

In T353464#9413167, @bking wrote:

Forgive me for the drive-by comment, but would it be possible to create high IOPS tiers for Ganeti (RAID-0?) I'd recommend deploying in conjunction with non-DRDB VMs for services that have their own HA (such as Kubernetes control plane). I bring it up as I feel like Ganeti is an underused resource, and using it helps to avoid some of the management overhead associated with physical machines.

Yes, probably. But there would still be overhead and potential noisy neighbors on ganeti. With etcd being very sensitive in terms of IOPS this might still not give us the desired performance. Also we would need to build such a ganeti system (multiple to cover our redundancy needs).

In T353464#9412778, @JMeybohm wrote:

In T353464#9412761, @akosiaris wrote:

I am not so sure we actually do scratch that memory limit now. Looking at kubemaster2001 last week

I wasn't saying we do. There are still quite a number of nodes to come, even with those I won't suspect us hitting 16 or 12GB.

Agreed. That's my current theory as well.

But with the IOPS bottlenecks we saw with etcd on ganeti we probably need to move etcd servers to hardware and in that case it does not make sense to not move control-planes as well IMHO.

Oh, so collocate etcd with the rest of the control-plane. It will work ofc, and may even make our puppetization simpler. Just not super sold on it yet.

Do we track the IOPS bottlenecks we witnessed in some task?

Do we track the IOPS bottlenecks we witnessed in some task?

I'm also curious about the IOPS issues, since I assume the majority of etcd instances out in the wild are running on VMs and shared hardware. Might not worth be the effort for this particular project, but I'm game to help build a higher IOPS tier for Ganeti if anyone else thinks that would be helpful.

In T353464#9415585, @akosiaris wrote:

Do we track the IOPS bottlenecks we witnessed in some task?

Track no, but what triggered creating T348466: Rethink kubernetes etcd storage was investigating T348228: KubernetesAPILatency alert fires on scap deploy.

In T353464#9431952, @Clement_Goubert wrote:

In T353464#9415585, @akosiaris wrote:

Do we track the IOPS bottlenecks we witnessed in some task?

Track no, but what triggered creating T348466: Rethink kubernetes etcd storage was investigating T348228: KubernetesAPILatency alert fires on scap deploy.

Thanks, that's the context I was missing.

I ran a couple of very basic benchmarks (commands in the attached filed) against single node etcd instances running on:

• A mediawiki application server, ex4 on LVM on RAID1

• A (empty) ganeti node, ext4 on LVM on RAID5

• A ganeti instance running as the only instance on the above node, ext4 on LVM on RAID5

• A ganeti instance running in the prod ganeti cluster (together with other instances), ext4 on LVM on RAID5

tl;dr is:
Puts with 3 clients, 100 connections (roughly k8s prod) show Requests/sec: 3963.8976 vs. 6750.9583 vs. 11895.5644 (isolated vm, ganeti node, appserver) and p99.9 latency: 0.1026s vs. 0.0599s vs. 0.0399s

Change #1032805 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] appservers: 6 appservers to insetup before reimaging

https://gerrit.wikimedia.org/r/1032805