Due to puppet ENC failure at one point, all tools VMs ended up with NFS mounted. That wouldn't seem like a huge deal if not for the set of tc rules that come with them (see modules/labstore/manifests/traffic_shaping.pp and the related script modules/labstore/templates/tc-setup.sh.erb). These tc settings dramatically limit network on VMs, making them run well below what one would expect for a server machine that serves scores of other VMs simultaneously (about 240Mbps maximum possible egress for all processes combined) see also T218338. It also exposes the VMs to problems with load and IO issues due to NFS connectivity quirks, which is really not good either.
The only way to be 100% sure all NFS-focused settings are removed is to rebuild the VM (though you can manually increase tc limits and edit /etc/fstab by hand, things are still left behind). Any vms that have the mount_nfs: false hiera key applied should not have NFS now.
Each cluster requires a different method for replication and failover to ensure that we don't lose state. The puppetmaster is standalone and the commits in /var/lib/git/labs/private need to be copied to a new one before cutover or important state will be lost.
VMs currently in that category include:
- tools-k8s-etcd-*
- Should be created with an anti-affinity server group
- Has a special image from T267078 g2.cores1.ram2.disk20.4xiops
- tools-k8s-haproxy-*
- Additional ask here: apply a strong (not soft) anti-affinity server group to these if there isn't one already and the VIP system used in PAWS (automatic failover) if it isn't already there. see: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Keepalived
- tools-redis-*
- Beware, has a server group
- tools-elastic-*
- Beware, has a server group
- tools-puppetmaster-02.tools.eqiad1.wikimedia.cloud
- tools-k8s-ingress-1 and tools-k8s-ingress-2
- Beware, has a server group
- tools-prometheus-*
- Should be created with an anti-affinity server group. It's possible something like https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Keepalived could also be applied here, but not sure.
- tools-proxy-*
- Should be created with an anti-affinity server group
When any of these are rebuilt entirely with anti-affinity server groups, they can be removed from the spread alarm in puppet.
It may be a good idea to try rebuilding the tools-k8s-control-* servers without NFS as well. Those currently have NFS mounted, and do not yet have the mount_nfs: false setting.
No review has been done regarding how the settings in the tc-setup script may affect the management of the iptables mangle table and similar by calico and kube-proxy, which is really not good either. Unfortunately, that's where we are.