Change Details

Due to puppet ENC failure at one point, all tools VMs ended up with NFS mounted. That wouldn't seem like a huge deal if not for the set of `tc` rules that come with them (see `modules/labstore/manifests/traffic_shaping.pp` and the related script `modules/labstore/templates/tc-setup.sh.erb`). These `tc` settings dramatically limit network on VMs, making them run well below what one would expect for a server machine that serves scores of other VMs simultaneously (about 240Mbps maximum possible egress for all processes combined) see also T218338. It also exposes the VMs to problems with load and IO issues due to NFS connectivity quirks, which is really not good either. The only way to be 100% sure all NFS-focused settings are removed is to rebuild the VM (though you can manually increase `tc` limits and edit /etc/fstab by hand, things are still left behind). Any vms that have the `mount_nfs: false` hiera key applied should not have NFS now. Each cluster requires a different method for replication and failover to ensure that we don't lose state. The puppetmaster is standalone and the commits in /var/lib/git/labs/private need to be copied to a new one before cutover or important state **will** be lost. VMs currently in that category include: [] tools-k8s-etcd-* [] tools-k8s-haproxy-* [] tools-redis-* [] tools-elastic-* [] tools-puppetmaster-02.tools.eqiad1.wikimedia.cloud [] tools-k8s-ingress-1 and tools-k8s-ingress-2 [] tools-prometheus-* [] tools-proxy-* It may be a good idea to try rebuilding the tools-k8s-control-* servers without NFS as well. Those currently have NFS mounted, and do not yet have the `mount_nfs: false` setting. No review has been done regarding how the settings in the tc-setup script may affect the management of the iptables mangle table and similar by calico and kube-proxy, which is really not good either. Unfortunately, that's where we are.

Due to puppet ENC failure at one point, all tools VMs ended up with NFS mounted. That wouldn't seem like a huge deal if not for the set of `tc` rules that come with them (see `modules/labstore/manifests/traffic_shaping.pp` and the related script `modules/labstore/templates/tc-setup.sh.erb`). These `tc` settings dramatically limit network on VMs, making them run well below what one would expect for a server machine that serves scores of other VMs simultaneously (about 240Mbps maximum possible egress for all processes combined) see also T218338. It also exposes the VMs to problems with load and IO issues due to NFS connectivity quirks, which is really not good either. The only way to be 100% sure all NFS-focused settings are removed is to rebuild the VM (though you can manually increase `tc` limits and edit /etc/fstab by hand, things are still left behind). Any vms that have the `mount_nfs: false` hiera key applied should not have NFS now. Each cluster requires a different method for replication and failover to ensure that we don't lose state. The puppetmaster is standalone and the commits in /var/lib/git/labs/private need to be copied to a new one before cutover or important state **will** be lost. VMs currently in that category include: [] tools-k8s-etcd-* [] tools-k8s-haproxy-* - Additional ask here: apply a strong (not soft) anti-affinity server group to these if there isn't one already and the VIP system used in PAWS (automatic failover) if it isn't already there. [] tools-redis-* - Beware, has a server group [] tools-elastic-* - Beware, has a server group [] tools-puppetmaster-02.tools.eqiad1.wikimedia.cloud [] tools-k8s-ingress-1 and tools-k8s-ingress-2 - Beware, has a server group [] tools-prometheus-* - Should be created with an anti-affinity server group [] tools-proxy-* - Should be created with an anti-affinity server group When any of these are rebuilt entirely with anti-affinity server groups, they can be removed from the spread alarm in puppet. It may be a good idea to try rebuilding the tools-k8s-control-* servers without NFS as well. Those currently have NFS mounted, and do not yet have the `mount_nfs: false` setting. No review has been done regarding how the settings in the tc-setup script may affect the management of the iptables mangle table and similar by calico and kube-proxy, which is really not good either. Unfortunately, that's where we are.

Due to puppet ENC failure at one point, all tools VMs ended up with NFS mounted. That wouldn't seem like a huge deal if not for the set of `tc` rules that come with them (see `modules/labstore/manifests/traffic_shaping.pp` and the related script `modules/labstore/templates/tc-setup.sh.erb`). These `tc` settings dramatically limit network on VMs, making them run well below what one would expect for a server machine that serves scores of other VMs simultaneously (about 240Mbps maximum possible egress for all processes combined) see also T218338. It also exposes the VMs to problems with load and IO issues due to NFS connectivity quirks, which is really not good either. The only way to be 100% sure all NFS-focused settings are removed is to rebuild the VM (though you can manually increase `tc` limits and edit /etc/fstab by hand, things are still left behind). Any vms that have the `mount_nfs: false` hiera key applied should not have NFS now. Each cluster requires a different method for replication and failover to ensure that we don't lose state. The puppetmaster is standalone and the commits in /var/lib/git/labs/private need to be copied to a new one before cutover or important state **will** be lost. VMs currently in that category include: [] tools-k8s-etcd-* [] tools-k8s-haproxy-* - Additional ask here: apply a strong (not soft) anti-affinity server group to these if there isn't one already and the VIP system used in PAWS (automatic failover) if it isn't already there. [] tools-redis-* - Beware, has a server group [] tools-elastic-* - Beware, has a server group [] tools-puppetmaster-02.tools.eqiad1.wikimedia.cloud [] tools-k8s-ingress-1 and tools-k8s-ingress-2 - Beware, has a server group [] tools-prometheus-* - Should be created with an anti-affinity server group [] tools-proxy-* - Should be created with an anti-affinity server group When any of these are rebuilt entirely with anti-affinity server groups, they can be removed from the spread alarm in puppet. It may be a good idea to try rebuilding the tools-k8s-control-* servers without NFS as well. Those currently have NFS mounted, and do not yet have the `mount_nfs: false` setting. No review has been done regarding how the settings in the tc-setup script may affect the management of the iptables mangle table and similar by calico and kube-proxy, which is really not good either. Unfortunately, that's where we are.