Thanks @aborrero

Yeah there is a potential problem for cloud VMs talking to UDP (or other non-TCP) services on the internet. For instance if an internet server sends a 1500-byte packet to a VM with do-not-fragment set, and somewhere on the internet path there is an ICMP "blackhole" (preventing path mtu discovery working), the packet won't get back to the machine.

Most things are TCP, where it is not a problem as the VM will send the correct MSS value which will limit the size of packets the other side of the connection creates.

But overall it's safer/there are no edge cases if the VMs could do 1500.

Moving to jumbos

In terms of what would be needed to support that ideally we could just focus on the cloud-private networks, which would be easier as with few exceptions traffic on that network is all internal to the cluster (not talking to the internet or wikimedia production).

Unfortunately as the cloud-private is configured as a vlan sub-interface of the main network port this doesn't seem to be possible. The parent physical port MTU limits what can be set on the sub-interface. So we'd probably need to increase the MTU for all the networks the cloud hosts connect to, which is trickier again.

In theory we could make the 10.x prod-realm interface a vlan sub-int also, which would allow the parent and cloud sub-int to use 9000, and the prod realm sub-int 1500, but this would mess up our provisioning (needing tagged traffic) so let's not go there.