IPv6-enabled VMs in Cloud VPS have mtu=1450 on their network interface. This is a problem if you run Docker in those VMs, because Docker assumes the MTU to be 1500. This can lead to network errors like seen in T405742: tofu-provisioning: Failed to install provider.
We should either:
- find a way to raise the MTU to 1500 on all VMs
- make sure that Docker uses the right MTU setting on all VMs
In gitlab-runners VMs this was fixed by modifying /etc/docker/daemon.json, see patches https://gerrit.wikimedia.org/r/c/operations/puppet/+/1196493 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1196929.
In other VMs, this is still an issue, for example I verified I can reproduce the network errors in tools-harbor-2
root@tools-harbor-2:~# docker run --rm -it docker-registry.wikimedia.org/bookworm [...] root@ca1585e47d54:/# TESTURL='https://github.com/terraform-provider-openstack/terraform-provider-openstack/releases/download/v3.3.2/terraform-provider-openstack_3.3.2_SHA256SUMS' root@ca1585e47d54:/# for i in {1..50}; do curl $TESTURL -s -o /dev/null -L --connect-timeout 1 && echo -n '.' || echo -n 'F'; done; echo '' ...F.F..FF..F..FF..F.F....FF.FFFF..FFFF..F.F....F.
Running apt update and apt install to install curl also failed randomly a few times before I could execute the test above. I assume that was also caused by the MTU mismatch.