Sometime today, suddenly the paws k8s cluster had a total networking meltdown. Pods couldn't talk to each other or the outside internet. Hosts could talk to the pods that were running on them but nobody else. After a lot of digging, the following command, run on all hosts, worked: sudo iptables -P FORWARD ACCEPT. It came from https://github.com/kubernetes/kubernetes/issues/40182, but shouldn't have affected us since we have been running Docker 1.13 pretty much from the beginning on these hosts. Nothing seems to have triggered this 'stop working' phase, and rebooting didn't fix it either. Now after running this command, rebooting doesn't seem to make a difference - node still works fine. The appropriate sysctl has always been on.

No idea what happened!