I noticed this increase of ICMP dest unreachable on the eqiad:spare cluster:
https://grafana.wikimedia.org/d/000000366/network-performances-global?panelId=20&fullscreen&orgId=1&from=1561686984227&to=1562039140502
Drilling down on it, it seems like this change caused the issue:
https://github.com/wikimedia/puppet/commit/331ded70742ebc36e5c08ec1b129da72d367ff6b#diff-99ec8ea2d6f28c30f9a0d57bb66577e6
A tcpdump shows a lot of loopback traffic:
restbase1007:~$ sudo tcpdump -p icmp -i lo 03:48:38.275443 IP localhost > localhost: ICMP localhost udp port 10514 unreachable, length 399 ayounsi@restbase1007:~$ sudo tcpdump -p -i lo port 10514 03:52:22.255194 IP localhost.40195 > localhost.10514: UDP, length 328
Quick guess to fix it is either stop the process sending the syslog, or make sure the process that receives it doesn't get removed when the serve role is changed to spare.