While investigating https://phabricator.wikimedia.org/T115231#2483454,
redis-cli 127.0.0.1:6379> hgetall prefix:dplbot (empty list or set) 127.0.0.1:6379>
yuvipanda@tools-proxy-01:~$ sudo service kube2proxy restart yuvipanda@tools-proxy-01:~$ redis-cli 127.0.0.1:6379> hgetall prefix:dplbot 1) ".*" 2) "http://192.168.0.8:8000" 127.0.0.1:6379>
So kube2proxy either missed a change or deleted it accidentally.
Long term we should move away from kube2proxy to a different ingress solution.