It seems that pybal stops logging after a while. For example, at the time of this writing:
lvs1007.eqiad.wmnet: -rw-r--r-- 1 root root 0 Mar 8 06:25 /var/log/pybal.log lvs1008.eqiad.wmnet: -rw-r--r-- 1 root root 0 Mar 8 06:25 /var/log/pybal.log lvs1011.eqiad.wmnet: -rw-r--r-- 1 root root 0 Mar 8 06:25 /var/log/pybal.log lvs1010.eqiad.wmnet: -rw-r--r-- 1 root root 0 Mar 8 06:25 /var/log/pybal.log lvs1004.wikimedia.org: -rw-r----- 1 root adm 0 Mar 10 06:25 /var/log/pybal.log lvs1005.wikimedia.org: -rw-r----- 1 root adm 0 Mar 13 06:25 /var/log/pybal.log lvs1002.wikimedia.org: -rw-r----- 1 root adm 0 Mar 8 06:25 /var/log/pybal.log lvs1001.wikimedia.org: -rw-r----- 1 root adm 0 Mar 10 06:25 /var/log/pybal.log lvs1003.wikimedia.org: -rw-r----- 1 root adm 1366881 Mar 14 11:33 /var/log/pybal.log lvs1009.eqiad.wmnet: -rw-r--r-- 1 root root 257997 Mar 14 11:30 /var/log/pybal.log lvs1012.eqiad.wmnet: -rw-r--r-- 1 root root 272986 Mar 14 11:33 /var/log/pybal.log lvs1006.wikimedia.org: -rw-r----- 1 root adm 1380554 Mar 14 11:33 /var/log/pybal.log
While trying to figure out what's happening, I've found that pybal's logrotate script is using reload rsyslog instead of service rsyslog rotate:
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscriptAlthough that's certainly wrong, it doesn't seem to be the cause of pybal's silence, which needs to be investigated further.