I got a warning from icinga low on space on / for db1047.

I took a quick look (I'm on mobile) and saw that there was the log for /usr/local/bin/eventlogging_sync.sh that was filling up all the space because of an infinite loop. The error message in the error log was:

/usr/local/bin/eventlogging_sync.sh: line 37: mysql: command not found

I saw that it is managed by puppet, so I decided to kill it, rotate manually the logs (both log and err that were pretty big) and run puppet to make it start, hoping that i would take the proper environment to work again.

Unfortunately it didn't but somehow changed the way it's failing. Now the error is just:

ERROR 1045 (28000): Access denied for user 'root'@'10.64.16.148' (using password: NO)

So it seems that now is not reading the credentials from /root/.my.cnf

After digging a bit I found that today was merged https://gerrit.wikimedia.org/r/#/c/285208/ that changed the way the script is run, but actually breaking it in all hosts where is configured to run.

I'm not sure why the behaviour is different between the two runs.