The text below is what i originally sent to the ops mailing list, but i am pasting into a ticket here was well so that @Paladox and @Zppix can read and follow-up on it.
We received LOTS of "mail delivery failed" mails originating from phabricator.phabricator.eqiad.wmflabs starting
today, starting about 7.5 hours ago. ( ~ 5 PM UTC)
Google moved all of them into Spam but i noticed when emptying spam folder.
We have puppetized in /etc/aliases
root root@wikimedia.org
and then when it tries to mail root@wikimedia.org, sender verification fails:: 550-Verification failed for <root@phabricator.phabricator.eqiad.wmflabs>"
It's from a root cronjob and the culprit is " /usr/local/bin/exim-to-gmetric"/
The reason for exim-to-gmetric failing is:
/usr/sbin/exim: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18: no version information available (required by /usr/sbin/exim)
and this should be from:
10:12 < paladox> !log phabricator upgrading mariadb to 10.2
because: /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18: symbolic link to libmariadb.so.3
The cron comes from class exim4::ganglia , so something that is deprecated.
What i am doing for now is comment out the include of exim4::ganglia in the phabricator role.
(It gets included in other places too, let's see if we shouldn't remove all of that right away?)