We got a queue length alert over night (hey it works!) and because our last foray in this regard I took a look. It seems innocuous but still problematic.
TLDR a prometheus cron for puppet status (I think) has been failing a lot and it's flooding the mail system with emails to a fictitious prometheus@tools.wmflabs.org from root.
@fgiunchedi anything change recently on prometheus side?
exim -bpc
69
exim -bp | exiqsumm
Count Volume Oldest Newest Domain
----- ------ ------ ------ ------
1 6553 20h 20h 4998565820.tools.wmflabs.org
1 6553 21h 21h 7584391580.tools.wmflabs.org
1 106KB 24h 24h caisson.bicuz.life
1 106KB 20h 20h ethyl.cuner.life
1 106KB 43h 43h kavwoz.life
1 5017 10h 10h proof.ocn.ne.jp
1 106KB 22h 22h releasable.nubag.life
691 1589KB 25h 13h tools.wmflabs.org
---------------------------------------------------------------
698 2031KB 43h 10h TOTALexim -bp | grep '@' | sort | uniq -c | sort -n
1 impotent@releasable.nubag.life 1 server@4998565820.tools.wmflabs.org 1 server@7584391580.tools.wmflabs.org 1 shingly@ethyl.cuner.life 1 totalhomeprotection@kavwoz.life 1 unrobe@caisson.bicuz.life 1 "www."@proof.ocn.ne.jp 691 prometheus@tools.wmflabs.org
exim -Mvh 1dztJd-0007r1-PR
1dztJd-0007r1-PR-H Debian-exim 107 110 <> 1507160581 0 -ident Debian-exim -received_protocol local -body_linecount 39 -max_received_linelength 143 -allow_unqualified_recipient -allow_unqualified_sender -frozen 1507160581 -localerror XX 1 prometheus@tools.wmflabs.org 164P Received: from Debian-exim by mail.tools.wmflabs.org with local (Exim 4.82) id 1dztJd-0007r1-PR for prometheus@tools.wmflabs.org; Wed, 04 Oct 2017 23:43:01 +0000 050 X-Failed-Recipients: prometheus@tools.wmflabs.org 029 Auto-Submitted: auto-replied 061F From: Mail Delivery System <Mailer-Daemon@tools.wmflabs.org> 033T To: prometheus@tools.wmflabs.org 059 Subject: Mail delivery failed: returning message to sender 055I Message-Id: <E1dztJd-0007r1-PR@mail.tools.wmflabs.org> 038 Date: Wed, 04 Oct 2017 23:43:01 +0000
exim -Mvb 1dztJd-0007r1-PR
1dztJd-0007r1-PR-D
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
prometheus@tools.wmflabs.org
Unrouteable address
------ This is a copy of the message, including all the headers. ------
Return-path: <prometheus@tools.wmflabs.org>
Received: from tools-bastion-03.tools.eqiad.wmflabs ([10.68.23.58] ident=Debian-exim)
by mail.tools.wmflabs.org with esmtp (Exim 4.82)
(envelope-from <prometheus@tools.wmflabs.org>)
id 1dztJd-0007qv-Eu
for prometheus@tools.wmflabs.org; Wed, 04 Oct 2017 23:43:01 +0000
Received: from prometheus by tools-bastion-03.tools.eqiad.wmflabs with local (Exim 4.82)
(envelope-from <prometheus@tools.wmflabs.org>)
id 1dztJd-00006j-DO
for prometheus@tools.wmflabs.org; Wed, 04 Oct 2017 23:43:01 +0000
From: root@tools.wmflabs.org (Cron Daemon)
To: prometheus@tools.wmflabs.org
Subject: Cron <prometheus@tools-bastion-03> /usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom
Content-Type: text/plain; charset=ANSI_X3.4-1968
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/lib/prometheus>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=prometheus>
Message-Id: <E1dztJd-00006j-DO@tools-bastion-03.tools.eqiad.wmflabs>
Date: Wed, 04 Oct 2017 23:43:01 +0000
Traceback (most recent call last):
File "/usr/local/bin/prometheus-puppet-agent-stats", line 117, in <module>
sys.exit(main())
File "/usr/local/bin/prometheus-puppet-agent-stats", line 111, in main
write_to_textfile(args.outfile, registry)
File "/usr/lib/python2.7/dist-packages/prometheus_client/exposition.py", line 115, in write_to_textfile
with open(tmppath, 'wb') as f:
IOError: [Errno 13] Permission denied: u'/var/lib/prometheus/node.d/puppet_agent.prom.411.140678593820480'