Page MenuHomePhabricator

tools-mail queue length alert from prometheus cron
Closed, ResolvedPublic

Description

We got a queue length alert over night (hey it works!) and because our last foray in this regard I took a look. It seems innocuous but still problematic.

TLDR a prometheus cron for puppet status (I think) has been failing a lot and it's flooding the mail system with emails to a fictitious prometheus@tools.wmflabs.org from root.

@fgiunchedi anything change recently on prometheus side?


exim -bpc

69

exim -bp | exiqsumm

Count  Volume  Oldest  Newest  Domain
-----  ------  ------  ------  ------

    1    6553     20h     20h  4998565820.tools.wmflabs.org
    1    6553     21h     21h  7584391580.tools.wmflabs.org
    1   106KB     24h     24h  caisson.bicuz.life
    1   106KB     20h     20h  ethyl.cuner.life
    1   106KB     43h     43h  kavwoz.life
    1    5017     10h     10h  proof.ocn.ne.jp
    1   106KB     22h     22h  releasable.nubag.life
  691  1589KB     25h     13h  tools.wmflabs.org
---------------------------------------------------------------
  698  2031KB     43h     10h  TOTAL

exim -bp | grep '@' | sort | uniq -c | sort -n

  1           impotent@releasable.nubag.life
  1           server@4998565820.tools.wmflabs.org
  1           server@7584391580.tools.wmflabs.org
  1           shingly@ethyl.cuner.life
  1           totalhomeprotection@kavwoz.life
  1           unrobe@caisson.bicuz.life
  1           "www."@proof.ocn.ne.jp
691           prometheus@tools.wmflabs.org

exim -Mvh 1dztJd-0007r1-PR

1dztJd-0007r1-PR-H
Debian-exim 107 110
<>
1507160581 0
-ident Debian-exim
-received_protocol local
-body_linecount 39
-max_received_linelength 143
-allow_unqualified_recipient
-allow_unqualified_sender
-frozen 1507160581
-localerror
XX
1
prometheus@tools.wmflabs.org

164P Received: from Debian-exim by mail.tools.wmflabs.org with local (Exim 4.82)
	id 1dztJd-0007r1-PR
	for prometheus@tools.wmflabs.org; Wed, 04 Oct 2017 23:43:01 +0000
050  X-Failed-Recipients: prometheus@tools.wmflabs.org
029  Auto-Submitted: auto-replied
061F From: Mail Delivery System <Mailer-Daemon@tools.wmflabs.org>
033T To: prometheus@tools.wmflabs.org
059  Subject: Mail delivery failed: returning message to sender
055I Message-Id: <E1dztJd-0007r1-PR@mail.tools.wmflabs.org>
038  Date: Wed, 04 Oct 2017 23:43:01 +0000

exim -Mvb 1dztJd-0007r1-PR

1dztJd-0007r1-PR-D
This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

  prometheus@tools.wmflabs.org
    Unrouteable address

------ This is a copy of the message, including all the headers. ------

Return-path: <prometheus@tools.wmflabs.org>
Received: from tools-bastion-03.tools.eqiad.wmflabs ([10.68.23.58] ident=Debian-exim)
	by mail.tools.wmflabs.org with esmtp (Exim 4.82)
	(envelope-from <prometheus@tools.wmflabs.org>)
	id 1dztJd-0007qv-Eu
	for prometheus@tools.wmflabs.org; Wed, 04 Oct 2017 23:43:01 +0000
Received: from prometheus by tools-bastion-03.tools.eqiad.wmflabs with local (Exim 4.82)
	(envelope-from <prometheus@tools.wmflabs.org>)
	id 1dztJd-00006j-DO
	for prometheus@tools.wmflabs.org; Wed, 04 Oct 2017 23:43:01 +0000
From: root@tools.wmflabs.org (Cron Daemon)
To: prometheus@tools.wmflabs.org
Subject: Cron <prometheus@tools-bastion-03> /usr/local/bin/prometheus-puppet-agent-stats --outfile /var/lib/prometheus/node.d/puppet_agent.prom
Content-Type: text/plain; charset=ANSI_X3.4-1968
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/lib/prometheus>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=prometheus>
Message-Id: <E1dztJd-00006j-DO@tools-bastion-03.tools.eqiad.wmflabs>
Date: Wed, 04 Oct 2017 23:43:01 +0000

Traceback (most recent call last):
  File "/usr/local/bin/prometheus-puppet-agent-stats", line 117, in <module>
    sys.exit(main())
  File "/usr/local/bin/prometheus-puppet-agent-stats", line 111, in main
    write_to_textfile(args.outfile, registry)
  File "/usr/lib/python2.7/dist-packages/prometheus_client/exposition.py", line 115, in write_to_textfile
    with open(tmppath, 'wb') as f:
IOError: [Errno 13] Permission denied: u'/var/lib/prometheus/node.d/puppet_agent.prom.411.140678593820480'

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

I think the cron may be ok now so I'm purging frozen messages with

exiqgrep -z -i | xargs exim -Mrm

Let's see if this comes back. I'm throwing to @bd808 since he's on cloud clinic

• chasemp renamed this task from tools-mail queue length alert to tools-mail queue length alert from prometheus cron.Oct 5 2017, 12:27 PM
shinken shinken@shinken-01.shinken.eqiad.wmflabs via wmflabs.org 
7:32 AM (0 minutes ago)

to me 
Notification Type: RECOVERY

Service: exim queue length
Host: tools-mail
Address: 10.68.16.27
State: OK

Date/Time: Thu 05 Oct 12:32:30 UTC 2017

Additional Info:

OK: Less than 1.00% above the threshold [600.0]

@chasemp I bet that's a side effect of T166561: Rollout prometheus-node-exporter 0.14 in labs, is it persisting or has been transient during package upgrades?

• chasemp closed this task as Resolved.EditedOct 6 2017, 2:10 PM

@chasemp I bet that's a side effect of T166561: Rollout prometheus-node-exporter 0.14 in labs, is it persisting or has been transient during package upgrades?

I haven't seen it again, I this it's a side effect. I'm going to close and wait for resurface :)