I'm getting spammed pretty badly by Mailer-Daemon@tools.wmflabs.org. The messages seem to be from promethus@tools.wmflabs.org so this is probably just tools admins getting the flood. A quick search didn't turn up a duplicate ticket, so here's one. If I can find some time, I'll see if I can find out what's wrong, but my inbox is hurting. Please fix this if you can. Thank you!!
The ultimate problem seems to be OSError: [Errno 28] No space left on device on tools-prometheus-05 (and perhaps the other in the pair), but the email alias/list or something like that clearly has some kind of issue.
An example message is below.
This message was created automatically by mail delivery software.
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
root@wmcloud.org
(ultimately generated from prometheus@tools.wmflabs.org)
all hosts for 'wmcloud.org' have been failing for a long time (and retry time not reached)
----------------------------------------------
message/delivery-status
----------------------------------------------
Reporting-MTA: dns; mail.tools.wmflabs.org
Action: failed
Final-Recipient: rfc822;root@wmcloud.org
Status: 5.0.0
----------------------------------------------
message/rfc822
----------------------------------------------
Return-path: <prometheus@tools.wmflabs.org>
Received: from tools-prometheus-05.tools.eqiad1.wikimedia.cloud ([172.16.0.103])
by mail.tools.wmflabs.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.92)
(envelope-from <prometheus@tools.wmflabs.org>)
id 1n4blr-0005C8-S6
for prometheus@tools.wmflabs.org; Tue, 04 Jan 2022 04:50:04 +0000
Received: from prometheus by tools-prometheus-05.tools.eqiad1.wikimedia.cloud with local (Exim 4.92)
(envelope-from <prometheus@tools.wmflabs.org>)
id 1n4blr-00071b-Pi
for prometheus@tools.wmflabs.org; Tue, 04 Jan 2022 04:50:03 +0000
From: root@tools.wmflabs.org (Cron Daemon)
To: prometheus@tools.wmflabs.org
Subject: Cron <prometheus@tools-prometheus-05> /usr/local/bin/prometheus-labs-targets --port 9051 --prefix tools-flannel-etcd- > /srv/prometheus/tools/targets/etcd_flannel.$$ && mv /srv/prometheus/tools/targets/etcd_flannel.$$ /srv/prometheus/tools/targets/etcd_flannel.yml
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/var/lib/prometheus>
X-Cron-Env: <PATH=/usr/bin:/bin>
X-Cron-Env: <LOGNAME=prometheus>
Message-Id: <E1n4blr-00071b-Pi@tools-prometheus-05.tools.eqiad1.wikimedia.cloud>
Date: Tue, 04 Jan 2022 04:50:03 +0000
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
OSError: [Errno 28] No space left on device