Page MenuHomePhabricator

mail from testlabs to ops list
Closed, ResolvedPublic

Description

list admins of the ops mailing list got this:

Pending posts:
From: root@testlabs-puppet-is-broken-here-on-purpose.testlabs.eqiad.wmflabs on Thu Jan 21 08:15:05 2016
Subject: Alert:  puppet failed on testlabs-puppet-is-broken-here-on-purpose.testlabs.eqiad.wmflabs
Cause: Post by non-member to a members-only list

the details and headers of the original message can be seen here:

https://lists.wikimedia.org/mailman/admindb/ops?details=all

..
for ops@wikimedia.org; Thu, 21 Jan 2016 08:15:02 +0000
...
To: ops@lists.wikimedia.org
Subject: Alert:  puppet failed on testlabs-puppet-is-broken-here-on-purpose.testlabs.eqiad.wmflabs
Message-Id: <E1aMAOU-0002fO-89@testlabs-puppet-is-broken-here-on-purpose.testlabs.eqiad.wmflabs>
From: root <root@testlabs-puppet-is-broken-here-on-purpose.testlabs.eqiad.wmflabs>
Date: Thu, 21 Jan 2016 08:15:02 +0000

it is first sent to the ops@wm.org address (without "list"), then forwarded via the alias that we did not expect anyone to use (-> T123581#1934528) to ops@lists and does not get accepted there because the sender address is not a list member

the content looks like this:

Puppet is failing to run on the "testlabs-puppet-is-broken-here-on-purpose" instance in the Wikimedia Labs
project "testlabs"

Working puppet runs are needed to maintain instance security and logins.
As long as puppet continues to fail, this system is in danger of becoming
unreachable.

You are receiving this email because you are listed as an administrator
for the project that contains this instance.  Please take steps to repair
this instance or contact a Labs admin for assistance.

For further support, visit #wikimedia-labs on freenode or visit
http://www.wikitech.org

@Andrew asked me to create a ticket for debugging this

Details

Related Gerrit Patches:
operations/puppet : productionDon't send puppet nags to the novaadmin user.

Event Timeline

Dzahn created this task.Jan 23 2016, 1:39 AM
Dzahn raised the priority of this task from to Needs Triage.
Dzahn updated the task description. (Show Details)
Dzahn added projects: Operations, Cloud-VPS.
Dzahn added subscribers: Dzahn, Andrew.
Restricted Application added a project: Cloud-Services. · View Herald TranscriptJan 23 2016, 1:39 AM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald Transcript
scfc added a subscriber: scfc.Jan 23 2016, 2:52 PM

/usr/local/sbin/puppetalert.py sends mail to the project administrators. One of those is (always?) novaadmin which has the mail address ops@wikimedia.org.

So /usr/local/sbin/puppetalert.py should ignore novaadmin when compiling the list of recipients.

Change 266192 had a related patch set uploaded (by Andrew Bogott):
Don't send puppet nags to the novaadmin user.

https://gerrit.wikimedia.org/r/266192

akosiaris triaged this task as Medium priority.Jan 25 2016, 11:52 AM
akosiaris raised the priority of this task from Medium to High.
akosiaris added a subscriber: akosiaris.
Dzahn added a comment.Jan 25 2016, 5:24 PM

also let's stop using ops@ wherever we see it and use ops@lists , the proper list address

ops@lists

@Dzahn, can you clarify? What is the full email address I should be using here?

Dzahn added a comment.Jan 25 2016, 6:04 PM

@Dzahn, can you clarify? What is the full email address I should be using here?

ops@lists.wikimedia.org please. without the "lists" part it's just an alias to the list and we were about to remove that in T123581 thinking it's not used

Andrew claimed this task.Jan 25 2016, 6:09 PM

Change 266192 merged by Andrew Bogott:
Don't send puppet nags to the novaadmin user.

https://gerrit.wikimedia.org/r/266192

Andrew closed this task as Resolved.Jan 26 2016, 9:08 PM