Page MenuHomePhabricator

Change notification email from jenkins-bot@wikimedia.org to releng internal list
Open, LowPublic

Description

Cron jobs should send notifications directly to Release-Engineering-Team instead of using the alias jenkins-bot@wikimedia.org which only reach a few people. From our internal discussion:

If it's from a @wikimedia.org email address, [greg has] whitelisted those (for people [outside of releng]).

The list Jenkins sends email to are:

ListStatus of releng@lists.wikimedia.org
betacluster-alerts@lists.wikimedia.orgNon-member, Held for moderation
discovery-alerts@lists.wikimedia.orgDone by @Gehel Non-member, Held for moderation
portals@lists.wikimedia.org?? No bounce and list is private
qa-alerts@lists.wikimedia.orgOK, Non-member, set per user moderation to Accept immediately
releng@lists.wikimedia.orgOK
security-admin-feed@lists.wikimedia.orgNon-member, Held for moderation
wdqs-gui-build@lists.wikimedia.org?? No bounce and list is private

Event Timeline

For jenkins@contint1001.wikimedia.org the emails will have the header:

From: "Jenkins,,," <jenkins@contint1001.wikimedia.org>

Is everything done here or are changes still being made?

Nothing has been done. It is a random idea I had a couple months ago which I filled as a task to make sure it is not forgotten. I haven't even looked at all places using the jenkins-bot email address.

I've recently been getting a few of these (mostly from @Jdforrester-WMF cleaning up portalsbuild?). It's not a major problem, but looking at the list of who's on jenkins-bot@ (https://groups.google.com/a/wikimedia.org/g/jenkins-bot/members ) this seems like an odd setup indeed.

Anyway, still Low priority, just keeping logs for next time this happens and I search phab again :)

I've recently been getting a few of these (mostly from @Jdforrester-WMF cleaning up portalsbuild?).

Sorry! ;-)

Change 732968 had a related patch set uploaded (by Hashar; author: Hashar):

[operations/puppet@production] zuul: use releng list rather than jenkins-bot for email

https://gerrit.wikimedia.org/r/732968

Found a few in Puppet though they don't trigger any email.

Icinga got done a couple years ago by https://gerrit.wikimedia.org/r/c/operations/puppet/+/559858

There is no trace of jenkins-bot email in the JJB config

I have updated CI and release Jenkins admin email address to releng@lists.wikimedia.org

Puppet patch https://gerrit.wikimedia.org/r/732968 is the one left to mark this resolved.

Change 732968 merged by Dzahn:

[operations/puppet@production] zuul: use releng list rather than jenkins-bot for email

https://gerrit.wikimedia.org/r/732968

After I have updated the CI Jenkins admin address from jenkins-bot to releng, email notifications sent to qa-alerts@lists.wikimedia.org would bounce back:

Your message to the QA-Alerts mailing-list was rejected for the following reasons:

The message is not from a list member

Before changing the system admin email, we should audit all the lists being recipients, make sure releng is a member (and disable mail notification to it).

(I have rolled back the change on CI Jenkins which is now back to using the jenkins-bot email)

I have created a jenkins-bot account on lists.wikimedia.org and stored its credentials in releng secrets store.

Whenever Jenkins emits an email from jenkins-bot@lists.wikimedia.org it might get bounced by the recipient mailing list, over time the email has been added to the target lists and we would need to do the same for releng@lists.wikimedia.org.

From integration/config.git we can retrieve all the emails Jenkins jobs send email too with git grep -hoP '[\w-]+@lists.wikimedia.org'|sort|uniq.

In a job?List member ?ListDescription
YESYESbetacluster-alerts@lists.wikimedia.org
YESYESdiscovery-alerts@lists.wikimedia.orgAutomated notifications to search platform team
NOYESmultimedia-alerts@lists.wikimedia.orgAutomated Multimedia Alerts
YESYESportals@lists.wikimedia.orgWMF Portal Updates
YESYESqa-alerts@lists.wikimedia.orgMailing list for Quality Assurance alerts
YESNOreleng@lists.wikimedia.orgRelease engineering list
YESYESsecurity-admin-feed@lists.wikimedia.orgAutomated messages from security related tooling.
YESNOwdqs-gui-build@lists.wikimedia.org

The jobs sending to multimedia-alerts have been removed a while back.

I tried to create an account on mailman for releng@lists.wikimedia.org and that results in a server side error, I guess cause it is a list.

So in the end, we need releng@lists.wikimedia.org to be allowed to post on the list, which in mailman 3 can be granted even if the email is not an account nor a member of the list. But that requires each list owners to set the preference.

Example for qa-alerts for which I am a owner. Looking at https://lists.wikimedia.org/postorius/lists/qa-alerts.lists.wikimedia.org/members/nonmember/?q=releng gives:

releng_non-member_qa-alerts.png (412×819 px, 38 KB)

Since I am a list owner, I have access Non-members Options for that email which let me change the Delivery Status (none for a non member) or more importantly how the list react when it receives an email from that email:

qa-alerts-releng-moderation.png (227×947 px, 42 KB)

And it shows List default which depends on the list. I could not find the default setting for the list, but I guess that for each recipient list we can set it this non member to either Accept or Default Processing (which does "additional checks" whatever it can mean).

The list Jenkins sends email to are:

List
betacluster-alerts@lists.wikimedia.org
discovery-alerts@lists.wikimedia.org
portals@lists.wikimedia.org
qa-alerts@lists.wikimedia.org
releng@lists.wikimedia.org
security-admin-feed@lists.wikimedia.org
wdqs-gui-build@lists.wikimedia.org

So in the end, we need releng@lists.wikimedia.org to be allowed to post on the list, which in mailman 3 can be granted even if the email is not an account nor a member of the list. But that requires each list owners to set the preference.
<snip>

List
betacluster-alerts@lists.wikimedia.org
discovery-alerts@lists.wikimedia.org
portals@lists.wikimedia.org
qa-alerts@lists.wikimedia.org
releng@lists.wikimedia.org
security-admin-feed@lists.wikimedia.org
wdqs-gui-build@lists.wikimedia.org

Done for all of these lists.

Thank you so much @Legoktm for proposing your help and fixing the rights. I will switch Jenkins to emit emails from releng@ tomorrow \o/

discovery-alerts@lists.wikimedia.org is now accepting emails from releng@

Thank you @Gehel.

I have send an email to each mailing lists owners with the detailed instructions.

I received the email, but it doesn’t look like there’s anything left to be done for wdqs-gui-build, releng@ is already set to “accept immediately” there. (I guess that’s what @Legoktm did, but then I’m not sure why the email was sent.)

@hashar: Removing task assignee as this open task has been assigned for more than two years - see the email sent to all task assignees on 2024-04-15.
Please assign this task to yourself again if you still realistically [plan to] work on this task - it would be welcome! :)
If this task has been resolved in the meantime, or should not be worked on by anybody ("declined"), please update its task status via "Add Action… 🡒 Change Status".
Also see https://www.mediawiki.org/wiki/Bug_management/Assignee_cleanup for tips how to best manage your individual work in Phabricator. Thanks!