Page MenuHomePhabricator
Create Task
Maniphest T196920

Add email queueing/failover to services currently using mail_smarthost[0]
Closed, ResolvedPublic
Actions

Description

Several services are configured by puppet with a mail server of mail_smarthost[0]. This selects only the first mail server from a per-site ordered list of mail servers (defined in manifests/realm.pp), and results in an SPOF.

To address this I propose configuring these services to use the exim localhost SMTP listener. This moves the SPOF to the local MTA which has no external networking or service dependencies beyond the locally running daemon. The local MTA then handles relay to the appropriate smarthost with more robust queueing and failover.

This is a follow-up from T196598: Phab and Gerrit emails stopped at around 1900 UTC 6th June.

Details

SubjectRepoBranchLines +/-
oozie: change smtp_host to localhostoperations/puppetproduction+1 -1
iegreview: change smtp_host to localhostoperations/puppetproduction+1 -1
wikimania_scholarships: change smtp_host to localhostoperations/puppetproduction+1 -1
sentry: change EMAIL_HOST to localhostoperations/puppetproduction+1 -1
wikidump: change smtpserver to localhostoperations/puppetproduction+1 -1
gerrit: use localhost exim as smtp serveroperations/puppetproduction+1 -1
hue: change smtp_host to localhostoperations/puppetproduction+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

herron created this task.Jun 11 2018, 4:00 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 11 2018, 4:00 PM
herron added a project: Wikimedia-Incident.Jun 11 2018, 7:51 PM
herron moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
Joe triaged this task as High priority.Jun 18 2018, 10:48 AM
gerritbot subscribed.Jun 18 2018, 8:41 PM

Change 440970 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] gerrit: use localhost exim as smtp server

https://gerrit.wikimedia.org/r/440970

gerritbot added a project: Patch-For-Review.Jun 18 2018, 8:41 PM
gerritbot added a comment.Jun 19 2018, 8:48 PM

Change 441130 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] hue: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441130

gerritbot added a comment.Jun 19 2018, 8:48 PM

Change 441131 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] iegreview: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441131

gerritbot added a comment.Jun 19 2018, 8:48 PM

Change 441132 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] oozie: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441132

gerritbot added a comment.Jun 19 2018, 8:48 PM

Change 441133 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] wikimania_scholarships: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441133

gerritbot added a comment.Jun 19 2018, 8:48 PM

Change 441134 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] sentry: change EMAIL_HOST to localhost

https://gerrit.wikimedia.org/r/441134

gerritbot added a comment.Jun 19 2018, 8:48 PM

Change 441135 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] wikidump: change smtpserver to localhost

https://gerrit.wikimedia.org/r/441135

herron added a project: User-herron.Jun 19 2018, 8:51 PM
herron moved this task from Backlog to Awaiting Input/Review on the User-herron board.Jun 28 2018, 6:58 PM
• Vvjjkkii renamed this task from Add email queueing/failover to services currently using mail_smarthost[0] to b9aaaaaaaa.Jul 1 2018, 1:04 AM
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
ArielGlenn renamed this task from b9aaaaaaaa to Add email queueing/failover to services currently using mail_smarthost[0].Jul 1 2018, 6:22 AM
ArielGlenn removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
ArielGlenn updated the task description. (Show Details)
ArielGlenn added subscribers: Aklapper, GerritBot.
gerritbot added a comment.Jul 3 2018, 6:07 PM

Change 440970 merged by Herron:
[operations/puppet@production] gerrit: use localhost exim as smtp server

https://gerrit.wikimedia.org/r/440970

gerritbot added a comment.Jul 9 2018, 6:24 AM

Change 441130 merged by Elukey:
[operations/puppet@production] hue: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441130

Stashbot subscribed.Jul 9 2018, 6:25 AM

Mentioned in SAL (#wikimedia-operations) [2018-07-09T06:25:00Z] <elukey> restart hue on thorium to pick up new smtp changes - T196920

Legoktm updated the task description. (Show Details)Jul 11 2018, 7:16 PM
gerritbot added a comment.Aug 30 2018, 5:03 PM

Change 441135 merged by Herron:
[operations/puppet@production] wikidump: change smtpserver to localhost

https://gerrit.wikimedia.org/r/441135

gerritbot added a comment.Aug 30 2018, 5:10 PM

Change 441134 merged by Herron:
[operations/puppet@production] sentry: change EMAIL_HOST to localhost

https://gerrit.wikimedia.org/r/441134

gerritbot added a comment.Aug 30 2018, 5:18 PM

Change 441133 merged by Herron:
[operations/puppet@production] wikimania_scholarships: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441133

gerritbot added a comment.Sep 4 2018, 4:34 PM

Change 441131 merged by Herron:
[operations/puppet@production] iegreview: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441131

gerritbot added a comment.Sep 5 2018, 6:58 AM

Change 441132 merged by Elukey:
[operations/puppet@production] oozie: change smtp_host to localhost

https://gerrit.wikimedia.org/r/441132

herron closed this task as Resolved.Sep 5 2018, 2:02 PM
herron claimed this task.

The last service using mail_smarthost[0] was migrated to the localhost smtp listener today. Resolving.

herron added a parent task: T175361: Upgrade mx1001/mx2001 to stretch.Sep 5 2018, 2:14 PM
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 28 2020, 10:13 PM
Aklapper edited subscribers, added: gerritbot; removed: GerritBot.May 16 2023, 10:18 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits