Create a Cloud VPS SMTP smarthost
Open, HighPublic

Description

Currently Cloud VPS tenant instances use the production exim servers to relay mail from. This is bad for a number of reasons (possible spam/RBLs, unnecessary load etc.) We need to have cloud-specific smarthosts.

Additionally, we should integrate this with LDAP and have root@ go to sysadmins of particular projects (T61142).

The introduction of the new 172.x private range for the eqiad1-r deployment makes this more urgent as instances in the new range cannot access the existing exim servers at all because they are outside of the 10.x allowed range.

Details

Reference
bz39785

Related Objects

There are a very large number of changes, so older changes are hidden. Show Older Changes
faidon created this task.Aug 30 2012, 1:45 AM
  • Bug 36511 has been marked as a duplicate of this bug. ***
  • Bug 36996 has been marked as a duplicate of this bug. ***
Aklapper removed RyanLane as the assignee of this task.Apr 26 2015, 12:11 PM

Labs still route through prod servers:

$ exim -bt foo@example.org
foo@example.org
  router = smart_route, transport = remote_smtp
  host polonium.wikimedia.org [2620:0:861:3:208:80:154:90] 
  host polonium.wikimedia.org [208.80.154.90]              
  host lead.wikimedia.org     [2620:0:861:3:208:80:154:89] 
  host lead.wikimedia.org     [208.80.154.89]              
$ 
faidon merged a task: Restricted Task.Jun 6 2016, 11:14 PM
hashar removed a subscriber: hashar.Jul 23 2016, 9:51 PM
faidon edited projects, added Mail; removed Cloud-Services.Aug 30 2017, 2:40 PM
faidon assigned this task to herron.
faidon added a subscriber: herron.

This is a WMCS task, but since this use case is currently supported by the production mailservers and that's a long-standing problem (and risk) for us, perhaps it's still worth it for prod ops to spend the time for setting it up. @herron, is that something that you could help with?

We'd still need a home for it and I think ideally it belongs to a Cloud VPS instance, rather than something in prod. cloud-services-team, what do you think?

For sure! Maybe a pair of instances in different locations for durability?

Regarding handling root@ mail for individual instances. If it would be useful for some projects to have inbound mail capability, and presuming that instances reside under the wmflabs.org zone, we could consider deploying a wildcard MX record of *.wmflabs.org pointed at this new mail system. Mail to user@host.wmflabs.org could be centrally filtered and routed onward to that host MTA where aliases (like root@) could be expanded, messages delivered, processed, etc.

Indeed! Note that ToolForge already has something like that for tool authors that does LDAP calls etc. if I recall correctly, so perhaps these two efforts could complement each other or even coalesced. Let's split into separate relays first, then all kinds of possibilities exist on how to route WMCS emails :)

chasemp added a subscriber: chasemp.EditedAug 30 2017, 4:17 PM

That Toolforge mail server is a real mess. It may be the remaining holdover from the early days of un-puppetized things and we have been kicking that can down the road for a good long while. Any chance at improvement there is very welcome.

bd808 added a subscriber: bd808.Aug 30 2017, 4:27 PM

I was actually just asking @bd808 about the status of mail access for Cloud VPS projects yesterday. At least for my use case, it would be great if mail to <something>@<projectname>.wmflabs.org (where something is fixed string(s) or any string) just worked for forwarding to the project members, without any setup required in the individual project.

bd808 added a comment.Aug 30 2017, 6:21 PM

it would be great if mail to <something>@<projectname>.wmflabs.org (where something is fixed string(s) or any string) just worked for forwarding to the project members, without any setup required in the individual project.

I'd love to see some solution for this too. One potentially tricky step here is that project membership information has been moved out of the LDAP directory and is now completely managed in OpenStack's keystone API. Next tricky step is that <projectname>.wmflabs.org does not exist in DNS by default. This could be handled by the *.wmflabs.org MX record suggested by @herron in T41785#3566374 though. It is possible for projectX to register an HTTP proxy named projectY.wmflabs.org which could complicate/confuse things based on hostname level forwarding. It might be simpler to use addresses like <projectname>@<vps mail relay identifier>.wmflabs.org for the "contact a project's members" use-case. <vps mail relay identifier> could be something like projects or vps.

For sure! Maybe a pair of instances in different locations for durability?

You mean on separate physical hosts, right? I think we're still limited to eqiad if it is to be a Cloud VPS instance :)

Next tricky step is that <projectname>.wmflabs.org does not exist in DNS by default.

We could just say these projects can't get mail until they get their domain registered under wmflabsdotorg (and don't use it for an HTTP proxy)
There's probably still some ticket around here somewhere for allowing an HTTP proxy at the root of a wmflabs.org subdomain.

! In T41785#3567277, @Krenair wrote:

You mean on separate physical hosts, right? I think we're still limited to eqiad if it is to be a Cloud VPS instance :)

Nah, this doesn't seem like a workload that justifies physical hardware. I just mean that if we can avoid a SPOF that's a good thing, and if multi-site is possible all the better.

! In T41785#3567277, @Krenair wrote:

You mean on separate physical hosts, right? I think we're still limited to eqiad if it is to be a Cloud VPS instance :)

Nah, this doesn't seem like a workload that justifies physical hardware. I just mean that if we can avoid a SPOF that's a good thing, and if multi-site is possible all the better.

I meant ensuring that your two labs instances live on separate existing virtualisation hosts. multi-site isn't an option AFAIK

Also see T47827, T47828, T47829 and T61142. This task is supposed to be for the smarthost which sounds like a good first step. I'd recommend keeping separate instances for inbound and outbound email for configuration simplicity too (something that we really should do for production as well).

bd808 added a comment.Aug 30 2017, 7:40 PM

Should we start a Cloud-VPS project in the spirit of bastion to host VMs for inbound/outbound SMTP services?

Dzahn removed a subscriber: Dzahn.Sep 6 2017, 5:30 PM
faidon moved this task from Backlog to Up Next on the Mail board.Aug 31 2018, 3:39 PM
bd808 added a subscriber: Harej.Fri, Sep 21, 8:10 PM

@Andrew & @Harej: The bump of this ticket after figuring out that the eqiad-r region needs a new set of mail hosts means that we do not want to delete the project-smtp project.

bd808 renamed this task from Create a labs SMTP smarthost to Create a Cloud VPS SMTP smarthost.Fri, Sep 21, 8:14 PM
bd808 updated the task description. (Show Details)
herron raised the priority of this task from Normal to High.Fri, Sep 21, 8:15 PM

So in terms of IP addressing we will need to assign the cloud outbound smtp instance a public IP (or static NAT) address with matching forward and reverse DNS. Will a floating IP address accomplish that? If so, could we add quota for one (or more) to project-smtp?

Instances smtp-out1001 and smtp-out1002 have been created in project-smtp and floating IPs assigned. Do DNS entries for these floating IPs (to be seen by downstream mail systems) belong in a certain subdomain for cloud services? I don't have a strong preference on the name so long as forward and reverse DNS are matching.

We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should keep the project-smtp reference. It probably doesn't matter much.

We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should keep the project-smtp reference. It probably doesn't matter much.

I do think the name should reflect "out" in some way since they are smarthosts for outbound mail. Something along the lines of mx-out, smtp-out, etc. but no strong preference.

Krenair added a comment.EditedMon, Sep 24, 3:47 PM

We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should keep the project-smtp reference. It probably doesn't matter much.

I do think the name should reflect "out" in some way since they are smarthosts for outbound mail. Something along the lines of mx-out, smtp-out, etc. but no strong preference.

well, ok, something like:
mx-out0[12].wmflabs.org
mx-out0[12].project-smtp.wmflabs.org
?

herron moved this task from Up Next to In Progress on the Mail board.Mon, Sep 24, 4:01 PM

Public DNS records have been created, but it looks like like reverse dns will have to wait until T199374 is resolved. This doesn't block the setup of the smarthosts themselves, but I think we'll want reverse DNS in place before directing any important mail flow through these hosts.

Ended up mirroring the instance names in hopes of making things intuitive for future troubleshooters.

smtp-out1001.project-smtp.wmflabs.org has address 185.15.56.15
smtp-out1002.project-smtp.wmflabs.org has address 185.15.56.17

Two new systems have been set up with role::mail::mx. A cursory test of outbound mail routing through a smarthost in project-smtp works, as long as the recipient does not have a wikimedia.org address. Wikimedia.org addresses fail initially on an ldap lookup, which isn't something the outbound smarthost needs to be doing.

H=keith-puppetmaster.puppet.eqiad.wmflabs [10.68.20.66]:52536 I=[172.16.1.228]:25 F=<root@wmflabs.org> temporarily rejected RCPT <kherron@wikimedia.org>: failed to bind the LDAP connection to server ldap-corp.codfw.wikimedia.org:389 - ldap_bind() returned -1

Really I think we need a new role here since much of the inbound mail handling config (recipient verification, alias expansion, spamassassin, rbl lookups, conditional routing, etc.) do not apply to outbound mail routing. So next up will be working on a new outbound smarthost role.

Public DNS records have been created, but it looks like like reverse dns will have to wait until T199374 is resolved. This doesn't block the setup of the smarthosts themselves, but I think we'll want reverse DNS in place before directing any important mail flow through these hosts.

It's solved but @faidon's comment at T199374: Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron) says to wait a couple of days before relying on it.

This is exciting to see materialize, it has been a long time coming :)

So, a few different things:

  • This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?). I'm not sure if it makes sense, but my suggestion would also be to expand the scope and possibly name of this "project-smtp" project to cover all kinds of shared infrastructure needed for VPS instances and managed by the "core" admin team, e.g. labspuppetmasters, cumin masters, monitoring tools etc. in the future.
  • The outgoing/incoming split is something that needs to happen in prod as well (T175362), so it's great that these goals align and we could share manifests for outgoing relays! I foresee some differences still (the wiki-mail stuff, DKIM, etc.), but we could potentially factor these out as well.
  • I think in prod these we might go for the "mx-outNNNN" names to make it similar to the existing "mxNNNN" names, so it may make sense to align here? It doesn't matter that much though, and my apologies for the bikeshedding :)
  • Once the provisioning of these new relays completes, we'll need to a) adjust realm.pp in Puppet to point to these instances (cf. the FIXME there) and b) delete the router ACL holes from WMCS->prod's mx1001/mx2001. I'm not very familiar with the Neutron project; would this need to be blocked on the Neutron migration, or is connectivity between the "old" and "new" networks possible (and not frowned upon)?
  • This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?)

That's my understanding. It is easy to point a name directly under wmflabs.org (without project-smtp) at this.

I'm not sure if it makes sense, but my suggestion would also be to expand the scope and possibly name of this "project-smtp" project to cover all kinds of shared infrastructure needed for VPS instances and managed by the "core" admin team, e.g. labspuppetmasters, cumin masters, monitoring tools etc. in the future.

Possibly. I don't know how easy it is to rename tenants, to rename it's likely a case of replacing all the instances. 'project-smtp' is named after 'project-proxy' which contains the webproxy instances, and I wonder if the restricted bastions might fall under the same umbrella in some ways.

I'm not very familiar with the Neutron project; would this need to be blocked on the Neutron migration, or is connectivity between the "old" and "new" networks possible (and not frowned upon)?

Connectivity between the old and new labs networks appears to be working as a result of T202636 (I think it's pretty much required to do a smooth migration without widespread downtime), I'm not sure how willing people are to depend on it more than as necessary for the migration. There are/were some problems with it still.

  • This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?)

That's my understanding. It is easy to point a name directly under wmflabs.org (without project-smtp) at this.

Hopefully we can settle on a single record under wmflabs.org for each host. That would simplify configuration of components like let's encrypt. FQDN (or as close as possible) can be helpful when tracing/troubleshooting messages that have hopped through remote mail systems.

  • I think in prod these we might go for the "mx-outNNNN" names to make it similar to the existing "mxNNNN" names, so it may make sense to align here?

Additionally in prod we could consider adding a layer of frontend MX service records to decouple our MX records from actual hostnames freeing us to do things like depool individual backend MX hosts for maintenance. Probably worth some further discussion, and thought about outbound service records too. E.g. will we create an outbound mail service record like smtp.wikimedia.org (analogous to smtp.gmail.com, smtp.fastmail.com, etc.)? And if so does the distinction then become mx (inbound) and smtp (outbound)? Maybe we could even drop the "out" and simplify to mxNNNN and smtpNNNN?

  • This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?)

That's my understanding. It is easy to point a name directly under wmflabs.org (without project-smtp) at this.

Hopefully we can settle on a single record under wmflabs.org for each host. That would simplify configuration of components like let's encrypt. FQDN (or as close as possible) can be helpful when tracing/troubleshooting messages that have hopped through remote mail systems.

Let's create mx-out0[12].wmflabs.org records pointing at these hosts then?

I don't care a whole lot about having one infrastructure project or a bunch of small ones but it would be moderately easier to manage security policy if there's one big one. So, pursuant to that I've created a new project named 'cloudinfra' and added Keith to it. @herron, if you feel like it and it's not a ton of trouble you could rebuild those VMs there, and then we can delete the project-smtp project. We'll wind up with new IPs and all over there and then I can set up DNS. Adding foo.wmflabs.org records is easy but I'll need to do it.

Change 463143 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] WIP: smarthost: create mail smarthost role/profile

https://gerrit.wikimedia.org/r/463143

rebuild those VMs there, and then we can delete the project-smtp project. We'll wind up with new IPs and all over there and then I can set up DNS.

Sounds good! New VMs, floating IPs and DNS records have been spun up, this time as mx-out0[12].cloudinfra.eqiad.wmflabs. Project-smtp can be torn down at earliest convenience.

And a quick update -- The WIP patch above is applied to mx-out01 via a standalone puppetmaster, and cursory testing shows this working in that mail sent through the smarthost is relayed off to a remote mail system. Of course more work and testing to do, but progressing.

I created public DNS:

Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out01.cloudinfra.wmflabs.org
185.15.56.18
Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out02.cloudinfra.wmflabs.org
185.15.56.19
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.

Is there anything else I can do to help with this?

I think we were going to do DNS directly under wmflabs.org, if the name is decided?

Yeah, we were, and I did, and then I pasted the wrong thing :)

Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out01.wmflabs.org
185.15.56.18
Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out02.wmflabs.org
185.15.56.19
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.
mx-out01.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
mx-out02.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.
mx-out01.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
mx-out02.wmflabs.org.

Multiple PTRs are not wrong per se, but they're unusual and given the relationship between DNS and email reputation, you may run into trouble there. Or not, I've never experienced this kind of scenario before so I'm not sure what will happen :)

Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.
mx-out01.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
mx-out02.wmflabs.org.

Multiple PTRs are not wrong per se, but they're unusual and given the relationship between DNS and email reputation, you may run into trouble there. Or not, I've never experienced this kind of scenario before so I'm not sure what will happen :)

PTRs get set up automatically. Can just remove the mx-out02.cloudinfra.wmflabs.org A record and it should soon drop the extra PTR.

Ok, I've deleted the A records mx-outNN.cloudinfra.wmflabs.org. FWIW was not intending to have two sets of records. Like I mentioned in T41785#4615256 going shorthand here will make it a bit less clear what VM/project these relate to when tracing a message through received headers or logs (requires more institutional knowledge). But that's not a big problem.

With this then I think we'll use mx-outNN.wmflabs.org in client configs (to keep certs simple), which means mail traffic will continue flowing through the various public outbound NAT IPs depending on the instance config. @Andrew to that end what subnets should be added to the security group, and relay_from_hosts to accept mail for relay from cloud hosts?

Krenair added a comment.EditedFri, Sep 28, 2:50 PM

mail traffic will continue flowing through the various public outbound NAT IPs depending on the instance config

Hm, I think labsaliaser isn't working as expected for eqiad1 hosts, but I'm not sure it matters as the old restriction against routing from a labs instance to a labs floating IP doesn't appear to apply to eqiad1 floating IPs?

JKSTNK added a subscriber: JKSTNK.Fri, Sep 28, 3:02 PM

@herron, the IP ranges for VMs are:

eqiad1: 172.16.0.0/21
eqiad: 10.68.16.0/21

We often just use 10.0.0.0/8 for eqiad since production is unable to connect directly anyway.

@herron, the IP ranges for VMs are:

eqiad1: 172.16.0.0/21
eqiad: 10.68.16.0/21

We often just use 10.0.0.0/8 for eqiad since production is unable to connect directly anyway.

@Andrew Thanks, and then what about on the public side? Afaict we'll use the mx-outNN.wmflabs.org records in cloud/labs mail client configs and will need to whitelist the public ranges that this traffic will be NAT to.

Afaict we'll use the mx-outNN.wmflabs.org records in cloud/labs mail client configs and will need to whitelist the public ranges that this traffic will be NAT to.

Is this right? From what I can tell there are at least two options for accepting mail from clients in cloud, each with pros/cons.

  1. Clients send mail to mx-outNN.wmflabs.org (floating IP). This simplifies certificate config, with the downside of mail traffic from cloud hosts masquerading as outbound public NAT IP addresses like internal-server-nat.wmflabs.org (unless there is a workaround to have RFC1918 space route to/from the floating IPs without NAT)
  2. Clients send mail to mx-outNN.cloudinfra.eqiad.wmflabs. Mx-out hosts will see private IP addresses this way (useful for incident response, rate limiting, etc.), with the downside of certificate complexity. We could use puppet exposed certs here, but with standalone puppetmasters ca certs/validation might get messy. I'm assuming we wouldn't want to put RFC1918 addresses in the wmflabs.org zone, but if we did that we should be able to use a LE cert with an extra SAN for that record.
faidon added a comment.Tue, Oct 2, 3:18 PM

I'm not familiar with WMCS' networking -- does this floating IP DNAT imply SNAT as well? i.e. would it not be possible to have flows of the form (RFC1918) -> mx-outNN.wmflabs.org (floating IP)?

Krenair added a comment.EditedTue, Oct 2, 3:30 PM

In the current 'main' deployment there's a thing called labsaliaser which causes the DNS recursors to substitute responses containing labs public floating IPs with the equivalent private fixed IPs. It doesn't appear to be doing that for eqiad1 (it doesn't seem necessary as there isn't the old restriction around routing to floating IPs here), but I guess it could be fixed/enabled (the fact that it's not currently doing this against eqiad1 might be a side effect of the way both regions are currently set up and may magically work when the main deployment is gone) if it would help such cases? I think the outcome would be that they'd send to mx-outNN.wmflabs.org but the IP address they'd use for it would be internal.

herron added a comment.Tue, Oct 2, 3:50 PM

In the current 'main' deployment there's a thing called labsaliaser which causes the DNS recursors to substitute responses containing labs public floating IPs with the equivalent private fixed IPs.

More or less automatic split-brain DNS? Sounds like a workable approach. Is it on the roadmap for eqiad1?

would it not be possible to have flows of the form (RFC1918) -> mx-outNN.wmflabs.org (floating IP)?

Curious about this too. IMHO this would be preferable over split-brain DNS.

I suppose if either of these is on the roadmap (or can be added) we can move forward with using the dns names associated with the floating IPs (mx-outNN.wmflabs.org) in client configs. We would just need to enumerate and whitelist the NAT addresses being used today and plan on their removal in the future.

Harej removed a subscriber: Harej.Tue, Oct 2, 4:03 PM
herron moved this task from Backlog to Working on on the User-herron board.Tue, Oct 2, 5:05 PM
Andrew added a comment.Thu, Oct 4, 9:25 PM

For what it's worth, the aliases in eqiad1 can be fixed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/464722/ or something like it.

labsaliaser is working now:

krenair@bastion-eqiad1-01:~$ dig eqiad1.bastion.wmflabs.org @8.8.8.8 +short
185.15.56.13
krenair@bastion-eqiad1-01:~$ dig eqiad1.bastion.wmflabs.org +short
172.16.1.136
krenair@bastion-eqiad1-01:~$ dig mx-out01.wmflabs.org @8.8.8.8 +short
185.15.56.18
krenair@bastion-eqiad1-01:~$ dig mx-out01.wmflabs.org +short
172.16.1.239