@Andrew & @Harej: The bump of this ticket after figuring out that the eqiad-r region needs a new set of mail hosts means that we do not want to delete the project-smtp project.

So in terms of IP addressing we will need to assign the cloud outbound smtp instance a public IP (or static NAT) address with matching forward and reverse DNS. Will a floating IP address accomplish that? If so, could we add quota for one (or more) to project-smtp?

Instances smtp-out1001 and smtp-out1002 have been created in project-smtp and floating IPs assigned. Do DNS entries for these floating IPs (to be seen by downstream mail systems) belong in a certain subdomain for cloud services? I don't have a strong preference on the name so long as forward and reverse DNS are matching.

We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should keep the project-smtp reference. It probably doesn't matter much.

In T41785#4611050, @Krenair wrote:

We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should keep the project-smtp reference. It probably doesn't matter much.

I do think the name should reflect "out" in some way since they are smarthosts for outbound mail. Something along the lines of mx-out, smtp-out, etc. but no strong preference.

In T41785#4611107, @herron wrote:

In T41785#4611050, @Krenair wrote:

We could modify the wmflabs.org zone itself to call them mx01.wmflabs.org and mx02.wmflabs.org or something if we don't think these should keep the project-smtp reference. It probably doesn't matter much.

I do think the name should reflect "out" in some way since they are smarthosts for outbound mail. Something along the lines of mx-out, smtp-out, etc. but no strong preference.

well, ok, something like:
mx-out0[12].wmflabs.org
mx-out0[12].project-smtp.wmflabs.org
?

Public DNS records have been created, but it looks like like reverse dns will have to wait until T199374 is resolved. This doesn't block the setup of the smarthosts themselves, but I think we'll want reverse DNS in place before directing any important mail flow through these hosts.

Ended up mirroring the instance names in hopes of making things intuitive for future troubleshooters.

smtp-out1001.project-smtp.wmflabs.org has address 185.15.56.15
smtp-out1002.project-smtp.wmflabs.org has address 185.15.56.17

Two new systems have been set up with role::mail::mx. A cursory test of outbound mail routing through a smarthost in project-smtp works, as long as the recipient does not have a wikimedia.org address. Wikimedia.org addresses fail initially on an ldap lookup, which isn't something the outbound smarthost needs to be doing.

H=keith-puppetmaster.puppet.eqiad.wmflabs [10.68.20.66]:52536 I=[172.16.1.228]:25 F=<root@wmflabs.org> temporarily rejected RCPT <kherron@wikimedia.org>: failed to bind the LDAP connection to server ldap-corp.codfw.wikimedia.org:389 - ldap_bind() returned -1

Really I think we need a new role here since much of the inbound mail handling config (recipient verification, alias expansion, spamassassin, rbl lookups, conditional routing, etc.) do not apply to outbound mail routing. So next up will be working on a new outbound smarthost role.

In T41785#4612211, @herron wrote:

Public DNS records have been created, but it looks like like reverse dns will have to wait until T199374 is resolved. This doesn't block the setup of the smarthosts themselves, but I think we'll want reverse DNS in place before directing any important mail flow through these hosts.

It's solved but @faidon's comment at T199374: Delegate public IP range for Eqiad1-r OpenStack deployment to designate (neutron) says to wait a couple of days before relying on it.

This is exciting to see materialize, it has been a long time coming :)

So, a few different things:

• This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?). I'm not sure if it makes sense, but my suggestion would also be to expand the scope and possibly name of this "project-smtp" project to cover all kinds of shared infrastructure needed for VPS instances and managed by the "core" admin team, e.g. labspuppetmasters, cumin masters, monitoring tools etc. in the future.
• The outgoing/incoming split is something that needs to happen in prod as well (T175362), so it's great that these goals align and we could share manifests for outgoing relays! I foresee some differences still (the wiki-mail stuff, DKIM, etc.), but we could potentially factor these out as well.
• I think in prod these we might go for the "mx-outNNNN" names to make it similar to the existing "mxNNNN" names, so it may make sense to align here? It doesn't matter that much though, and my apologies for the bikeshedding :)
• Once the provisioning of these new relays completes, we'll need to a) adjust realm.pp in Puppet to point to these instances (cf. the FIXME there) and b) delete the router ACL holes from WMCS->prod's mx1001/mx2001. I'm not very familiar with the Neutron project; would this need to be blocked on the Neutron migration, or is connectivity between the "old" and "new" networks possible (and not frowned upon)?

In T41785#4614434, @faidon wrote:

• This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?)

That's my understanding. It is easy to point a name directly under wmflabs.org (without project-smtp) at this.

In T41785#4614434, @faidon wrote:

I'm not sure if it makes sense, but my suggestion would also be to expand the scope and possibly name of this "project-smtp" project to cover all kinds of shared infrastructure needed for VPS instances and managed by the "core" admin team, e.g. labspuppetmasters, cumin masters, monitoring tools etc. in the future.

Possibly. I don't know how easy it is to rename tenants, to rename it's likely a case of replacing all the instances. 'project-smtp' is named after 'project-proxy' which contains the webproxy instances, and I wonder if the restricted bastions might fall under the same umbrella in some ways.

In T41785#4614434, @faidon wrote:

I'm not very familiar with the Neutron project; would this need to be blocked on the Neutron migration, or is connectivity between the "old" and "new" networks possible (and not frowned upon)?

Connectivity between the old and new labs networks appears to be working as a result of T202636 (I think it's pretty much required to do a smooth migration without widespread downtime), I'm not sure how willing people are to depend on it more than as necessary for the migration. There are/were some problems with it still.

In T41785#4614826, @Krenair wrote:

In T41785#4614434, @faidon wrote:

• This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?)

That's my understanding. It is easy to point a name directly under wmflabs.org (without project-smtp) at this.

Hopefully we can settle on a single record under wmflabs.org for each host. That would simplify configuration of components like let's encrypt. FQDN (or as close as possible) can be helpful when tracing/troubleshooting messages that have hopped through remote mail systems.

In T41785#4614434, @faidon wrote:

• I think in prod these we might go for the "mx-outNNNN" names to make it similar to the existing "mxNNNN" names, so it may make sense to align here?

Additionally in prod we could consider adding a layer of frontend MX service records to decouple our MX records from actual hostnames freeing us to do things like depool individual backend MX hosts for maintenance. Probably worth some further discussion, and thought about outbound service records too. E.g. will we create an outbound mail service record like smtp.wikimedia.org (analogous to smtp.gmail.com, smtp.fastmail.com, etc.)? And if so does the distinction then become mx (inbound) and smtp (outbound)? Maybe we could even drop the "out" and simplify to mxNNNN and smtpNNNN?

In T41785#4615256, @herron wrote:

In T41785#4614826, @Krenair wrote:

In T41785#4614434, @faidon wrote:

• This is of course completely up to the WMCS team, but I'd recommend not using a per-project domain name, but a global one, as these are going to be WMCS-wide relays (right?)

That's my understanding. It is easy to point a name directly under wmflabs.org (without project-smtp) at this.

Hopefully we can settle on a single record under wmflabs.org for each host. That would simplify configuration of components like let's encrypt. FQDN (or as close as possible) can be helpful when tracing/troubleshooting messages that have hopped through remote mail systems.

Let's create mx-out0[12].wmflabs.org records pointing at these hosts then?

I don't care a whole lot about having one infrastructure project or a bunch of small ones but it would be moderately easier to manage security policy if there's one big one. So, pursuant to that I've created a new project named 'cloudinfra' and added Keith to it. @herron, if you feel like it and it's not a ton of trouble you could rebuild those VMs there, and then we can delete the project-smtp project. We'll wind up with new IPs and all over there and then I can set up DNS. Adding foo.wmflabs.org records is easy but I'll need to do it.

Change 463143 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] WIP: smarthost: create mail smarthost role/profile

https://gerrit.wikimedia.org/r/463143

In T41785#4616409, @Andrew wrote:

rebuild those VMs there, and then we can delete the project-smtp project. We'll wind up with new IPs and all over there and then I can set up DNS.

Sounds good! New VMs, floating IPs and DNS records have been spun up, this time as mx-out0[12].cloudinfra.eqiad.wmflabs. Project-smtp can be torn down at earliest convenience.

And a quick update -- The WIP patch above is applied to mx-out01 via a standalone puppetmaster, and cursory testing shows this working in that mail sent through the smarthost is relayed off to a remote mail system. Of course more work and testing to do, but progressing.

I created public DNS:

Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out01.cloudinfra.wmflabs.org
185.15.56.18
Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out02.cloudinfra.wmflabs.org
185.15.56.19
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.

Is there anything else I can do to help with this?

I think we were going to do DNS directly under wmflabs.org, if the name is decided?

Yeah, we were, and I did, and then I pasted the wrong thing :)

Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out01.wmflabs.org
185.15.56.18
Andrews-MacBook-Pro-3:~ andrew$ dig +short mx-out02.wmflabs.org
185.15.56.19
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.
mx-out01.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
mx-out02.wmflabs.org.

In T41785#4622337, @Andrew wrote:

Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.
mx-out01.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
mx-out02.wmflabs.org.

Multiple PTRs are not wrong per se, but they're unusual and given the relationship between DNS and email reputation, you may run into trouble there. Or not, I've never experienced this kind of scenario before so I'm not sure what will happen :)

In T41785#4624771, @faidon wrote:

In T41785#4622337, @Andrew wrote:

Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.18
mx-out01.cloudinfra.wmflabs.org.
mx-out01.wmflabs.org.
Andrews-MacBook-Pro-3:~ andrew$ dig +short -x 185.15.56.19
mx-out02.cloudinfra.wmflabs.org.
mx-out02.wmflabs.org.

Multiple PTRs are not wrong per se, but they're unusual and given the relationship between DNS and email reputation, you may run into trouble there. Or not, I've never experienced this kind of scenario before so I'm not sure what will happen :)

PTRs get set up automatically. Can just remove the mx-out02.cloudinfra.wmflabs.org A record and it should soon drop the extra PTR.

Ok, I've deleted the A records mx-outNN.cloudinfra.wmflabs.org. FWIW was not intending to have two sets of records. Like I mentioned in T41785#4615256 going shorthand here will make it a bit less clear what VM/project these relate to when tracing a message through received headers or logs (requires more institutional knowledge). But that's not a big problem.

With this then I think we'll use mx-outNN.wmflabs.org in client configs (to keep certs simple), which means mail traffic will continue flowing through the various public outbound NAT IPs depending on the instance config. @Andrew to that end what subnets should be added to the security group, and relay_from_hosts to accept mail for relay from cloud hosts?

In T41785#4625330, @herron wrote:

mail traffic will continue flowing through the various public outbound NAT IPs depending on the instance config

Hm, I think labsaliaser isn't working as expected for eqiad1 hosts, but I'm not sure it matters as the old restriction against routing from a labs instance to a labs floating IP doesn't appear to apply to eqiad1 floating IPs?

@herron, the IP ranges for VMs are:

eqiad1: 172.16.0.0/21
eqiad: 10.68.16.0/21

We often just use 10.0.0.0/8 for eqiad since production is unable to connect directly anyway.

In T41785#4625386, @Andrew wrote:

@herron, the IP ranges for VMs are:

eqiad1: 172.16.0.0/21
eqiad: 10.68.16.0/21

We often just use 10.0.0.0/8 for eqiad since production is unable to connect directly anyway.

@Andrew Thanks, and then what about on the public side? Afaict we'll use the mx-outNN.wmflabs.org records in cloud/labs mail client configs and will need to whitelist the public ranges that this traffic will be NAT to.

In T41785#4625537, @herron wrote:

Afaict we'll use the mx-outNN.wmflabs.org records in cloud/labs mail client configs and will need to whitelist the public ranges that this traffic will be NAT to.

Is this right? From what I can tell there are at least two options for accepting mail from clients in cloud, each with pros/cons.

1. Clients send mail to mx-outNN.wmflabs.org (floating IP). This simplifies certificate config, with the downside of mail traffic from cloud hosts masquerading as outbound public NAT IP addresses like internal-server-nat.wmflabs.org (unless there is a workaround to have RFC1918 space route to/from the floating IPs without NAT)
2. Clients send mail to mx-outNN.cloudinfra.eqiad.wmflabs. Mx-out hosts will see private IP addresses this way (useful for incident response, rate limiting, etc.), with the downside of certificate complexity. We could use puppet exposed certs here, but with standalone puppetmasters ca certs/validation might get messy. I'm assuming we wouldn't want to put RFC1918 addresses in the wmflabs.org zone, but if we did that we should be able to use a LE cert with an extra SAN for that record.

I'm not familiar with WMCS' networking -- does this floating IP DNAT imply SNAT as well? i.e. would it not be possible to have flows of the form (RFC1918) -> mx-outNN.wmflabs.org (floating IP)?

In the current 'main' deployment there's a thing called labsaliaser which causes the DNS recursors to substitute responses containing labs public floating IPs with the equivalent private fixed IPs. It doesn't appear to be doing that for eqiad1 (it doesn't seem necessary as there isn't the old restriction around routing to floating IPs here), but I guess it could be fixed/enabled (the fact that it's not currently doing this against eqiad1 might be a side effect of the way both regions are currently set up and may magically work when the main deployment is gone) if it would help such cases? I think the outcome would be that they'd send to mx-outNN.wmflabs.org but the IP address they'd use for it would be internal.

In T41785#4634182, @Krenair wrote:

In the current 'main' deployment there's a thing called labsaliaser which causes the DNS recursors to substitute responses containing labs public floating IPs with the equivalent private fixed IPs.

More or less automatic split-brain DNS? Sounds like a workable approach. Is it on the roadmap for eqiad1?

In T41785#4634146, @faidon wrote:

would it not be possible to have flows of the form (RFC1918) -> mx-outNN.wmflabs.org (floating IP)?

Curious about this too. IMHO this would be preferable over split-brain DNS.

I suppose if either of these is on the roadmap (or can be added) we can move forward with using the dns names associated with the floating IPs (mx-outNN.wmflabs.org) in client configs. We would just need to enumerate and whitelist the NAT addresses being used today and plan on their removal in the future.

For what it's worth, the aliases in eqiad1 can be fixed by https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/464722/ or something like it.

labsaliaser is working now:

krenair@bastion-eqiad1-01:~$ dig eqiad1.bastion.wmflabs.org @8.8.8.8 +short
185.15.56.13
krenair@bastion-eqiad1-01:~$ dig eqiad1.bastion.wmflabs.org +short
172.16.1.136

krenair@bastion-eqiad1-01:~$ dig mx-out01.wmflabs.org @8.8.8.8 +short
185.15.56.18
krenair@bastion-eqiad1-01:~$ dig mx-out01.wmflabs.org +short
172.16.1.239

Change 463143 merged by Herron:
[operations/puppet@production] smarthost: create mail smarthost profile and wmcs smarthost role

https://gerrit.wikimedia.org/r/463143

Smarthosts mx-out01.wmflabs.org and mx-out02.wmflabs.org (wmcs instances within the cloudinfra project) are now configured using role::mail::smarthost::wmcs and cursory testing shows them relaying mail as expected.

I think we're ready to begin migrating some canaries over for final validation, and then update realm.pp. What could we cut over first as canaries?

shinken-01.shinken.eqiad.wmflabs might be a good test.

Yeah. Me and I think @valhallasw get fairly frequent emails from that host ^

Sounds good. Do you need any help from me to test? I don't have access to shinken-01.shinken.eqiad.wmflabs

I think we'd just need to disable puppet and change the exim config to use the new host. Although Andrew (?) is working on puppet there at the moment because it's broken.
Annoying thing about puppet on that instance is that it's responsible for running shinkengen, which is what adds monitoring for new instances and deletes it for old ones. :/

Sorry if this was discussed before but I'm seeing the following in messages sent through mx-out01:

Return-Path: <root@wmflabs.org>
From: root@gtirloni-puppetmaster-01.testlabs.eqiad.wmflabs

While if I sent through tools-mail.tools.eqiad.wmflabs, this is what I get:

Return-Path: <gtirloni@tools.wmflabs.org>
From: GTirloni <gtirloni@tools.wmflabs.org>

Chipping in because I'm not sure if @herron is aware: tools (i.e. Toolforge) has its own (very) special exim configuration. A comparison with a random non-tools VPS may be more appropriate, but even that may not be great -- I wouldn't assume it works correctly now :) What is the desired behavior of WMCS' mx-outs?

I think it depends too on the path the message takes. (i.e. was this sent directly via tools-mail, or was it first relayed through the local exim)

There is a rewrite performed by the local exim instance (https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/standard/templates/mail/exim4.minimal.labs.erb$65) that has this effect on messages that flow through it.

Fwiw mx-out is currently set to mimic what production does in terms of rewrites, though that can be easily adjusted here https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/manifests/mail/smarthost/wmcs.pp$15

Apples to apples comparison here should be mx1001.wikimedia.org vs mx-out01.wmflabs.org from "old" eqiad region since the production MXes are currently the smarthosts for labs (https://phabricator.wikimedia.org/source/operations-puppet/browse/production/manifests/realm.pp$259) and the networks associated with the new eqiad1 region are not permitted to relay through the prod MXes.

I think that makes perfect sense (mx1001 vs mx-out01). Let me just add, a recently provisioned Cloud VPS instance has exim4 configured to send through mx1001/mx2001 and mail sent through it behaves exactly like mx-out01 so it all looks fine. Sorry for the distraction. We just need to configure tools-mail to use mx-out01 in the future. I was a bit confused about scope, sorry.

Change 469522 had a related patch set uploaded (by Alex Monk; owner: Alex Monk):
[operations/puppet@production] exim smarthosts: Allow setting helo_data on transports

https://gerrit.wikimedia.org/r/469522

Change 469522 merged by Andrew Bogott:
[operations/puppet@production] exim smarthosts: Allow setting helo_data on transports

https://gerrit.wikimedia.org/r/469522

Change 470028 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] profile::mail::smarthost add primary_hostname setting

https://gerrit.wikimedia.org/r/470028

Change 470028 merged by Herron:
[operations/puppet@production] profile::mail::smarthost add primary_hostname setting

https://gerrit.wikimedia.org/r/470028

How are these working so far? Hopefully no news is good news! Is there anything that needs to be addressed before resolving this?

In T41785#4703228, @herron wrote:

How are these working so far? Hopefully no news is good news! Is there anything that needs to be addressed before resolving this?

Right now to use them properly (i.e. changing systems' mail settings to point at them) you need a puppetmaster where you can change stuff. Otherwise it's all hardcoded prod MXes from operations/puppet.git. I've made https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/469524/ to fix that.

Just to close the loop within this task https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/469524/ has been merged

It appears to be working. Tested in T207887: shinken: Use new mx-out* servers

So how do we want to roll this out? Do it on a per-project basis while moving a project across regions? Just flip the big switch in hieradata/labs.yaml?

In T41785#4704964, @Krenair wrote:

So how do we want to roll this out? Do it on a per-project basis while moving a project across regions? Just flip the big switch in hieradata/labs.yaml?

I vote for just switching everything at once. The change should be largely invisible to users unless something goes horribly wrong :)

Everything at once sounds good to me as well. Should we test/confirm connectivity from various other projects to the new smarthosts on tcp/25, or is it safe to assume that's allowed across the board?

In T41785#4706148, @herron wrote:

Everything at once sounds good to me as well. Should we test/confirm connectivity from various other projects to the new smarthosts on tcp/25, or is it safe to assume that's allowed across the board?

It should probably be fine unless people are using outbound security groups that restrict stuff like this. I'd check but I don't think security group information for any given project is available to people who aren't projectadmins.

Change 472720 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] wmcs: cut over to new smarthosts

https://gerrit.wikimedia.org/r/472720

Now that it looks like eqiad1-r migrations are under way I think we'll want to do the cut-over sooner rather than later.

Change 472720 merged by Herron:
[operations/puppet@production] wmcs: cut over to new smarthosts

https://gerrit.wikimedia.org/r/472720

Mentioned in SAL (#wikimedia-operations) [2018-11-13T20:22:58Z] <herron> updated labs realm smarthosts (via hiera) to mx-out0[12].wmflabs.org T41785