Mailman3 is not as stateful as mailman2 but still there are things that are stateful, like search index is still on the VM. We can't do it unless we give it to a dedicated search cluster. exim4 is also complex :(. We definitely should do it but maybe not in the upgrade?

Indeed! Thanks for the task.
The IP behind lists.wikimedia.org is tied to eqiad row A, which brings many limitation in term of HA for the service (a row is a failure domain, so services should be designed across rows) and best practices in general (standardizing how public services are accessed, reducing tech-debt).
A few services grew organically that way, and we're trying to reduce them. Now seems a good time ;)

I'd like to gather the requirements and understand what the blockers are first.

What I remember from past conversations (and looking around with my little mailman knowledge):

• There is currently no HA, mailman runs on a single VM
• Does mailman need to make outbound connection to send mails (so it needs a public IP)? or does it relays them off mx1001?
• If the above is correct, there are concerns about the IP reputation the emails are sent from, but this IP did change multiple times previously (last time 3 years ago) and could have to change again for the same reasons

The cleanest way to have a stable IP is to "go behind LVS" where lists.w.o is a LVS service IP, and the LVS forwards traffic to the real server. Even if there is only 1 real server it would be a win. That realserver could stay in the public vlan but be more carefully firewalled off, or go in the private vlan if it doesn't need direct outbound connections.
That way whatever happens to the real server, the VIP would stay fixed. Daemons on the realservear could bind on the VIP as well to make outbound connections.
Easiest HA is an active/passive setup, failover done by pointing the LVS to the healthy server (only one at a time). More complex setup like active/active would depends on the mailman software stack.

In T278495#6949180, @ayounsi wrote:

Indeed! Thanks for the task.
The IP behind lists.wikimedia.org is tied to eqiad row A, which brings many limitation in term of HA for the service (a row is a failure domain, so services should be designed across rows) and best practices in general (standardizing how public services are accessed, reducing tech-debt).
A few services grew organically that way, and we're trying to reduce them. Now seems a good time ;)

I'd like to gather the requirements and understand what the blockers are first.

What I remember from past conversations (and looking around with my little mailman knowledge):

• There is currently no HA, mailman runs on a single VM

Yup

• Does mailman need to make outbound connection to send mails (so it needs a public IP)? or does it relays them off mx1001?

It doesn't relay mail to mx1001 nor receive mails from it. Both mailman2 and mailman3 have their own exim4 process and configuration (same with otrs and some other services)

• If the above is correct, there are concerns about the IP reputation the emails are sent from, but this IP did change multiple times previously (last time 3 years ago) and could have to change again for the same reasons

I honestly wouldn't worry too much about IP reputation as I hope they have our range and not our single IP. Plus lists-next.wikimedia.org is already on a completely new IP and not many emails from it got warnings (none got warnings) or went to spam (not more than the old mailman) in my commercial service (Google) so I think we will be fine.

The cleanest way to have a stable IP is to "go behind LVS" where lists.w.o is a LVS service IP, and the LVS forwards traffic to the real server. Even if there is only 1 real server it would be a win. That realserver could stay in the public vlan but be more carefully firewalled off, or go in the private vlan if it doesn't need direct outbound connections.

We can have drifting DNS records, one for A/AAAA and one for MX. I know it wouldn't be ideal but at least for now, the http traffic would go through LVS

Oh, I forgot to mention the old mailman has a standby host (look at profile::lists::primary_host and profile::lists::standby_host hiera values). I like to have proper HA for the new mailman3 but I'm a little hesitant to treat it in a tier higher than phabricator or gerrit.

And another note (I'm sorry, I keep remembering things). The way we are planning to do the migration is to actually have both mailman2 and 3 on the same host: T256539: Figure out a way to sync old and new mailman so I think this is not doable until we are fully done with migrating all mailing lists.

It's also worth pointing out that HTTPS, and inbound/outbound SMTP traffic for lists.wikimedia.org can each take different paths while using the same subdomain since inbound mail is going to be routed according to the MX records and outbound via the smtp config in mailman. We wouldn't necessarily need LVS in the path for mail routing in either direction.

In a setup where the lists server doesn't have a public IP itself I think it makes sense to look at inbound and outbound mail independently (today it's simplified since the host has a public IP and sends/receives directly)

Inbound mail -- An option here would be to use the mx[12]001 as the MX records for lists.wikimedia.org, and configure the MXes to relay lists.wikimedia.org messages onwards to the lists hosts for processing. That would offload inbound SMTP high availability and message queueing from the lists server(s), at the expense of some additional complexity in the MX config.

Outbound mail -- IMO lists mail should continue to be kept separate in terms of IP addresses used from production and wiki mail. An option would be to bind a new set of lists.wikimedia.org outbound interfaces to the prod MXes and handle it much like wiki mail, or alternatively deploy a production outbound MX cluster with dedicated IPs for lists. The latter of course being a larger effort.

In T278495#6947777, @Ladsgroup wrote:

Mailman3 is not as stateful as mailman2 but still there are things that are stateful, like search index is still on the VM. We can't do it unless we give it to a dedicated search cluster.

Ah, that's right.

exim4 is also complex :(. We definitely should do it but maybe not in the upgrade?

Yeah, this is mostly figuring out what the requirements/recommendations from netops/infra is and then once we have a plan, we can then figure out when and how to get there.

In T278495#6949180, @ayounsi wrote:

Indeed! Thanks for the task.
The IP behind lists.wikimedia.org is tied to eqiad row A, which brings many limitation in term of HA for the service (a row is a failure domain, so services should be designed across rows) and best practices in general (standardizing how public services are accessed, reducing tech-debt).
A few services grew organically that way, and we're trying to reduce them. Now seems a good time ;)

I'd like to gather the requirements and understand what the blockers are first.

What I remember from past conversations (and looking around with my little mailman knowledge):

• There is currently no HA, mailman runs on a single VM

Correct. This is mostly fixable in mailman3, though we'll need to figure out details about stuff like search.

• Does mailman need to make outbound connection to send mails (so it needs a public IP)? or does it relays them off mx1001?

Currently mailman handles all mail by itself, no relays. Port 25 (exim) is open for incoming mail, and it sends mail by itself too. I do not know if this is a hard requirement or not. In the past mailman has been attacked through subscription requests, clogging up all list mail. It may be useful to keep it isolated from the main mail relays for that reason.

mailman also serves HTTP/HTTPS but that's easy to move behind LVS.

• If the above is correct, there are concerns about the IP reputation the emails are sent from, but this IP did change multiple times previously (last time 3 years ago) and could have to change again for the same reasons

For lists-next, which used a new IP: some initial emails were greylisted (for debian.org at least, with a link to https://postgrey.schweikert.ch/help/debian.org.html), but that seems to have cleared up now...and the logs got rotated unfortunately. Also at least one report of mails ending up in spam: T278434#6947325.

So I think if we have to switch IPs it's not the end of the world but the more we can avoid it the better.

The cleanest way to have a stable IP is to "go behind LVS" where lists.w.o is a LVS service IP, and the LVS forwards traffic to the real server. Even if there is only 1 real server it would be a win. That realserver could stay in the public vlan but be more carefully firewalled off, or go in the private vlan if it doesn't need direct outbound connections.

Ack, I think the main question is then how does this work with exim - can we run that behind LVS? Is that something we already do?

I made T279296 to illustrate the best path about services IPs.
I don't know about Exim specifically, but any UDP or TCP service can go behind LVS, even with a single back-end.
What has not been done previously (AFAIK) is to have the server make outbound connection from the VIP (behind LVS).
Which shouldn't be an issue as long as:
1/ there is only 1 back-end node active at a time
2/ the the daemon making outbound connection binds in the VIP (itself on the loopback).