Page MenuHomePhabricator

Upgrade GNU Mailman from 2.1 to Mailman3
Open, MediumPublic

Assigned To
None
Authored By
AzaToth
Jul 6 2013, 2:48 PM
Tokens
"Love" token, awarded by xSavitar."Burninate" token, awarded by danshick-wmde."Like" token, awarded by AdHuikeshoven."Like" token, awarded by Asaf."100" token, awarded by Titodutta."Love" token, awarded by MusikAnimal."Like" token, awarded by Gaurav."Party Time" token, awarded by Kaartic."Love" token, awarded by Kizule."Like" token, awarded by Daimona."Like" token, awarded by sbassett."Orange Medal" token, awarded by Krinkle."Party Time" token, awarded by kolbert."Like" token, awarded by Dalba."Like" token, awarded by Sjoerddebruin."Like" token, awarded by Ladsgroup."Like" token, awarded by Pcoombe."Like" token, awarded by MarcoAurelio."Like" token, awarded by He7d3r."Like" token, awarded by MGChecker."Mountain of Wealth" token, awarded by Man77."Love" token, awarded by Steinsplitter."Like" token, awarded by Slaporte."The World Burns" token, awarded by Vituzzu."Like" token, awarded by Addshore."Love" token, awarded by greg."Like" token, awarded by dr0ptp4kt.

Description

As a Wikimedian I want to be able to follow discussions (read, reply, create) per project or theme in a convenient way, whether through email client on a (mobile) device, webmail on a (mobile) device, or through a web interface of the discussion system itself, so I'm up to date informed about what is going on and can join the conversation anytime as I like.

Alternatives to consider:

  • Keep Mailman 2.1
  • Migrate to Mailman 3.0 which has a new Django-based web user interface for end users and list administrators named Postorius (not yet officially sanctioned by GNU)
  • Consider Discourse as web interface for Mailman mailing list (requires development of synchronization)
  • Consider (flow enabled) talk pages on wiki - add support for reply by email to topics on a talk page for example

Not a user story (original task description)
We should update Mailman to version 3.

The new version, among others, stores hashed passwords, which could have minimized the impact of last weeks security incident

Details

Reference
bz50864

Related Objects

StatusSubtypeAssignedTask
OpenSecurityNone
OpenNone
OpenNone
OpenNone
OpenNone
ResolvedNone
ResolvedNone
ResolvedLadsgroup
ResolvedMarostegui
ResolvedNone
ResolvedLegoktm
DeclinedLadsgroup
Resolvedbd808
ResolvedLadsgroup
Resolvedbd808
ResolvedLegoktm
OpenNone
ResolvedDzahn
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedLadsgroup
ResolvedLegoktm
ResolvedLadsgroup
ResolvedMarostegui
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Mentioned in SAL (#wikimedia-releng) [2020-06-06T15:19:54Z] <Amir1> created deployment-mailman01 in beta cluster for testing upgrade to mailmain 3.1 (T52864 and T130554)

Ladsgroup changed the task status from Stalled to Open.Jun 6 2020, 4:39 PM
Ladsgroup added a subscriber: Ladsgroup.

The official support for upgrading from mailman2 to mailman3 is there. You can see it in https://docs.mailman3.org/en/latest/migration.html

The part I like about the upgrade is that it can be done mailing list by mailing list. It even has a configuration migration script.
Some things to consider here:

URLs to archived messages will break, unless you take extra steps to keep them around. Upgrade mechanism makes sure to import all your archived messages in the new system, but, all the URLs to the new messages are going to be different.

This can be slightly problematic but still there's a way to fix it (just keep the archives but make them read-only)

This is also an example mailman3 installation you can see to get how it feels: https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

So my suggestion on upgrade path:

I'm trying to install it on beta cluster, right now I don't know how to hook it into beta cluster's MTA. I post a link once it's ready.

If anyone from SRE is willing to help with this, I'd be more than happy to help this move forward. This is the only place in production where the password is stored as plain text.

I started a simple one that half works: https://lists-beta.wmflabs.org/

The sending part of MTA integration works out of the box, you will receive an email to confirm your email address but the receive part doesn't work in beta cluster yet. I haven't managed to get the DNS record for it working because horizon just gives me an error every time I try to set an MX record and associate it with 185.15.56.7 (even tried with lists.beta.wmflabs.org and still didn't work).

MX records cannot have IP addresses. They must be associated to a hostname (plus a priority)

Or alternatively, just giving an A record, it should work through that legacy fallback. So a message to test-l@instance-deployment-mailman01.deployment-prep.wmflabs.org should arrive to the right box.

Note: The receiving Exim doesn't seem to be configured to accept list mail:

RCPT TO:<test-l@lists-beta.wmflabs.org>
550 Administrative prohibition

MX records cannot have IP addresses. They must be associated to a hostname (plus a priority)

Or alternatively, just giving an A record, it should work through that legacy fallback. So a message to test-l@instance-deployment-mailman01.deployment-prep.wmflabs.org should arrive to the right box.

Thanks. I got it to some degree work:

amsa@amsa-Latitude-7480:~$ dig lists.beta.wmflabs.org

; <<>> DiG 9.11.5-P4-5.1ubuntu2.1-Ubuntu <<>> lists.beta.wmflabs.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14976
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;lists.beta.wmflabs.org.		IN	A

;; ANSWER SECTION:
lists.beta.wmflabs.org.	2876	IN	A	185.15.56.7

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Jun 07 04:15:11 CEST 2020
;; MSG SIZE  rcvd: 67

amsa@amsa-Latitude-7480:~$ telnet lists.beta.wmflabs.org 25
Trying 185.15.56.7...

I don't know why all of incoming traffic just go to blackhole (also on port 80 which it should have worked as lists-beta.wmflabs.org works)

I went to have a look but both security groups and iptables on the box looked fine and exim was listening on that port, then realised it works for me anyway, I can connect to it:

alex@alex-laptop:~$ telnet lists.beta.wmflabs.org 25
Trying 185.15.56.7...
Connected to lists.beta.wmflabs.org.
Escape character is '^]'.
220 deployment-mailman01.deployment-prep.eqiad.wmflabs ESMTP Exim 4.92 Sun, 07 Jun 2020 02:38:38 +0000

Unless that's been fixed in the past 25 minutes, maybe your ISP is blocking you from connecting out to port 25?

The interface looks awesome already. I went on and tried to subscribe to test-l but that didn't work. There was no confirmation URL and had to be done by email so I'm not sure this was prevented by T194032.

The interface looks awesome already. I went on and tried to subscribe to test-l but that didn't work. There was no confirmation URL and had to be done by email so I'm not sure this was prevented by T194032.

I can see in the logs that it went to mx-out, Can you check your spam folder?

As the list owner I can see it requires confirmation for joining a mailing list:

(Maybe it immediately accepts users whose their email is confirmed?)

I still can't make LMTP to work incoming emails :(((

It doesn't automatically accept; it requires confirmation by email. However, the confirmation gets bounced with "550 Administrative prohibition". So you need to get incoming mail working.

It doesn't automatically accept; it requires confirmation by email. However, the confirmation gets bounced with "550 Administrative prohibition". So you need to get incoming mail working.

Okay after a couple of hours of wrestling. I reworked the ACL rules so it accepts them. It now responds with another error. Probably can't route the emails to mailman. I will pick it later (also it should be in another ticket, I should stop spamming 60 people)

It works now, you can try it in https://lists-beta.wmflabs.org

I haven't managed to get the archive working but you can now join mailing lists and send mail!

I made the archiver work and you can now see it: https://lists-beta.wmflabs.org/hyperkitty/list/test-high-volume@lists.beta.wmflabs.org/thread/TOFSYCOMTGUWZPXNZGGIK3TBRCYAKAQJ/

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

In T52864#6242301, @Tgr wrote:

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

Having that in production probably requires a bigger discussion (like with security and SRE and legal). For the beta cluster instance, that's a piece of cake.

Ladsgroup renamed this task from Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 to Upgrade GNU Mailman from 2.1 to 3.3.Jun 27 2020, 3:35 PM

After talking to @herron we decided that we start the upgrade (slowly) and hopefully we will get it deployed and upgrade in a couple of months (maybe a year). I do this in my volunteer capacity so please be kind to me. I start by creating subtasks.

Ladsgroup renamed this task from Upgrade GNU Mailman from 2.1 to 3.3 to Upgrade GNU Mailman from 2.1 to Mailman3.Aug 8 2020, 11:51 PM

In case you missed the announcement, a test server with mailman3 is now available: https://lists.wikimedia.org/pipermail/wikitech-l/2021-March/094382.html

Thanks for taking care of this hard task! I hope we can then get rid of all Google Groups. ;-) (Is there a task for importing these already?)

Swap URLs of lists-next.wikimedia.org to lists.wikimedia.org and lists.wikimedia.org to lists-old.wikimedia.org (make sure archive URLs don't break)

So what's the plan here? Redirect all requests to (public) pipermail-style URLs from lists.wikimedia.org to a static copy on lists-old.wikimedia.org after the switch? (Remember not to regenerate the archives from the mbox, or the new archives will have different IDs.)

Redirecting requests to /pipermail should do most of the job but I'm not sure what to do with the mbox files and the private list archives currently served under the /private path. Preserving links to private archives would be nice, as it will otherwise be a pain to find the equivalents in the new archives. However a simple static mirror cannot handle authentication and translating from old URLs to hyperkitty URLs seems hopeless (unless we have some idea how those thread and message IDs are created).

Swap URLs of lists-next.wikimedia.org to lists.wikimedia.org and lists.wikimedia.org to lists-old.wikimedia.org (make sure archive URLs don't break)

Note that this is no longer the plan. In discussion on T256539: Figure out a way to sync old and new mailman the plan is to serve both mailman2 and mailman3 from the same server, so we can gradually migrate lists individually. This is tracked as T278610: Install mailman3 on lists1001.wikimedia.org. As mentioned in the announcement, lists-next is a temporary testing ground and will be fully deleted once we're satisfied with the process.

Redirect all requests to (public) pipermail-style URLs from lists.wikimedia.org to a static copy on lists-old.wikimedia.org after the switch? (Remember not to regenerate the archives from the mbox, or the new archives will have different IDs.)

In theory hyperkitty actually supports redirecting pipermail URLs (https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/views/compat.py) but we need to continue testing the migration process to see whether it's good enough. FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer updated.

Redirecting requests to /pipermail should do most of the job but I'm not sure what to do with the mbox files and the private list archives currently served under the /private path. Preserving links to private archives would be nice, as it will otherwise be a pain to find the equivalents in the new archives. However a simple static mirror cannot handle authentication and translating from old URLs to hyperkitty URLs seems hopeless (unless we have some idea how those thread and message IDs are created).

We cannot continue to support /private URLs because as we're getting rid of the mailman2 authentication system. So either a redirector will need to be in place or we break links. Note that mailman3 has fulltext search, which hopefully will help finding old mails. If you have the Message-ID, you can use the hyperkitty API to get the new archive URL: T256539#6456966.

the plan is to serve both mailman2 and mailman3 from the same server [...] FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer update

So the public HTML and txt.gz archives will keep being served indefinitely from the same URLs as before? That will reduce the pain considerably.

the plan is to serve both mailman2 and mailman3 from the same server [...] FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer update

So the public HTML and txt.gz archives will keep being served indefinitely from the same URLs as before? That will reduce the pain considerably.

I mostly figured out how to redirect public HTML URLs, see T280731: Implement static redirects from pipermail archives to hyperkitty archives. Should get deployed tomorrow or early next week.

I don't think hyperkitty has an equivalent for the txt.gz files, so we can keep them around (by not deleting them) if there's value.

hyperkitty allows you to download gzip files (click on "Download" in https://lists.wikimedia.org/hyperkitty/list/listadmins@lists.wikimedia.org/2021/5/) Thankfully, building a redirect for it would be pretty easy. Doesn't need any hash.