Page MenuHomePhabricator

Upgrade GNU Mailman from 2.1 to Mailman3
Open, MediumPublic

Assigned To
None
Authored By
AzaToth
Jul 6 2013, 2:48 PM
Tokens
"Like" token, awarded by Asaf."100" token, awarded by Titodutta."Love" token, awarded by MusikAnimal."Like" token, awarded by Gaurav."Party Time" token, awarded by Kaartic."Love" token, awarded by Kizule."Like" token, awarded by Daimona."Like" token, awarded by sbassett."Orange Medal" token, awarded by Krinkle."Party Time" token, awarded by kolbert."Like" token, awarded by Dalba."Like" token, awarded by Sjoerddebruin."Like" token, awarded by Ladsgroup."Like" token, awarded by Pcoombe."Like" token, awarded by MarcoAurelio."Like" token, awarded by He7d3r."Like" token, awarded by MGChecker."Mountain of Wealth" token, awarded by Man77."Love" token, awarded by Steinsplitter."Like" token, awarded by Slaporte."The World Burns" token, awarded by Vituzzu."Like" token, awarded by Addshore."Love" token, awarded by greg."Like" token, awarded by dr0ptp4kt.

Description

As a Wikimedian I want to be able to follow discussions (read, reply, create) per project or theme in a convenient way, whether through email client on a (mobile) device, webmail on a (mobile) device, or through a web interface of the discussion system itself, so I'm up to date informed about what is going on and can join the conversation anytime as I like.

Alternatives to consider:

  • Keep Mailman 2.1
  • Migrate to Mailman 3.0 which has a new Django-based web user interface for end users and list administrators named Postorius (not yet officially sanctioned by GNU)
  • Consider Discourse as web interface for Mailman mailing list (requires development of synchronization)
  • Consider (flow enabled) talk pages on wiki - add support for reply by email to topics on a talk page for example

Not a user story (original task description)
We should update Mailman to version 3.

The new version, among others, stores hashed passwords, which could have minimized the impact of last weeks security incident

Details

Reference
bz50864

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: alanajjar. · View Herald TranscriptJan 26 2018, 9:42 PM

mailman3-core, mailman3-hyperkitty, postorius and mailmanclient have been accepted into stretch-backports today.

Does that mean we can start considering our migration?

That's dependent on goal planning / road map considerations, I only meant to point out the availability in backports since it was mentioned earlier on this task.

mailman3-core, mailman3-hyperkitty, postorius and mailmanclient have been accepted into stretch-backports today.

@Legoktm @herron where does this put us now then? Thanks.

I think we need to lobby/convince/remind @faidon and other roadmap deciders to allocate resources for this :)

This comment was removed by MarcoAurelio.
Reedy updated the task description. (Show Details)May 11 2018, 6:24 PM
Reedy removed subscribers: wikibugs-l-list, JohnLewis.

I think we need to lobby/convince/remind @faidon and other roadmap deciders to allocate resources for this :)

Hello @faidon. Do you think we can move on this? Thanks.

Hi. Any status updates here? Thanks.

Hi. Any status updates here? Thanks.

If you are in a hurry to switch to mailman 3, maybe you could talk with the development team about increasing its translations as I suggested?

Re https://lists.wikimedia.org/pipermail/wikimedia-l/2017-August/088350.html , I think adding Mailman 3 to translatewiki.net would a reasonable first step to increase awareness and help its development. Erik, are you willing to open this discussion in the relevant venues?

I am not in any hurry. I am just requesting an status update.

Meno25 removed a subscriber: Meno25.Nov 23 2018, 7:48 AM
RuyP added a subscriber: RuyP.Jan 17 2019, 7:45 PM
Stryn added a subscriber: Stryn.Jan 17 2019, 7:56 PM
Base added a subscriber: Base.Mar 6 2019, 8:22 PM
Masti added a subscriber: Masti.Mar 6 2019, 8:40 PM

@Legoktm @herron Could we have an update on this one, please? Thanks.

Presumably it’s the usual no activity == no update...

Tgr added a subscriber: Qgil.Mar 13 2019, 10:14 PM

There has been a lot of activity on Discourse, OTOH. @Qgil might be able to say more on that.

@Reedy As far as I could see, T52864#3241375 is resolved, and T52864#3940724 is as well. GNU seems happy with Postorius and HyperKitty as well (http://list.org/features.html). Therefore, is there any blockers that prevent this to go forward?

@Reedy As far as I could see, T52864#3241375 is resolved, and T52864#3940724 is as well. GNU seems happy with Postorius and HyperKitty as well (http://list.org/features.html). Therefore, is there any blockers that prevent this to go forward?

Potentially whatever host mailman is running on may need an os upgrade, or a reinstall... which may potentially mean new hardware depending on warranty status

Then someone to do test migrations, before upgrading the live versions. I don’t know if SRE has this in their goals/timelines

Qgil added a comment.Mar 14 2019, 10:40 AM
In T52864#5022889, @Tgr wrote:

There has been a lot of activity on Discourse, OTOH. @Qgil might be able to say more on that.

A few weeks ago I set up two scenarios to test:

So far so good, although more testing and feedback is definitely welcome. There seem to be some corner cases / bugs related to identifying signatures and attachments, but the Discourse team has been very responsive.

I am trying to build a case around Discourse that includes mailing lists / email users, but as you know or may imagine there are many pieces involved. I believe Discourse may provide a better end result for email and web users, integrating forum and mailing lists under a common umbrella (same usernames, common search...).

This is just a quick update. I need a bit more time to prepare a decent proposal for discussion.

Qgil added a comment.Jun 27 2019, 3:37 PM

Sorry, it took more time than I expected when I posted the comment above, but here it is: T226727: Integrate mailing lists in Wikimedia Space & https://discuss-space.wmflabs.org/t/integrating-mailing-lists-to-wikimedia-space/136

How would you prefer to have this conversation about migrating from GNU Mailman 2.1 to... ?

I am well aware that a plan for a potential migration from Mailman to Discourse would be complex and require many details defined that today are vague or unknown. I just want to say that we (the team working on Wikimedia Space) are ready to take part in this conversation and assume our share in the work required to proceed with this migration.

@Qgil Thanks for your comment. As a user of some mailing lists, I am still interesting in upgrading to Mailman 3+. At T52864#5022944 we have some potential issues. I guess that, to be totally sure, we could create a subtask and check what really needs to be done to move this forward?

Qgil added a comment.Jun 27 2019, 11:31 PM

To be clear, the proposal is that users of a Mailing mailing list about X could keep using the same email features subscribing in mailing list mode to a Discourse category about X (same topic, same email functionality). The hypothesis is that Discourse can roughly cover the functionality that Mailman 3+ provides.

bd808 added a subscriber: bd808.Nov 9 2019, 9:53 PM

I briefly talked with @herron about this today. I think we are still blocked on the lack of Debian packaging. Looking at https://qa.debian.org/developer.php?email=pkg-mailman-hackers%40lists.alioth.debian.org, none of the packages were included in stretch (and some aren't in testing yet), so we'd either need stretch backports or wait until the buster release.

Packaging for Mailman 3.1 is now in Buster (and Stretch backports): https://packages.debian.org/buster/mailman3

I think that current ganetti vm that is hosting lists.wikimedia.org is running Debian Jessie{{cn}}. If it is, our SRE folks will likely want to replace that vm with one using Buster soon. It would be more work, but appreciated by many if that rebuild included the upgrade from mailman2 to mailman3.

chasemp moved this task from Incoming to Watching on the Security-Team board.

Mentioned in SAL (#wikimedia-releng) [2020-06-06T15:19:54Z] <Amir1> created deployment-mailman01 in beta cluster for testing upgrade to mailmain 3.1 (T52864 and T130554)

Ladsgroup changed the task status from Stalled to Open.Jun 6 2020, 4:39 PM
Ladsgroup added a subscriber: Ladsgroup.

The official support for upgrading from mailman2 to mailman3 is there. You can see it in https://docs.mailman3.org/en/latest/migration.html

The part I like about the upgrade is that it can be done mailing list by mailing list. It even has a configuration migration script.
Some things to consider here:

URLs to archived messages will break, unless you take extra steps to keep them around. Upgrade mechanism makes sure to import all your archived messages in the new system, but, all the URLs to the new messages are going to be different.

This can be slightly problematic but still there's a way to fix it (just keep the archives but make them read-only)

This is also an example mailman3 installation you can see to get how it feels: https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/

So my suggestion on upgrade path:

I'm trying to install it on beta cluster, right now I don't know how to hook it into beta cluster's MTA. I post a link once it's ready.

If anyone from SRE is willing to help with this, I'd be more than happy to help this move forward. This is the only place in production where the password is stored as plain text.

Dalba awarded a token.Jun 6 2020, 5:33 PM
kolbert added a subscriber: kolbert.

I started a simple one that half works: https://lists-beta.wmflabs.org/

The sending part of MTA integration works out of the box, you will receive an email to confirm your email address but the receive part doesn't work in beta cluster yet. I haven't managed to get the DNS record for it working because horizon just gives me an error every time I try to set an MX record and associate it with 185.15.56.7 (even tried with lists.beta.wmflabs.org and still didn't work).

MX records cannot have IP addresses. They must be associated to a hostname (plus a priority)

Or alternatively, just giving an A record, it should work through that legacy fallback. So a message to test-l@instance-deployment-mailman01.deployment-prep.wmflabs.org should arrive to the right box.

Note: The receiving Exim doesn't seem to be configured to accept list mail:

RCPT TO:<test-l@lists-beta.wmflabs.org>
550 Administrative prohibition

MX records cannot have IP addresses. They must be associated to a hostname (plus a priority)

Or alternatively, just giving an A record, it should work through that legacy fallback. So a message to test-l@instance-deployment-mailman01.deployment-prep.wmflabs.org should arrive to the right box.

Thanks. I got it to some degree work:

amsa@amsa-Latitude-7480:~$ dig lists.beta.wmflabs.org

; <<>> DiG 9.11.5-P4-5.1ubuntu2.1-Ubuntu <<>> lists.beta.wmflabs.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14976
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;lists.beta.wmflabs.org.		IN	A

;; ANSWER SECTION:
lists.beta.wmflabs.org.	2876	IN	A	185.15.56.7

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sun Jun 07 04:15:11 CEST 2020
;; MSG SIZE  rcvd: 67

amsa@amsa-Latitude-7480:~$ telnet lists.beta.wmflabs.org 25
Trying 185.15.56.7...

I don't know why all of incoming traffic just go to blackhole (also on port 80 which it should have worked as lists-beta.wmflabs.org works)

Krenair added a comment.EditedJun 7 2020, 2:39 AM

I went to have a look but both security groups and iptables on the box looked fine and exim was listening on that port, then realised it works for me anyway, I can connect to it:

alex@alex-laptop:~$ telnet lists.beta.wmflabs.org 25
Trying 185.15.56.7...
Connected to lists.beta.wmflabs.org.
Escape character is '^]'.
220 deployment-mailman01.deployment-prep.eqiad.wmflabs ESMTP Exim 4.92 Sun, 07 Jun 2020 02:38:38 +0000

Unless that's been fixed in the past 25 minutes, maybe your ISP is blocking you from connecting out to port 25?

The interface looks awesome already. I went on and tried to subscribe to test-l but that didn't work. There was no confirmation URL and had to be done by email so I'm not sure this was prevented by T194032.

The interface looks awesome already. I went on and tried to subscribe to test-l but that didn't work. There was no confirmation URL and had to be done by email so I'm not sure this was prevented by T194032.

I can see in the logs that it went to mx-out, Can you check your spam folder?

As the list owner I can see it requires confirmation for joining a mailing list:

(Maybe it immediately accepts users whose their email is confirmed?)

I still can't make LMTP to work incoming emails :(((

It doesn't automatically accept; it requires confirmation by email. However, the confirmation gets bounced with "550 Administrative prohibition". So you need to get incoming mail working.

Majavah added a subscriber: Majavah.Jun 7 2020, 3:23 PM

It doesn't automatically accept; it requires confirmation by email. However, the confirmation gets bounced with "550 Administrative prohibition". So you need to get incoming mail working.

Okay after a couple of hours of wrestling. I reworked the ACL rules so it accepts them. It now responds with another error. Probably can't route the emails to mailman. I will pick it later (also it should be in another ticket, I should stop spamming 60 people)

It works now, you can try it in https://lists-beta.wmflabs.org

I haven't managed to get the archive working but you can now join mailing lists and send mail!

I made the archiver work and you can now see it: https://lists-beta.wmflabs.org/hyperkitty/list/test-high-volume@lists.beta.wmflabs.org/thread/TOFSYCOMTGUWZPXNZGGIK3TBRCYAKAQJ/

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

Tgr added a comment.Jun 20 2020, 10:20 PM

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

In T52864#6242301, @Tgr wrote:

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

Having that in production probably requires a bigger discussion (like with security and SRE and legal). For the beta cluster instance, that's a piece of cake.

Ladsgroup renamed this task from Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 to Upgrade GNU Mailman from 2.1 to 3.3.Jun 27 2020, 3:35 PM

After talking to @herron we decided that we start the upgrade (slowly) and hopefully we will get it deployed and upgrade in a couple of months (maybe a year). I do this in my volunteer capacity so please be kind to me. I start by creating subtasks.

Izno added a subscriber: Izno.Jun 27 2020, 4:28 PM
Ladsgroup renamed this task from Upgrade GNU Mailman from 2.1 to 3.3 to Upgrade GNU Mailman from 2.1 to Mailman3.Aug 8 2020, 11:51 PM
Jony added a subscriber: Jony.Aug 9 2020, 6:12 AM
Krd added a subscriber: Krd.Aug 9 2020, 7:21 AM
Gaurav awarded a token.Aug 9 2020, 5:26 PM
Gaurav added a subscriber: Gaurav.
Asaf awarded a token.Sep 16 2020, 8:08 PM
Meno25 added a subscriber: Meno25.Oct 18 2020, 10:20 PM