Page MenuHomePhabricator

Upgrade GNU Mailman from 2.1 to Mailman3
Closed, ResolvedPublic

Assigned To
None
Authored By
AzaToth
Jul 6 2013, 2:48 PM
Referenced Files
F31857215: image.png
Jun 7 2020, 2:43 PM
Tokens
"Mountain of Wealth" token, awarded by Nemo_bis."Barnstar" token, awarded by Quiddity."Barnstar" token, awarded by R4356th."Party Time" token, awarded by Asartea."Barnstar" token, awarded by Zblace."Love" token, awarded by xSavitar."Burninate" token, awarded by danshick-wmde."Like" token, awarded by AdHuikeshoven."Like" token, awarded by Asaf."100" token, awarded by Titodutta."Love" token, awarded by MusikAnimal."Like" token, awarded by Gaurav."Party Time" token, awarded by Kaartic."Love" token, awarded by Kizule."Like" token, awarded by Daimona."Like" token, awarded by sbassett."Orange Medal" token, awarded by Krinkle."Party Time" token, awarded by kolbert."Like" token, awarded by Dalba."Like" token, awarded by Sjoerddebruin."Like" token, awarded by Ladsgroup."Like" token, awarded by Pcoombe."Like" token, awarded by MarcoAurelio."Like" token, awarded by He7d3r."Like" token, awarded by MGChecker."Mountain of Wealth" token, awarded by Man77."Love" token, awarded by Steinsplitter."Like" token, awarded by Slaporte."The World Burns" token, awarded by Vituzzu."Like" token, awarded by Addshore."Love" token, awarded by greg."Like" token, awarded by dr0ptp4kt.

Description

As a Wikimedian I want to be able to follow discussions (read, reply, create) per project or theme in a convenient way, whether through email client on a (mobile) device, webmail on a (mobile) device, or through a web interface of the discussion system itself, so I'm up to date informed about what is going on and can join the conversation anytime as I like.

Alternatives to consider:

  • Keep Mailman 2.1
  • Migrate to Mailman 3.0 which has a new Django-based web user interface for end users and list administrators named Postorius (not yet officially sanctioned by GNU)
  • Consider Discourse as web interface for Mailman mailing list (requires development of synchronization)
  • Consider (flow enabled) talk pages on wiki - add support for reply by email to topics on a talk page for example

Not a user story (original task description)
We should update Mailman to version 3.

The new version, among others, stores hashed passwords, which could have minimized the impact of last weeks security incident

Related Objects

StatusSubtypeAssignedTask
ResolvedSecurityNone
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedNone
ResolvedLadsgroup
ResolvedMarostegui
ResolvedNone
ResolvedLegoktm
DeclinedLadsgroup
Resolvedbd808
ResolvedLadsgroup
Resolvedbd808
ResolvedLegoktm
OpenNone
ResolvedDzahn
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedLegoktm
ResolvedLadsgroup
ResolvedLegoktm
ResolvedLadsgroup
ResolvedMarostegui
ResolvedLadsgroup
DuplicateNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

It works now, you can try it in https://lists-beta.wmflabs.org

I haven't managed to get the archive working but you can now join mailing lists and send mail!

I made the archiver work and you can now see it: https://lists-beta.wmflabs.org/hyperkitty/list/test-high-volume@lists.beta.wmflabs.org/thread/TOFSYCOMTGUWZPXNZGGIK3TBRCYAKAQJ/

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

In T52864#6242301, @Tgr wrote:

The only thing is that with disabling gravatar (which we can't enable due to our privacy policy), the profile pictures look weird. I filed a bug against hyperkitty about this: https://gitlab.com/mailman/hyperkitty/-/issues/303 let's see how it goes.

It might be simpler (not to mention more user-friendly) to set up something like https://github.com/ThomasLeister/gravatar-privacy-proxy.

Having that in production probably requires a bigger discussion (like with security and SRE and legal). For the beta cluster instance, that's a piece of cake.

Ladsgroup renamed this task from Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 to Upgrade GNU Mailman from 2.1 to 3.3.Jun 27 2020, 3:35 PM

After talking to @herron we decided that we start the upgrade (slowly) and hopefully we will get it deployed and upgrade in a couple of months (maybe a year). I do this in my volunteer capacity so please be kind to me. I start by creating subtasks.

Ladsgroup renamed this task from Upgrade GNU Mailman from 2.1 to 3.3 to Upgrade GNU Mailman from 2.1 to Mailman3.Aug 8 2020, 11:51 PM

In case you missed the announcement, a test server with mailman3 is now available: https://lists.wikimedia.org/pipermail/wikitech-l/2021-March/094382.html

Thanks for taking care of this hard task! I hope we can then get rid of all Google Groups. ;-) (Is there a task for importing these already?)

Swap URLs of lists-next.wikimedia.org to lists.wikimedia.org and lists.wikimedia.org to lists-old.wikimedia.org (make sure archive URLs don't break)

So what's the plan here? Redirect all requests to (public) pipermail-style URLs from lists.wikimedia.org to a static copy on lists-old.wikimedia.org after the switch? (Remember not to regenerate the archives from the mbox, or the new archives will have different IDs.)

Redirecting requests to /pipermail should do most of the job but I'm not sure what to do with the mbox files and the private list archives currently served under the /private path. Preserving links to private archives would be nice, as it will otherwise be a pain to find the equivalents in the new archives. However a simple static mirror cannot handle authentication and translating from old URLs to hyperkitty URLs seems hopeless (unless we have some idea how those thread and message IDs are created).

Swap URLs of lists-next.wikimedia.org to lists.wikimedia.org and lists.wikimedia.org to lists-old.wikimedia.org (make sure archive URLs don't break)

Note that this is no longer the plan. In discussion on T256539: Figure out a way to sync old and new mailman the plan is to serve both mailman2 and mailman3 from the same server, so we can gradually migrate lists individually. This is tracked as T278610: Install mailman3 on lists1001.wikimedia.org. As mentioned in the announcement, lists-next is a temporary testing ground and will be fully deleted once we're satisfied with the process.

Redirect all requests to (public) pipermail-style URLs from lists.wikimedia.org to a static copy on lists-old.wikimedia.org after the switch? (Remember not to regenerate the archives from the mbox, or the new archives will have different IDs.)

In theory hyperkitty actually supports redirecting pipermail URLs (https://gitlab.com/mailman/hyperkitty/-/blob/master/hyperkitty/views/compat.py) but we need to continue testing the migration process to see whether it's good enough. FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer updated.

Redirecting requests to /pipermail should do most of the job but I'm not sure what to do with the mbox files and the private list archives currently served under the /private path. Preserving links to private archives would be nice, as it will otherwise be a pain to find the equivalents in the new archives. However a simple static mirror cannot handle authentication and translating from old URLs to hyperkitty URLs seems hopeless (unless we have some idea how those thread and message IDs are created).

We cannot continue to support /private URLs because as we're getting rid of the mailman2 authentication system. So either a redirector will need to be in place or we break links. Note that mailman3 has fulltext search, which hopefully will help finding old mails. If you have the Message-ID, you can use the hyperkitty API to get the new archive URL: T256539#6456966.

the plan is to serve both mailman2 and mailman3 from the same server [...] FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer update

So the public HTML and txt.gz archives will keep being served indefinitely from the same URLs as before? That will reduce the pain considerably.

the plan is to serve both mailman2 and mailman3 from the same server [...] FWIW the mailman mailing lists preserved their old pipermail archives and just left a note on top saying that the archives are no longer update

So the public HTML and txt.gz archives will keep being served indefinitely from the same URLs as before? That will reduce the pain considerably.

I mostly figured out how to redirect public HTML URLs, see T280731: Implement static redirects from pipermail archives to hyperkitty archives. Should get deployed tomorrow or early next week.

I don't think hyperkitty has an equivalent for the txt.gz files, so we can keep them around (by not deleting them) if there's value.

hyperkitty allows you to download gzip files (click on "Download" in https://lists.wikimedia.org/hyperkitty/list/listadmins@lists.wikimedia.org/2021/5/) Thankfully, building a redirect for it would be pretty easy. Doesn't need any hash.

Change 693599 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] lists: Stop routing mail to mailman2

https://gerrit.wikimedia.org/r/693599

Change 693600 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] lists: Stop mailman2 service

https://gerrit.wikimedia.org/r/693600

Change 693599 merged by Legoktm:

[operations/puppet@production] lists: Stop routing mail to mailman2

https://gerrit.wikimedia.org/r/693599

Change 693600 merged by Legoktm:

[operations/puppet@production] lists: Stop mailman2 service

https://gerrit.wikimedia.org/r/693600

Mentioned in SAL (#wikimedia-operations) [2021-06-01T15:38:21Z] <legoktm> stopped mailman2 service on lists1001 (T52864)

Finally calling this done. The clean up will be handled in T282303: The Great Clean Up of Mailman2