Page MenuHomePhabricator

Upgrade mailing lists from mailman2 to 3 in batches
Open, MediumPublic

Description

With T278610: Install mailman3 on lists1001.wikimedia.org being done soon. We will start migrating mailing lists to the new mailman in batches.

These are the batches:

  • Group A: Mailing lists that have archiving disabled. (44 mailing lists: P15505)
  • Group B: Mailing lists with mbox archive below 100KB (98 mailing lists: P15375)
  • Group C: Below 1MB (138: P15376)
  • Group D: Below 5MB (129 P15382)
  • Group E: Below 10MB (63 P15381)
  • Group F: Below 50MB (132: P15379)
  • Group G: Below 100MB (48 P15377)
  • Group H: Below 1GB (79: P15378)
  • Group I: The rest (reading-web-team, wmfall, arbcom-l, wmfsf, wikinews-pl, wikimedia-in-exec, daily-article-l)
  • The leftovers:
    • mailing lists without an owner or disabled (~80 mailing lists)
    • daily-article-l (Blocked on T282621)
    • arbcom-l (Blocked on T281328)

Mailing lists can go up (volunteer to be done sooner) or go down due to possible complexities of encoding issues in the archive if they are old.

The script to migrate mailing lists will first send an email to admins of that mailing list saying it's going to be disabled, then disable the mailing list, migrate it and then migrate the archive and at last index the archive for search and then send the "congratulation upgrade is finished" email to the admins again.

Links to mailing list pages and subscriptions will change and once it's all done, private archives won't be accessible in their old place anymore (and possibly will be deleted) but they will be retained in hyperkitty (An example: https://lists-next.wikimedia.org/hyperkitty/list/wikitech-l@lists-next.wikimedia.org/thread/CTYPGVR22FOHDFZOZ3RGNQNL34MIRS5S/) archives of public mailing lists will be kept in both old and new places.

Event Timeline

Ladsgroup updated the task description. (Show Details)

The script to migrate mailing lists will first send an email to that mailing list saying it's going to be disabled, then disable the mailing list, migrate it and then migrate the archive and at last index the archive for search and then send the "congratulation upgrade is finished" email to that mailing list.

I'm not sure we need to send an email to the list, maybe just to the list owners?

akosiaris triaged this task as Medium priority.Wed, Apr 21, 10:44 AM

wikimediacz-l and wikimediabr-l have volunteered to go first/early (thank you!).

Question: do we need to migrate lists that were renamed? e.g. there's a disabled "tool-labs-standards-committee" list as it was renamed to "toolforge-standards-committee". I think we can leave behind the old "list" as long as we keep the exim and HTTP redirects in place (see modules/profile/files/exim/listserver_aliases).

wikimediacz-l and wikimediabr-l have volunteered to go first/early (thank you!).

Question: do we need to migrate lists that were renamed? e.g. there's a disabled "tool-labs-standards-committee" list as it was renamed to "toolforge-standards-committee". I think we can leave behind the old "list" as long as we keep the exim and HTTP redirects in place (see modules/profile/files/exim/listserver_aliases).

If it's public mailing list, we can leave it as it is. But if it's private, then it should move over so people can read its archive. Does that sound good to you?

vereinat-l asked to delay until one of their admins is available at the beginning of May.

Question: do we need to migrate lists that were renamed? e.g. there's a disabled "tool-labs-standards-committee" list as it was renamed to "toolforge-standards-committee". I think we can leave behind the old "list" as long as we keep the exim and HTTP redirects in place (see modules/profile/files/exim/listserver_aliases).

If it's public mailing list, we can leave it as it is. But if it's private, then it should move over so people can read its archive. Does that sound good to you?

Looking at rename_list.sh, it copies over the mbox of the old archives to the new list so in theory they should have the full set of archives, but we should double check on the server that's how the lists were actually renamed.

oh I thought about just disabled mailing lists (not renamed ones) and preserving history. I agree with you on renamed ones too.

Feel free to move the wdqs-gui-build mailing list early (it’s currently in Group F), it’s pretty much written by bots and read by nobody (see T189810 / T210286#7034865).

I migrated a bunch of mailing lists now. Wrote a bash script (you can find it in my home) to automate it. It should probably shorten out if the mailing list already exists in mailman3, otherwise good to go

I am a list admin for wikinews-zh and I guess we are ready for the upgrade.

But before that I have two questions:

  1. For creating an account on Mailman3, apart from typing in all the information, will there be a way to connect their Wikimedia account there, probably via OAuth?
  2. Currently, interwiki prefix "mail:" and "mailarchive:" points to the old list and archive page respectively. Will they point to the new page and archive page in foreseeable future so that links on wiki need not to be changed?

Thanks

I am a list admin for wikinews-zh and I guess we are ready for the upgrade.

Thanks. I will migrate it on Monday.

But before that I have two questions:

  1. For creating an account on Mailman3, apart from typing in all the information, will there be a way to connect their Wikimedia account there, probably via OAuth?

It might be possible, we haven't looked into it in depth but it can be. But also, the most importantly, we need to get mailman3 out of door and deployed (and fully get rid of mailman2) and then we can work on improving current system which includes SUL integration (probably through OAuth). The good thing is that upstream is quite responsive.

  1. Currently, interwiki prefix "mail:" and "mailarchive:" points to the old list and archive page respectively. Will they point to the new page and archive page in foreseeable future so that links on wiki need not to be changed?

Yes, that needs fixing in m:Interwiki Map but that's for later. Once all mailing lists have been migrated (hopefully in a month), they'll get an automatic redirect.

Thanks

T249678: Add OAuth login to mailman for accessing list memberships/archive viewing is the task for OAuth, it needs someone to do some work upstream to add MediaWiki as an OAuth provider.

Mentioned in SAL (#wikimedia-operations) [2021-05-03T15:27:26Z] <Amir1> upgrade group A to mailman3 (T280322)

Change 684444 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/puppet@production] mailman3: Copy the config file before disabling the list

https://gerrit.wikimedia.org/r/684444

Change 684444 merged by Legoktm:

[operations/puppet@production] mailman3: Copy the config file before disabling the list

https://gerrit.wikimedia.org/r/684444

Group A is done.

It was really messy.

  • There are wildcard bans being overriden by userlist, please don't do that.
  • Some mailing lists simply don't have an owner (!)
  • The import script cuts the description while it doesn't need to.

Mailing lists that have a wild ban on all addresses:

b"bibliowiki: ban_list = ['^.*$', <redacted>]"
b"board-nominations: ban_list = ['^.*$']"
b"discovery-private: ban_list = ['^.*$', <redacted>]"
b"festivalsommer-teilnehmer: ban_list = ['^.*$']"
b"social-media: ban_list = [<redacted>, '^.*$']"
b"wikiro-l: ban_list =[<redacted>, '^.*$']"
b"wikispecies-admin: ban_list = [ <redacted>,'^.*$', <redacted>]"
b"arbcom-ko (obsolete): ban_list = ['^.*@.*']"
b"cep (obsolete): ban_list = ['^.*@.*']"
b"contact-nl: ban_list = [<redacted>, '^.*']"
b"crc-es-l (obsolete): ban_list = ['^.*@.*']"
b"daily-article-hy (obsolete): ban_list = ['^.*@.*']"
b"design-team (obsolete): ban_list = ['^.*@.*']"
b"glamtools (obsolete): ban_list = ['^.*@.*']"
b"inspire (obsolete): ban_list = ['^.*@.*']"
b"labs-admin (obsolete): ban_list = ['^.*@.*']"
b"labs-announce (obsolete): ban_list = ['^.*@.*']"
b"labs-l (obsolete): ban_list = ['^.*@.*']"
b"localisation-team (obsolete): ban_list = ['^.*@.*']"
b"mediation-en-l: ban_list = ['^.*@.*']"
b"mediawiki-core (obsolete): ban_list = ['^.*@.*']"
b"multimedia-team: ban_list = [<redacted>, '^.*']"
b"privacy-fr-wp: ban_list = ['^.*']"
b"pywikipedia-svn: ban_list = ['^.*']"
b"rcom-l (obsolete): ban_list = ['^.*@.*']"
b"reading-wmf (obsolete): ban_list = ['^.*@.*']"
b"structured-data-team: ban_list = [<redacted>, '^.*']"
b"wdqs-gui-build: ban_list = [<redacted>, '^.*']"
b"wiki-offline-reader-l (obsolete): ban_list = ['^.*@.*']"
b"wikibooksnl-moderator (obsolete): ban_list = ['^.*@.*']"
b"wikibugs-l (obsolete): ban_list = ['^.*@.*']"
b"wikimediameta-l (obsolete): ban_list = ['^.*@.*']"
b"wikimediaro-l (obsolete): ban_list = ['^.*@.*']"
b"wikimetrics (obsolete): ban_list = ['^.*@.*']"
b"wikino-bureaucrats (obsolete): ban_list = ['^.*@.*']"
b"wikisk-l (obsolete): ban_list = ['^.*@.*']"
b"wikistats (obsolete): ban_list = ['^.*@.*']"
b"wlm-africa (obsolete): ban_list = ['^.*@.*']"
b"wmfsf (obsolete): ban_list = ['^.*@.*']"
b"mediation-en-l: ban_list = ['^.*@.*']"

I disabled this one with disable_list yesterday, I wonder why it doesn't show up as obsolete. Also do you want to filter out the lists already migrated to MM3?

Yeah, it uses a week-old config file. I can update it. For now, I want ignore any mailing list in that list.

Mentioned in SAL (#wikimedia-operations) [2021-05-05T19:10:21Z] <Amir1> starting migration of public mailing lists in group b and c to mailman3 (T280322)

So public mailing lists in group B and C are now upgraded to mm3, it was less messy than the first upgrade. I also migrated some other mailing lists for various reasons. Later I'll do public mailing lists in group D and E

I forgot to give some numbers: Now more than 1/4th of all mailing lists are on mm3 \o/

Mentioned in SAL (#wikimedia-operations) [2021-05-06T15:58:26Z] <Amir1> starting upgrade of public mailing lists in group d and e (T280322)

I see that WikiIT-l is in group F. I talked with co-admin @Sannita and we agreed we can offer it as guinea pig before the others in the same group, if it's helpful. Just let me know so I can send a notice to the list.

Thanks. I'm already upgrading group F atm. It'll take an hour or so to reach wikiit-l. We are fairly confident this is stable now :)

I think it's safe to assume I'm doing this.

Mentioned in SAL (#wikimedia-operations) [2021-05-08T17:17:59Z] <Amir1> starting upgrade of batch G of mailing lists (T280322)

Mentioned in SAL (#wikimedia-operations) [2021-05-09T04:27:29Z] <Amir1> starting upgrade of batch H of mailing lists (T280322)

Mentioned in SAL (#wikimedia-operations) [2021-05-12T04:36:22Z] <Amir1> importing archives of wikitech-l (T280322)

Group H is basically done, hyperkitty import failed on wikitech-l and unblock-en-l, I try to import these two. wikien-l and unblock-zh had issues for upgrade altogether which requires redoing. I'll try that. The next group will have to wait for a bit.

Mentioned in SAL (#wikimedia-operations) [2021-05-15T06:54:52Z] <Amir1> migrating most of last mailing lists of T280322

Group H is basically done, hyperkitty import failed on wikitech-l and unblock-en-l

Let me guess: the HTML archives were meddled with (to remove some content someone didn't want to see indexed, without rebuilding from the mbox which would reset all the IDs and URLs).