Page MenuHomePhabricator

Migrating a private list into mailman3 briefly made archives public (CVE-2021-33038)
Closed, ResolvedPublicSecurity

Description

When migrating the wikimediacz-l mailing list archives into Mailman3, they were briefly public (for a minute or two) during the import process before the private setting kicked in.

<Majavah> I was able to see the list of threads including some summaries of message contents, didnt't test viewing individual threads

I (Legoktm) think we need to run the sync_with_mailman step so hyperkitty knows the list is private *before* we import the archives, right now it only happens afterwards.

Upstream: https://gitlab.com/mailman/hyperkitty/-/issues/380

Event Timeline

Legoktm triaged this task as High priority.Apr 28 2021, 7:02 PM
legoktm@lists1001:/var/log/apache2$ grep "http://lists.wikimedia.org/hyperkitty/list/wikimediacz-l@lists.wikimedia.org/" lists.wikimedia.org-access.log | grep -v static > ~/wikimediacz.log

So we can verify who all looked at the archives

cat ~/wikimediacz.log | awk '{print $3}' | sort | uniq gave 4 IPs, 3 of which I confirmed are myself, Urbanecm, Majavah and the last I'm 95% sure is Amir.

I added sudo mailman-web mailman_sync && between importing the mailing list and importing its archives but it didn't fix the issue :(

I couldn't find an easy solution to it but my guess is that we can close down /hyperkitty for a bit and migrate all private mailing lists.

To: mailman-security@python.org
Subject: hyperkitty_import command leaves archives public until import finishes

Hi again,

At Wikimedia we are working on migrating from Mailman2 to Mailman3. We discovered that during the archive import process (hyperkitty_import), the archives are entirely public until the import finishes (anywhere from minutes to an hour).

I reproduced this in our testing environment and believe I've identified the cause.

Steps to reproduce:

Create a list in Mailman2 and set the archives to be private.

Import the list in Mailman3:
mailman create test@example.org
mailman import21 test@example.org /var/...config.pck

Inspect the mailman3.mailinglist database table, archive_policy for that list should be 1.

Inspect the mailman3_web.hyperkitty_mailinglist database table, the list should not yet be present. At this point the hyperkitty web UI says the list does not exist.

Import the archive into hyperkitty (I created a dummy mbox with tens of thousands of emails so it wouldn't finish immediately):
mailman-web hyperkitty_import -l test@example.org test.mbox

Once the progress marker appears, the list archives should be public on the web, despite the archive policy being set to private.

If you inspect the mailman3_web.hyperkitty_mailinglist table, the list will be there with archive_policy set to 2 (public). Once the import finishes, it'll be synced with mailman and change to the correct setting of 1.

Likely cause:

hyperkitty_import does not explicitly create an entry in the hyperkitty_mailing list table, it only does so once the hyperkitty.lib.incoming.add_to_list() function is called on the first message:

    mlist = MailingList.objects.get_or_create(name=list_name)[0]
    if not getattr(settings, "HYPERKITTY_BATCH_MODE", False):
        update_from_mailman(mlist.name)
    mlist.save()

Because batch mode is set, the default settings for the MailingList object are used, which is a public archive.

Suggested remediation:

hyperkitty_import should explicitly create and insert the MailingList object and sync it with mailman before beginning the import so that the correct archive_policy is used.

MailingList.archive_policy should default to private. Given that in all cases update_from_mailman() should be immediately called, it shouldn't actually make a difference, except in cases like this where the list object isn't synced right away.

Sidenotes:

In postorius, the options for archiving are: public, private, do not archive. It's not obvious that switching from private to do not archive will actually make previous archives public. (I guess you're supposed to keep it as private and uncheck hyperkitty? I didn't test this). A fourth option of "do not archive but keeps archives private" might be sensible.

Credit: Amir Sarabadani (cc'd) and Kunal Mehta.

Please let me know if you have any questions.

Thanks,
-- Kunal Mehta

Upstream is still figuring out whether they consider it to be a security issue or not, but I've prepared the following patch for our hyperkitty package. I tested it in our Cloud VPS and it worked correctly

From: Kunal Mehta <legoktm@debian.org>
Date: Thu, 6 May 2021 14:15:03 -0700
Subject: Fix import

---
 hyperkitty/management/commands/hyperkitty_import.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/hyperkitty/management/commands/hyperkitty_import.py b/hyperkitty/management/commands/hyperkitty_import.py
index 7764fa8..782c8e4 100644
--- a/hyperkitty/management/commands/hyperkitty_import.py
+++ b/hyperkitty/management/commands/hyperkitty_import.py
@@ -48,7 +48,7 @@ from hyperkitty.lib.incoming import DuplicateMessage, add_to_list
 from hyperkitty.lib.mailman import sync_with_mailman
 from hyperkitty.lib.utils import get_message_id
 from hyperkitty.management.utils import setup_logging
-from hyperkitty.models import Email, Thread
+from hyperkitty.models import Email, MailingList, Thread
 
 
 # Allow all wierd line endings.
@@ -319,6 +319,9 @@ class Command(BaseCommand):
         # if (settings.DATABASES["default"]["ENGINE"]
         #     != "django.db.backends.sqlite3":
         #     transaction.set_autocommit(False)
+        mlist = MailingList.objects.get_or_create(name=list_address)[0]
+        mlist.update_from_mailman()
+        mlist.save()
         settings.HYPERKITTY_BATCH_MODE = True
         # Only import emails newer than the latest email in the DB
         latest_email_date = Email.objects.filter(

I initially tried the much smaller fix of changing the MailingList default to private, but that causes a ton of tests to fail.

Deployed the patched package to lists1001 and @Ladsgroup imported the private research-internal list while I watched and it correctly made the list private in hyperkitty *before* beginning to import messages.

So we're unblocked on importing private lists now, but I'll leave this open as we wait for an upstream resolution and disclosure.

Legoktm renamed this task from Migrating a private list into mailman3 briefly made archives public to Migrating a private list into mailman3 briefly made archives public (CVE-2021-33038).May 17 2021, 11:40 PM
Legoktm changed the visibility from "Custom Policy" to "Public (No Login Required)".
Legoktm changed the edit policy from "Custom Policy" to "All Users".