The attachments are available from outside but their list is not: https://lists.wikimedia.org/pipermail/wikidata-bugs/attachments/20200903/ gives you 403 but you can brute force it to get https://lists.wikimedia.org/pipermail/wikidata-bugs/attachments/20200901/7b1e75bd/attachment.htm.

A simple math says it'll be at most three trillion request (36^8) per month of archives so it's not super easy but doable (?) [citation needed]

I deleted all pre 2020 attachments, kept 2020 onwards for checking.

In T279237#6970906, @Ladsgroup wrote:

A simple math says it'll be at most three trillion request (36^8) per month of archives so it's not super easy but doable (?) [citation needed]

Looks like hex rather than arbitrary alphanumerics to me, i.e. base 16, not 36, right? So 16^8 ~= 4 billion requests. But it also looks like it's the day, not the month, in the URL. So 4 billion requests per archive-day per list. "Not super easy but doable" still sounds correct.

Do we have access logs that can confirm no one has done this? I guess there's no way we have them dating back the entire lifespan of mailman 2.

Also: What's the easiest way to turn off external access to those attachments? If you want to keep the 2020+ attachments for investigation for now (reasonable IMO) can we break the URLs with an Apache rule?

As discussed on IRC, a script and systemd timer to cleanup attachments that are over a month old. This gives people who receives digests a month to download any attachments before they're removed.

I ran the script, the log is at lists1001:/home/legoktm/attachments.txt. There were two lists with directories from 20070931 (a day that doesn't exist!) which I deleted by hand.

I deployed https://gerrit.wikimedia.org/r/c/678897, putting a systemd timer in place to run the purge script every day at midnight UTC.

FYI. After the systemd timer, the storage is now 159GB used which is 16GB smaller than the time of creating the ticket. It's safe to assume all of it were attachments that were not supposed to be kept.

Adding @jcrespo and @LSobanski to help dig into the two backups questions:

1. Excluding no-archive lists' attachment directories from our Bacula config, and then
2. Purging older backups of those directories.

Jaime tells me that we may not be able to purge subdirectories, and we'll instead have to purge the entire job -- which is why we'll work in that order, making sure we have at least one clean backup to keep and then purging the older ones.

For 1), I have sent https://gerrit.wikimedia.org/r/c/operations/puppet/+/679863 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/679869 based on @RLazarus list of excluded directories.

Please review on the mailman specific decisions in case something is not as expected (namespace and list of exclusions).

Once it is merged, I will run a full backup in order to verify dirs are no longer backed up.

For 2, I asked Reuven to ask if we need to physically and immediately delete the old backups. I would prefer to keep them- if not for the full retention period, at least for some days, as people have deleted manally files from there and that is the typical reason to require a restore. We can do a manual purge after some time, unless it is legally inadmissible.

I tested mailman3 today and it's much much better but doesn't solve everything. In mailman3 there are 3 digest modes, plain text, MIME, or summary. All 3 just send the contents of the message, including attachments in the message itself, without relying on any URLs, so there's no exposure there.

However until the digest is ready to send, the messages will be stored on disk:

legoktm@lists1002:/var/lib/mailman3/lists$ tree .
.
├── discovery-alerts.lists-next.wikimedia.org
├── pywikibot-bugs.lists-next.wikimedia.org
├── test-announce.lists-next.wikimedia.org
├── test-everyone-is-admin.lists-next.wikimedia.org
├── test-high-volume.lists-next.wikimedia.org
├── test.lists-next.wikimedia.org
│   └── digest.mmdf
├── test-no-archives.lists-next.wikimedia.org
│   └── digest.mmdf
└── wikitech-l.lists-next.wikimedia.org

Eventually we'll add /var/lib/mailman3 to backups, so we either need to exempt these digests using the same path-based mechanism as mailman2, or we just disable digesting if the archives are disabled (what I'm leaning toward).

A rather radical idea (hear me out). Let's not backup /var/lib/mailman3 ever. Everything is in the database that should be already backed up. The only thing not in the database is search index that is already regenerateable from the database content.

I don't have a say, that's up to the maintainers, but remember we can have easy pattern matching (wildcards) for backups if that helps. E.g. maybe we can exclude digest backups if that helps in any way.

Personally, I would prefer to have by default as much coverage with backups as legally, resonably and practically possible- while some data may not be canonical, I am not sure we would be able to regenerate fast enough certain "derived data" in all cases, so feel free to generate redundant backups if necessary. Being able to recover data after a long processing time (not sure what would be the case here) may not be a reasonable model. Backups always use TLS and are stored encrypted from the client, and purged automatically after 3 months, so it has a reasonable security model.

I don't have any specific suggestion, but I wanted to provide some input in case it was useful for a decision.

We got a:

CRITICAL - degraded: The following units failed: check_exclude_backups.service

I haven't researched further.

In T279237#7021472, @Ladsgroup wrote:

A rather radical idea (hear me out). Let's not backup /var/lib/mailman3 ever. Everything is in the database that should be already backed up. The only thing not in the database is search index that is already regenerateable from the database content.

What about /var/lib/mailman3/messages? Those are messages pending moderation where the contents aren't in the database yet (see upstream post).

I think it's basically impossible to ensure that a message copy isn't retained for some time (e.g. logs, exim queue, etc.), so this is mostly about making sure we're not doing it intentionally right? Then I think excluding all the digest files (regardless of private/public state) makes the most sense going forward.

In T279237#7021515, @jcrespo wrote:

I don't have a say, that's up to the maintainers, but remember we can have easy pattern matching (wildcards) for backups if that helps. E.g. maybe we can exclude digest backups if that helps in any way.

If we can exclude /var/lib/mailman3/lists/*/digest.mmdf I think that would be ideal.

In T279237#7023257, @jcrespo wrote:

We got a:

CRITICAL - degraded: The following units failed: check_exclude_backups.service

This is T280744: Mail to root@lists1001.wikimedia.org from noreply@lists1001.wikimedia.org doesn't work. I'll ack the alert.

/var/lib/mailman3/lists/*/digest.mmdf

I believe that will be doable.

Mailman2 is shut down now, I think @Ladsgroup deleted all the private archives for MM2 off of disk, can you confirm? If that's done, I think we can call this resolved? Any other steps that need to be taken care of before we close this task and make it public?

In T279237#7137212, @Legoktm wrote:

Mailman2 is shut down now, I think @Ladsgroup deleted all the private archives for MM2 off of disk, can you confirm?

I confirm

If that's done, I think we can call this resolved?

Fine by me.

Doesn't have to be now and here, but please report at some point changes needed for backups (e.g. stopping backing up mailman 2 files and config) in relation to mailman2 shutdown.

This would handle it https://gerrit.wikimedia.org/r/c/operations/puppet/+/697637/

In T279237#7153625, @Ladsgroup wrote:

We should disclose it I assume.

Hey - the task is fine to make public in Security-Team 's opinion - I can do that. I'd just want to confirm there's no lingering to-dos here or other issues.

Never mind. Per @RLazarus, we'll want to wait a bit for WMF-Legal to formally weigh in on this.

FWIW It's already kinda public, the Gerrit patches describe the issue and the Mailman2 upstream bug is public too.

Just heard back from Legal and they've cleared this to go public. @sbassett or @sguebo_WMF, unless you have any remaining reason to keep it private, go ahead and edit the task policy when ready.

In T279237#7282801, @RLazarus wrote:

Just heard back from Legal and they've cleared this to go public. @sbassett or @sguebo_WMF, unless you have any remaining reason to keep it private, go ahead and edit the task policy when ready.

Done.