Page MenuHomePhabricator

MediaModeration: Strategy for handling child protection media
Closed, ResolvedPublic

Description

Once the malicious content has been identified, it needs to be removed. According to the requirements, simply deleting the upload in MW sense is not enough, since users with elevated rights will be able to view/restore the deleted content.

In addition to deleting the MW file, we need to remove the actual media from the media store. Example of a maintenance script that is doing that is eraseArchivedFile.php.

However, on the other hand, letting automatic software irreversibly delete the content seems dangerous. Should we at least have a special page where admins can track what is being removed? What level of visibility into the deleted content is acceptable? Would we initially want to only flag potentially malicious content for human review and manual removal? Can we expose some of the file metadata to the administrators for review of the tool operation?

In case we don't want a special page, what metadata about child protection media removal could be logged?

Event Timeline

After a conversation with @CCicalese_WMF and Trust and Safety team, currently the strategy for permanent takedown is still unclear, so we need to do a deletion with suppression in a wiki sense. Basically, do LocalFileDeleteBatch with suppress flag set to true. And make sure it works for current and archived file versions.

Anything beyond that is still a bit in the air, so I propose not to do it just yet.

Also, this feature should be configurable with something like $wgMediaModerationEnableFileDelete

To clarify, for the MVP, we will only be notifying of hash matches. We will not be deleting any media. But, it will be good to explore options for the future.

Ok. What is the best option to notify the Trust and Safety team? Just a list of emails from Config and mail function? Does `Send mail' sdk exist in MediaWiki?

Just a list of emails from Config and mail function?

For the first iteration of the MVP - yes, just having a list of email in the config should be enough.

Does `Send mail' sdk exist in MediaWiki?

There is some higher level abstractions for notifying about page edits, but it wouldn't apply here.
There is a UserMailer class that has pretty low-level functionality for emailing, we should probably use that.

Additionally, there is an EmaillingJob in core that we could use. It's documented as being a legacy job, perhaps we could resurrect it.
The benefits of sending emails in jobs:

  • The jobs are persisted, so they're guaranteed to be executed
  • The deduplication of jobs is robust, and it's possible to make it even more global, so that we guarantee only one email per recipient per found violation regardless of the retries of the hash checking job.

The downsides of using the jobs are:

  • The jobs are persisted. So the information about the found violations will be in yet another place.

Overall, I think we would implement it as jobs eventually, but in the current iteration I thing we could just send those emails in place, with a simple list of recipients in the config.

Should the EmaillingJob be "upgraded" according to the new JobSpecification approach?

Also:

The downsides of using the jobs are:

  • The jobs are persisted. So the information about the found violations will be in yet another place.

Why this is a downside?

Why this is a downside?

That is a legal issue. It's actually illegal in a lot of countries to look at the images found by this tool. So, we want to limit the information about what is it doing (in the forms of logs, persisted jobs etc) to the lowest number of people possible, including internal developers.

Maybe it worth to change architectural flow then, and prohibit even uploading/saving such images. We have hooks that prevent saving images with the same names, descriptions, etc.

What was the reason not to save such images at all?

In the short term, during a period of testing, it is imperative that we not prevent upload in case there are false positives. In the long term, it is a fundamental value that we do not prevent upload.

Peter.ovchyn renamed this task from MediaModeration: Strategy for deleting child protection media to MediaModeration: Strategy for handling child protection media.Mar 10 2020, 10:07 AM

Change 578502 had a related patch set uploaded (by Peter.ovchyn; owner: Peter.ovchyn):
[mediawiki/extensions/MediaModeration@master] Send emails to recipients when suspicious content found

https://gerrit.wikimedia.org/r/578502

Change 578502 merged by jenkins-bot:
[mediawiki/extensions/MediaModeration@master] Send emails to recipients when suspicious content found

https://gerrit.wikimedia.org/r/578502