- Develop an extension that can be invoked through the job queue when image/video/audio files are uploaded or can be called through a maintenance script to process already uploaded image/video/audio files.
- The tool would use PhotoDNA to compare the files with hashes in the industry-wide data bases for child protection and terrorism content. The tool would pull directly from the hash corpus using the API, if feasible.
- If a match between the corpus and the file is detected, the tool will:
- Send an email with the link to the matching media file to the Trust and Safety team to review.
- If the content is flagged as a child protection issue, the tool will automatically delete (takedown) the matched content in a parallel action to the email.
- Terrorism content will not trigger an automatic takedown.
- There should be a flag that can be set to determine if the tool is automatically taking down content or flagging so testing can be done of both methods
- It is preferable that this functionality be built into an extension that is invoked asynchronously through the job queue after a file is uploaded. It should not prevent the file from being uploaded and processed. Processing would happen in the background.
- Create git project on gerrit
- Create a wireframe.
- PhotoDNA integration
- Run requests asynchronously using JobSpecification to
- Define the strategy for deleting files - Don't delete.
- Send emails to "Trust and Safety team"
- Create a page on MediaWiki:Extension portal (https://www.mediawiki.org/wiki/Extension:MediaModeration)
- Deployment - In Progress.
- Acceptance testing
- Turn off debug logging in production
Some background from Foundation Legal: The purpose is to improve the Foundation’s existing workflows for child protection and terrorism related content. Each of these types of material will be treated differently, but there are aspects of the tools underlying both that can be built out in this MVP.
Currently, when the Foundation receives a report of images that depict child sexual abuse, we delete it from the projects and report it to law enforcement according to our legal requirements. This setup requires volunteers, who unlike staff have no professional training or mental health support, to initially deal with this very emotionally taxing content.
This MVP aims to protect the community from being exposed to such content in nearly all cases and get it off the platform a lot faster. It would check images against a database of hashed, known images of child sexual abuse to allow Foundation staff to remove them and report their existence to law enforcement.
This MVP could eventually plug in to other Trust & Safety workflows dealing with terrorism content for Foundation staff to review to see if they meet our existing criteria for credible threats of immediate harm.
This MVP will not automatically remove any content without human review by Foundation staff.