Page MenuHomePhabricator
Create Task
Maniphest T351416

[S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
Closed, ResolvedPublic
Actions

Description

Files in the image, oldimage, and filearchive table can have the same SHA-1 value. This task is to determine the extent to which the duplications occur and as such what safeguards need to be in place for duplicated files.

To do this queries will be run to find the maximum number of duplications for a SHA-1 on commonswiki.

Acceptance criteria
  • Have these queries run and explore what this means for the code that reads images by SHA-1 value.

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDreamy_JazzT351406 Create a maintenance script to re-send emails for files determined to be a match through PhotoDNA
 ResolvedDreamy_JazzT351200 [EPIC] Start automated scanning
 OpenNoneT351398 Scan the uploadstash table on WMF wikis automatically and hourly
 OpenNoneT351396 Create a maintenance script to check the uploadstash table
 DuplicateNoneT351400 Run the maintenance script scanning images in mediamoderation_scan on WMF wikis
 ResolvedkostajhT351544 [Sub EPIC] Start test wiki scan
 ResolvedNoneT351546 [Sub EPIC] Set up rows
ResolvedTchandersT350865 Import all existing images to the mediamoderation_scan table on WMF wikis
 ResolvedDreamy_JazzT350863 Create maintenance script to import all existing images to mediamoderation_scan table
 ResolvedDreamy_JazzT350323 Write an empty row to scan table on file upload
 ResolvedDreamy_JazzT350864 [M] Create a service to manage rows in the mediamoderation_scan table
 ResolvedDreamy_JazzT351112 [S] Create an initial service to lookup scan rows from the database
 ResolvedDreamy_JazzT350321 [M] Create database table to store status of scans
 ResolvedBUG REPORTDreamy_JazzT352234 MediaModerationFileProcessor::canScanFile incorrectly registers audio and video files as scannable when TimedMediaHandler extension is installed
 ResolvedTchandersT351628 Run the maintenance script scanning images in mediamoderation_scan on a test wiki
 ResolvedDreamy_JazzT353442 Update MediaModerationPhotoDNAServiceProvider::getThumbnailForFile to choose a thumbnail that meets minimum height requirements for PhotoDNA API
 ResolvedBUG REPORTDreamy_JazzT353758 MediaModerationPhotoDNAServiceProvider provides false to FileBackend::getFileContents causing a PHP Notice
 ResolvedDreamy_JazzT351399 Create a maintenance script to automatically scan files listed in mediamoderation_scan
 ResolvedDreamy_JazzT351407 Create a service that sends an email when a file matches
 ResolvedNoneT351420 [Sub EPIC] Create Grafana charts, and Logstash filters
 ResolvedDreamy_JazzT351419 Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki
 ResolvedkostajhT353441 Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA
 ResolvedkostajhT351401 Create service(s) to send an image to PhotoDNA for a scan
 ResolvedDreamy_JazzT351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
 ResolvedDreamy_JazzT351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
 ResolvedSTranT351421 Create a Grafana chart to plot the number of checked and unchecked images per wiki
 ResolvedkostajhT353703 Implement daily runs of updateMetrics on WMF wikis
 ResolvedDreamy_JazzT353074 Create a maintenance script to publish to statsd metrics about the mediamoderation_scan table
 ResolvedDreamy_JazzT351422 Create a logstash filter for MediaModeration related logs

Event Timeline

Dreamy_Jazz created this task.Nov 16 2023, 2:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 2:33 PM
Dreamy_Jazz moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Bodhrán) board.Nov 16 2023, 2:33 PM
Dreamy_Jazz mentioned this in T351417: Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1.
Dreamy_Jazz added a parent task: T351417: Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1.
Dreamy_Jazz updated the task description. (Show Details)Nov 16 2023, 6:23 PM
Dreamy_Jazz added a comment.Nov 16 2023, 11:05 PM

Results:

From this, the code that gets files for a SHA-1 should expect that there could be thousands of file rows for a given SHA-1 and as such should appropriately limit the number of files selected.

It should also handle cases where no SHA-1 exists for a row. No SHA-1 existing for a row seems to mean that there is no image associated with that row, and as such these should be ignored.

Dreamy_Jazz closed this task as Resolved.Nov 16 2023, 11:06 PM
Dreamy_Jazz claimed this task.
Dreamy_Jazz moved this task from In Progress to Done on the Trust and Safety Product Sprint (Sprint Bodhrán) board.
Dreamy_Jazz added a parent task: T350863: Create maintenance script to import all existing images to mediamoderation_scan table.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits