Page MenuHomePhabricator
Create Task
Maniphest T351477

[S] Explore how many rows in the file tables have duplicated timestamp values on WMF wikis
Closed, ResolvedPublic
Actions

Description

Files in the image, oldimage, and filearchive table can have the same upload timestamp value. This task is to determine the extent to which the duplications occur and as such what safeguards need to be in place to deal with files with the same upload timestamp.

To do this queries will be run to find the maximum number of duplications for upload timestamps on commonswiki.

Acceptance criteria
  • Have these queries run and explore what this means for the code that uses the upload timestamp as some kind of uniqueness

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDreamy_JazzT351200 [EPIC] Start automated scanning
 DuplicateNoneT351400 Run the maintenance script scanning images in mediamoderation_scan on WMF wikis
 ResolvedkostajhT351544 [Sub EPIC] Start test wiki scan
 ResolvedNoneT351546 [Sub EPIC] Set up rows
ResolvedTchandersT350865 Import all existing images to the mediamoderation_scan table on WMF wikis
 ResolvedDreamy_JazzT350863 Create maintenance script to import all existing images to mediamoderation_scan table
 ResolvedDreamy_JazzT350323 Write an empty row to scan table on file upload
 ResolvedDreamy_JazzT350864 [M] Create a service to manage rows in the mediamoderation_scan table
 ResolvedBUG REPORTDreamy_JazzT352234 MediaModerationFileProcessor::canScanFile incorrectly registers audio and video files as scannable when TimedMediaHandler extension is installed
 ResolvedDreamy_JazzT351477 [S] Explore how many rows in the file tables have duplicated timestamp values on WMF wikis
 ResolvedDreamy_JazzT353442 Update MediaModerationPhotoDNAServiceProvider::getThumbnailForFile to choose a thumbnail that meets minimum height requirements for PhotoDNA API
 ResolvedBUG REPORTDreamy_JazzT353758 MediaModerationPhotoDNAServiceProvider provides false to FileBackend::getFileContents causing a PHP Notice
 ResolvedDreamy_JazzT351399 Create a maintenance script to automatically scan files listed in mediamoderation_scan
 ResolvedkostajhT351401 Create service(s) to send an image to PhotoDNA for a scan
 ResolvedDreamy_JazzT351407 Create a service that sends an email when a file matches
 ResolvedDreamy_JazzT351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
 ResolvedDreamy_JazzT351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
 ResolvedDreamy_JazzT351409 Expand the database lookup service to get rows from the mediamoderation_scan table
 ResolvedDreamy_JazzT351112 [S] Create an initial service to lookup scan rows from the database
 ResolvedDreamy_JazzT350321 [M] Create database table to store status of scans

Event Timeline

Dreamy_Jazz created this task.Nov 16 2023, 11:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 11:10 PM
Dreamy_Jazz added a parent task: T350863: Create maintenance script to import all existing images to mediamoderation_scan table.Nov 16 2023, 11:12 PM
Dreamy_Jazz added projects: Trust and Safety Product Sprint (Sprint Bodhrán), Trust and Safety Product Team.
Dreamy_Jazz moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Bodhrán) board.
Dreamy_Jazz closed this task as Resolved.EditedNov 16 2023, 11:32 PM
Dreamy_Jazz claimed this task.
Dreamy_Jazz moved this task from In Progress to Done on the Trust and Safety Product Sprint (Sprint Bodhrán) board.

Results:

This suggests that if the batch count is small (say less than 50), then if a upload timestamp with more than 50 usages is found raising the batch size to the number of rows using this timestamp plus 1 would not be a problem.

Dreamy_Jazz updated the task description. (Show Details)Nov 16 2023, 11:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits