Page MenuHomePhabricator
Create Task
Maniphest T351417

Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

A service is needed that given a SHA-1 returns all matching rows in the image, oldimage, and filearchive as as LocalFile or ArchivedFile objects up to a certain limit and potentially grouped by filename.

The service needs to return all the images so that if one of the image objects cannot be rendered or does not have a thumbnail, another object can be attempted. The number of objects returned will be limited, as quarry queries ran in T351416: [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis determined that some SHA-1 values have thousands of rows and filenames. Creating too many objects could unnecessarily use memory.

Acceptance criteria
  • A service that meets the requirements above is created
  • This service be fully tested

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Add MediaModerationFileFactory and MediaModerationFileLookupmediawiki/extensions/MediaModerationmaster+894 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDreamy_JazzT351200 [EPIC] Start automated scanning
 OpenNoneT351398 Scan the uploadstash table on WMF wikis automatically and hourly
 OpenNoneT351396 Create a maintenance script to check the uploadstash table
 DuplicateNoneT351400 Run the maintenance script scanning images in mediamoderation_scan on WMF wikis
 ResolvedkostajhT351544 [Sub EPIC] Start test wiki scan
 ResolvedNoneT351546 [Sub EPIC] Set up rows
ResolvedTchandersT350865 Import all existing images to the mediamoderation_scan table on WMF wikis
 ResolvedDreamy_JazzT350863 Create maintenance script to import all existing images to mediamoderation_scan table
 ResolvedDreamy_JazzT350323 Write an empty row to scan table on file upload
 ResolvedDreamy_JazzT350864 [M] Create a service to manage rows in the mediamoderation_scan table
 ResolvedBUG REPORTDreamy_JazzT352234 MediaModerationFileProcessor::canScanFile incorrectly registers audio and video files as scannable when TimedMediaHandler extension is installed
 ResolvedDreamy_JazzT351477 [S] Explore how many rows in the file tables have duplicated timestamp values on WMF wikis
 ResolvedBUG REPORTDreamy_JazzT351719 When virtual domains mapping config uses 'db' as false, update.php does not create the table
 DeclinedPRODUCTION ERRORNoneT353391 Deferred update '....UploadCompleteHandler->onUploadComplete' failed to run.
 ResolvedTchandersT351628 Run the maintenance script scanning images in mediamoderation_scan on a test wiki
 ResolvedDreamy_JazzT353442 Update MediaModerationPhotoDNAServiceProvider::getThumbnailForFile to choose a thumbnail that meets minimum height requirements for PhotoDNA API
 ResolvedBUG REPORTDreamy_JazzT353758 MediaModerationPhotoDNAServiceProvider provides false to FileBackend::getFileContents causing a PHP Notice
 ResolvedDreamy_JazzT351399 Create a maintenance script to automatically scan files listed in mediamoderation_scan
 ResolvedNoneT351420 [Sub EPIC] Create Grafana charts, and Logstash filters
 ResolvedDreamy_JazzT351419 Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki
 ResolvedkostajhT353441 Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA
 ResolvedkostajhT351401 Create service(s) to send an image to PhotoDNA for a scan
 ResolvedSTranT351421 Create a Grafana chart to plot the number of checked and unchecked images per wiki
 ResolvedkostajhT353703 Implement daily runs of updateMetrics on WMF wikis
 ResolvedDreamy_JazzT353074 Create a maintenance script to publish to statsd metrics about the mediamoderation_scan table
 ResolvedDreamy_JazzT351422 Create a logstash filter for MediaModeration related logs
 ResolvedDreamy_JazzT351406 Create a maintenance script to re-send emails for files determined to be a match through PhotoDNA
 ResolvedDreamy_JazzT351407 Create a service that sends an email when a file matches
 ResolvedDreamy_JazzT351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
 ResolvedDreamy_JazzT351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
 ResolvedDreamy_JazzT351409 Expand the database lookup service to get rows from the mediamoderation_scan table
 ResolvedDreamy_JazzT351112 [S] Create an initial service to lookup scan rows from the database
 ResolvedDreamy_JazzT350321 [M] Create database table to store status of scans

Event Timeline

Dreamy_Jazz created this task.Nov 16 2023, 2:35 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 2:35 PM
Dreamy_Jazz added a subtask: T351416: [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis.Nov 16 2023, 2:35 PM
Dreamy_Jazz added a parent task: T351401: Create service(s) to send an image to PhotoDNA for a scan.
Dreamy_Jazz added a subtask: T351409: Expand the database lookup service to get rows from the mediamoderation_scan table.
Dreamy_Jazz closed subtask T351416: [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis as Resolved.Nov 16 2023, 11:06 PM
Dreamy_Jazz added a project: Trust and Safety Product Sprint (Sprint Shamisen (20th Nov - 1st Dec. 2023)).Nov 20 2023, 4:31 PM
Dreamy_Jazz set the point value for this task to 2.Nov 20 2023, 4:42 PM
gerritbot added a comment.Dec 2 2023, 12:38 AM

Change 979434 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] [Very WIP] Add MediaModerationFileFactory

https://gerrit.wikimedia.org/r/979434

gerritbot added a project: Patch-For-Review.Dec 2 2023, 12:38 AM
Dreamy_Jazz claimed this task.Dec 2 2023, 12:41 AM
Dreamy_Jazz moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Shamisen (20th Nov - 1st Dec. 2023)) board.
Dreamy_Jazz added a parent task: T351407: Create a service that sends an email when a file matches.Dec 9 2023, 12:26 AM
Dreamy_Jazz edited projects, added Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)); removed Trust and Safety Product Sprint (Sprint Shamisen (20th Nov - 1st Dec. 2023)).Dec 11 2023, 4:43 PM
Dreamy_Jazz moved this task from Priority Backlog to Ready on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 11 2023, 4:52 PM
Dreamy_Jazz moved this task from Ready to Needs review on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 12 2023, 3:12 PM
gerritbot added a comment.Dec 12 2023, 8:32 PM

Change 979434 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Add MediaModerationFileFactory and MediaModerationFileLookup

https://gerrit.wikimedia.org/r/979434

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.10; 2023-12-19).Dec 12 2023, 9:00 PM
Maintenance_bot removed a project: Patch-For-Review.Dec 12 2023, 9:10 PM
Dreamy_Jazz moved this task from Needs review to Needs QA on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 12 2023, 10:35 PM

Similar to {T351417], the services added in this ticket are unused and so there is no easy way to QA this ticket. I would suggest, similar to T351417, that QA be done through T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan where these services will be used to implement the maintenance script. Moving to QA for QA Engineers to decide either way.

Djackson-ctr moved this task from Needs QA to Done on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 13 2023, 1:14 AM
Djackson-ctr subscribed.

I concur with Dreamy_Jazz's comment regarding QA and therefore am pushing this ticket to Done.

Dreamy_Jazz closed this task as Resolved.Dec 13 2023, 4:06 PM
Dreamy_Jazz closed subtask T351409: Expand the database lookup service to get rows from the mediamoderation_scan table as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits