Page MenuHomePhabricator
Create Task
Maniphest T351409

Expand the database lookup service to get rows from the mediamoderation_scan table
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

The MediaModerationDatabaseLookup service added in T351112: [S] Create an initial service to lookup scan rows from the database should be expanded so that it can provide a given number of mediamoderation_scan rows that match certain criteria related to the scan status and when the last scan timestamp.

Acceptance criteria
  • The MediaModerationDatabaseLookup service should provide the necessary methods to get files listed in the mediamoderation_scan that need a scan, or have been last scanned before a specific date.
  • These new methods should be properly tested.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update MediaModerationDatabaseLookupmediawiki/extensions/MediaModerationmaster+466 -5
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDreamy_JazzT351200 [EPIC] Start automated scanning
 OpenNoneT351398 Scan the uploadstash table on WMF wikis automatically and hourly
 OpenNoneT351396 Create a maintenance script to check the uploadstash table
 DuplicateNoneT351400 Run the maintenance script scanning images in mediamoderation_scan on WMF wikis
 ResolvedkostajhT351544 [Sub EPIC] Start test wiki scan
 ResolvedNoneT351546 [Sub EPIC] Set up rows
ResolvedTchandersT350865 Import all existing images to the mediamoderation_scan table on WMF wikis
 ResolvedDreamy_JazzT350863 Create maintenance script to import all existing images to mediamoderation_scan table
 ResolvedDreamy_JazzT350323 Write an empty row to scan table on file upload
 ResolvedDreamy_JazzT350864 [M] Create a service to manage rows in the mediamoderation_scan table
 ResolvedBUG REPORTDreamy_JazzT352234 MediaModerationFileProcessor::canScanFile incorrectly registers audio and video files as scannable when TimedMediaHandler extension is installed
 ResolvedDreamy_JazzT351477 [S] Explore how many rows in the file tables have duplicated timestamp values on WMF wikis
 ResolvedBUG REPORTDreamy_JazzT351719 When virtual domains mapping config uses 'db' as false, update.php does not create the table
 DeclinedPRODUCTION ERRORNoneT353391 Deferred update '....UploadCompleteHandler->onUploadComplete' failed to run.
 ResolvedTchandersT351628 Run the maintenance script scanning images in mediamoderation_scan on a test wiki
 ResolvedDreamy_JazzT353442 Update MediaModerationPhotoDNAServiceProvider::getThumbnailForFile to choose a thumbnail that meets minimum height requirements for PhotoDNA API
 ResolvedBUG REPORTDreamy_JazzT353758 MediaModerationPhotoDNAServiceProvider provides false to FileBackend::getFileContents causing a PHP Notice
 ResolvedDreamy_JazzT351399 Create a maintenance script to automatically scan files listed in mediamoderation_scan
 ResolvedNoneT351420 [Sub EPIC] Create Grafana charts, and Logstash filters
 ResolvedDreamy_JazzT351419 Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki
 ResolvedkostajhT353441 Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA
 ResolvedkostajhT351401 Create service(s) to send an image to PhotoDNA for a scan
 ResolvedDreamy_JazzT351406 Create a maintenance script to re-send emails for files determined to be a match through PhotoDNA
 ResolvedDreamy_JazzT351407 Create a service that sends an email when a file matches
 ResolvedDreamy_JazzT351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
 ResolvedDreamy_JazzT351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
 ResolvedDreamy_JazzT351409 Expand the database lookup service to get rows from the mediamoderation_scan table
 ResolvedDreamy_JazzT351112 [S] Create an initial service to lookup scan rows from the database
 ResolvedDreamy_JazzT350321 [M] Create database table to store status of scans

Event Timeline

Dreamy_Jazz created this task.Nov 16 2023, 2:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 2:13 PM
Dreamy_Jazz added a parent task: T351401: Create service(s) to send an image to PhotoDNA for a scan.Nov 16 2023, 2:14 PM
Dreamy_Jazz added a subtask: T351112: [S] Create an initial service to lookup scan rows from the database.
Dreamy_Jazz added a parent task: T351417: Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1.Nov 16 2023, 2:36 PM
Dreamy_Jazz added a project: Trust and Safety Product Sprint (Sprint Shamisen (20th Nov - 1st Dec. 2023)).Nov 20 2023, 4:31 PM
Dreamy_Jazz set the point value for this task to 2.Nov 20 2023, 4:42 PM
Dreamy_Jazz claimed this task.Dec 1 2023, 12:27 PM
Dreamy_Jazz moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Shamisen (20th Nov - 1st Dec. 2023)) board.
gerritbot added a comment.Dec 2 2023, 12:38 AM

Change 979433 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] [Very WIP] Update MediaModerationDatabaseLookup

https://gerrit.wikimedia.org/r/979433

gerritbot added a project: Patch-For-Review.Dec 2 2023, 12:38 AM
Dreamy_Jazz updated the task description. (Show Details)Dec 5 2023, 1:57 PM
Dreamy_Jazz added a parent task: T351421: Create a Grafana chart to plot the number of checked and unchecked images per wiki.Dec 8 2023, 6:08 PM
Dreamy_Jazz added a parent task: T353074: Create a maintenance script to publish to statsd metrics about the mediamoderation_scan table.Dec 8 2023, 6:30 PM
Dreamy_Jazz removed a parent task: T351421: Create a Grafana chart to plot the number of checked and unchecked images per wiki.
Dreamy_Jazz removed a parent task: T353074: Create a maintenance script to publish to statsd metrics about the mediamoderation_scan table.
Dreamy_Jazz edited projects, added Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)); removed Trust and Safety Product Sprint (Sprint Shamisen (20th Nov - 1st Dec. 2023)).Dec 11 2023, 4:43 PM
Dreamy_Jazz moved this task from Priority Backlog to Ready on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 11 2023, 4:52 PM
Dreamy_Jazz moved this task from Ready to Needs review on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 12 2023, 11:31 AM
gerritbot added a comment.Dec 12 2023, 8:18 PM

Change 979433 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Update MediaModerationDatabaseLookup

https://gerrit.wikimedia.org/r/979433

Maintenance_bot removed a project: Patch-For-Review.Dec 12 2023, 8:30 PM
ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.10; 2023-12-19).Dec 12 2023, 9:00 PM
Dreamy_Jazz moved this task from Needs review to Needs QA on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 12 2023, 10:32 PM

I am unsure if this ticket can be effectively QA'd because this service is currently unused and therefore there is no way reasonably test this. I've moved this to QA for that decision to be made by QA Engineers. If this ticket doesn't get QA, it should be indirectly QA'd by QA on T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan (where this service will be used).

Djackson-ctr subscribed.Dec 13 2023, 1:18 AM

I concur with Dreamy_Jazz's comment regarding QA and therefore am pushing this ticket to Done.

Djackson-ctr moved this task from Needs QA to Done on the Trust and Safety Product Sprint (Sprint Lute (11th Dec. - 22nd Dec. 2023)) board.Dec 13 2023, 1:19 AM
Dreamy_Jazz closed this task as Resolved.Dec 13 2023, 4:06 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits