Page MenuHomePhabricator
Create Task
Maniphest T351400

Run the maintenance script scanning images in mediamoderation_scan on WMF wikis
Closed, DuplicatePublic
Actions

Description

Once created, the maintenance script in T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan should be run daily on all WMF wikis. This should be started with a staged approach so that this can be tested on smaller wikis before running on wikis such as commonswiki.

Acceptance criteria

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedDreamy_JazzT351200 [EPIC] Start automated scanning
 DuplicateNoneT351400 Run the maintenance script scanning images in mediamoderation_scan on WMF wikis
 ResolvedkostajhT351544 [Sub EPIC] Start test wiki scan
 ResolvedNoneT351546 [Sub EPIC] Set up rows
ResolvedTchandersT350865 Import all existing images to the mediamoderation_scan table on WMF wikis
 ResolvedDreamy_JazzT350863 Create maintenance script to import all existing images to mediamoderation_scan table
 ResolvedDreamy_JazzT350323 Write an empty row to scan table on file upload
 ResolvedDreamy_JazzT350864 [M] Create a service to manage rows in the mediamoderation_scan table
 ResolvedBUG REPORTDreamy_JazzT352234 MediaModerationFileProcessor::canScanFile incorrectly registers audio and video files as scannable when TimedMediaHandler extension is installed
 ResolvedDreamy_JazzT351477 [S] Explore how many rows in the file tables have duplicated timestamp values on WMF wikis
 ResolvedBUG REPORTDreamy_JazzT351719 When virtual domains mapping config uses 'db' as false, update.php does not create the table
 DeclinedPRODUCTION ERRORNoneT353391 Deferred update '....UploadCompleteHandler->onUploadComplete' failed to run.
 ResolvedTchandersT351628 Run the maintenance script scanning images in mediamoderation_scan on a test wiki
 ResolvedDreamy_JazzT353442 Update MediaModerationPhotoDNAServiceProvider::getThumbnailForFile to choose a thumbnail that meets minimum height requirements for PhotoDNA API
 ResolvedBUG REPORTDreamy_JazzT353758 MediaModerationPhotoDNAServiceProvider provides false to FileBackend::getFileContents causing a PHP Notice
 ResolvedDreamy_JazzT351399 Create a maintenance script to automatically scan files listed in mediamoderation_scan
 ResolvedkostajhT351401 Create service(s) to send an image to PhotoDNA for a scan
 ResolvedDreamy_JazzT351407 Create a service that sends an email when a file matches
 ResolvedDreamy_JazzT351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
 ResolvedDreamy_JazzT351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
 ResolvedDreamy_JazzT351409 Expand the database lookup service to get rows from the mediamoderation_scan table
 ResolvedDreamy_JazzT351112 [S] Create an initial service to lookup scan rows from the database
 ResolvedDreamy_JazzT350321 [M] Create database table to store status of scans

Event Timeline

Dreamy_Jazz created this task.Nov 16 2023, 1:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 1:17 PM
Dreamy_Jazz added a project: Trust and Safety Product Team.Nov 16 2023, 1:19 PM
Dreamy_Jazz updated the task description. (Show Details)
Dreamy_Jazz added a parent task: T351200: [EPIC] Start automated scanning.
Dreamy_Jazz added a subtask: T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan.
Dreamy_Jazz added a subtask: T350865: Import all existing images to the mediamoderation_scan table on WMF wikis.
Dreamy_Jazz removed a subtask: T350865: Import all existing images to the mediamoderation_scan table on WMF wikis.Nov 20 2023, 10:51 AM
Dreamy_Jazz removed a subtask: T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan.Nov 20 2023, 11:01 AM
Dreamy_Jazz added a subtask: T351544: [Sub EPIC] Start test wiki scan.
Dreamy_Jazz added a subtask: T353442: Update MediaModerationPhotoDNAServiceProvider::getThumbnailForFile to choose a thumbnail that meets minimum height requirements for PhotoDNA API.Dec 14 2023, 12:20 PM
Dreamy_Jazz added a subtask: T353758: MediaModerationPhotoDNAServiceProvider provides false to FileBackend::getFileContents causing a PHP Notice.Dec 19 2023, 10:48 PM
Dreamy_Jazz closed subtask T353758: MediaModerationPhotoDNAServiceProvider provides false to FileBackend::getFileContents causing a PHP Notice as Resolved.Jan 3 2024, 3:35 PM
kostajh closed subtask T351544: [Sub EPIC] Start test wiki scan as Resolved.Jan 5 2024, 3:37 PM
Stashbot added a comment.Jan 16 2024, 3:13 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-16T15:13:30Z] <Dreamy_Jazz> T351400 running mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 20 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-20.txt

Stashbot added a comment.Jan 16 2024, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-16T15:18:52Z] <Dreamy_Jazz> T351400 running mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 25 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-20.txt

Stashbot added a comment.Jan 16 2024, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-16T15:29:07Z] <Dreamy_Jazz> T351400 running mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 30 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-30.txt

Stashbot added a comment.Jan 16 2024, 4:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-16T16:03:09Z] <Dreamy_Jazz> T351400 running on a tmux session mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 30 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-30.txt

Stashbot added a comment.Jan 16 2024, 6:38 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-16T18:38:02Z] <Dreamy_Jazz> T351400 running on a tmux session mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --sleep 1 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-non-job-queue.txt

Stashbot added a comment.Jan 17 2024, 11:09 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-17T11:09:53Z] <Dreamy_Jazz> T351400 running on a tmux session mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 30 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-30.txt

Tgr mentioned this in T355243: MediaModeration maintenance script scanFilesInScanTable.php indirectly calls $wgImageMagickConvertCommand.Jan 18 2024, 5:25 AM
Stashbot added a comment.Jan 18 2024, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T09:15:24Z] <Dreamy_Jazz> T351400 running on a tmux session mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --sleep 0 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-0-non-jobqueue.txt

Stashbot added a comment.Jan 18 2024, 10:09 AM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T10:09:10Z] <Dreamy_Jazz> T351400 running on a tmux session foreachwikiindblist group2.dblist extensions/MediaModeration/maintenance/scanFilesInScanTable.php --sleep 0 --verbose 2>&1 | tee ~/scan-files-in-scan-table-group2-sleep-0-non-jobqueue.txt

Stashbot added a comment.Jan 18 2024, 9:14 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T21:14:14Z] <Dreamy_Jazz> Stopped MediaModeration scanning script (T351400)

Stashbot added a comment.Jan 18 2024, 9:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-01-18T21:15:03Z] <Dreamy_Jazz> T351400 running on a tmux session mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 30 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-30-no-render-now.txt

kostajh moved this task from MediaModeration 2.0 to MediaModeration 2.1 on the MediaModeration board.Jan 31 2024, 9:44 AM
kostajh edited projects, added MediaModeration (MediaModeration 2.1); removed MediaModeration (MediaModeration 2.0).
Stashbot added a comment.Feb 21 2024, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-21T13:40:23Z] <Dreamy_Jazz> Re-started MediaModeration scanning script using mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 30 --verbose 2>&1 | tee ~/scan-files-in-scan-table-commonswiki-sleep-30-no-render-now.txt - See T351400

Dreamy_Jazz closed this task as a duplicate of T355169: Run scanFilesInScanTable.php automatically on WMF wikis.Dec 4 2024, 11:41 AM
kostajh closed subtask T353442: Update MediaModerationPhotoDNAServiceProvider::getThumbnailForFile to choose a thumbnail that meets minimum height requirements for PhotoDNA API as Resolved.May 2 2025, 7:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits