Page MenuHomePhabricator

Support parallelizing scans to PhotoDNA
Closed, ResolvedPublic3 Estimated Story Points

Description

We have a request per second limit of 200 items and a monthly limit of 10 million scans.

To more efficiently work through the backlog, let's parallelize scans (thumbnail retrieval and posting to PhotoDNA).

In the past, we didn't want to use job queue for this because it meant we had little insight into what was being scanned. But now that we are tracking state in the mediamoderation_scan table, and emitting more detailed info to statsd and Logstash, this is less of an issue.

In this task I'd propose that we:

  • add a --use-jobqueue option to maintenance/scanFilesInScanTable.php
  • Retain the default batch size of 200
  • Create a job for each item in the batch, that is responsible for retrieving the thumbnail and posting to PhotoDNA. Basically, the contents of foreach ( $this->generateSha1ValuesForScan() as $sha1 ) { would go into a job class.

Event Timeline

Dreamy_Jazz set the point value for this task to 3.Jan 9 2024, 12:48 PM

Change 989202 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] [WIP] Support parallelsing PhotoDNA requests in scanFilesInScanTable.php

https://gerrit.wikimedia.org/r/989202

Change 989202 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Support parallel PhotoDNA requests

https://gerrit.wikimedia.org/r/989202

Suggested QA steps for a local wiki instance:

  1. Install MediaModeration extension, if required
  2. Run ./maintenance/run MediaModeration:dev/PopulateImageTables.php to add some testing images to your wiki that have never been scanned
  3. Inspect the output and copy the filenames of 7 different images into a text file for further processing.
  4. For the filenames from step 3, replace any spaces in the name with _ and remove the File: prefix.
  5. Define $wgMediaModerationPhotoDNAMockServiceFiles in your LocalSettings.php replacing each occurrence of the phrase <file> with a filename from step 4 (making sure to not use a filename more than once). If you already have a definition of $wgMediaModerationPhotoDNAMockServiceFiles then remove it.
$wgMediaModerationPhotoDNAMockServiceFiles = [
	'FilesToIsMatchMap'	=> [
		// change to false if you want IsMatch to be false
		'<file>' => true,
		'<file>' => true,
	],
	'FilesToStatusCodeMap' => [
		// change to 3000 if you want the status code to be "OK"
		'<file>' => 3002,
		'<file>' => 3004,
		'<file>' => 3206,
		'<file>' => 3208,
		'<file>' => 3209,
	]
];
  1. Run ./maintenance/run MediaModeration:scanFilesInScanTable.php --use-jobqueue --verbose
  2. This step depends on whether you use MediaWiki-Docker. If you use it skip to step 8. Otherwise carry out step 7a.
    1. Run ./maintenance/run runJobs.php
  3. The output of the script in step 6 should look something like:
Added X SHA-1 value(s) for scanning via the job queue: ...
X SHA-1 value(s) currently being processed via jobs. Waiting until there are Y or less SHA-1 value(s) being processed before adding more jobs.
.....
  1. Open the DB of your local wiki
  2. Run the following SQL replacing YYYYMMDD with the current date in that format and verify the output is 2
SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_last_checked = YYYYMMDD AND mms_is_match = 1;
  1. Run the following SQL and verify that the return value is 0:
SELECT COUNT(*) FROM mediamoderation_scan WHERE mms_last_checked IS NULL;

Change 990754 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.13] Support parallel PhotoDNA requests

https://gerrit.wikimedia.org/r/990754

Change 990754 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.13] Support parallel PhotoDNA requests

https://gerrit.wikimedia.org/r/990754

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:04:14Z] <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:990754|Support parallel PhotoDNA requests (T354408)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:05:46Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:990754|Support parallel PhotoDNA requests (T354408)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:11:29Z] <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:990754|Support parallel PhotoDNA requests (T354408)]] (duration: 07m 14s)

Djackson-ctr subscribed.

I have verified that the new code has been implemented and is functioning and displaying as expected...
Thank you @Dreamy_Jazz.