We have a request per second limit of 200 items and a monthly limit of 10 million scans.
To more efficiently work through the backlog, let's parallelize scans (thumbnail retrieval and posting to PhotoDNA).
In the past, we didn't want to use job queue for this because it meant we had little insight into what was being scanned. But now that we are tracking state in the mediamoderation_scan table, and emitting more detailed info to statsd and Logstash, this is less of an issue.
In this task I'd propose that we:
- add a --use-jobqueue option to maintenance/scanFilesInScanTable.php
- Retain the default batch size of 200
- Create a job for each item in the batch, that is responsible for retrieving the thumbnail and posting to PhotoDNA. Basically, the contents of foreach ( $this->generateSha1ValuesForScan() as $sha1 ) { would go into a job class.