Page MenuHomePhabricator

Create service(s) to send an image to PhotoDNA for a scan
Closed, ResolvedPublic3 Estimated Story Points

Description

Service(s) need to be created that allow an image listed in the mediamoderation_scan table or present in the uploadstash table to be scanned using the PhotoDNA API. These services should meet the following requirements:

  • Failures or rate-limit exceeds should be properly handled, such that the file is still marked as unscanned
  • If the thumbnail for an image is not available and cannot be generated on-demand, the image should be skipped and still marked as unscanned
  • When a scan occurs, a statsd event should be created with relevant description about which code triggered the scan.
  • The service(s) should return the scan result to the caller or should update the mediamoderation_scan row.

A service currently exists in the extension to do this in the extension, but will be re-written to support the needs of T351399: Create a maintenance script to automatically scan files listed in mediamoderation_scan and T351396: Create a maintenance script to check the uploadstash table.

Acceptance criteria
  • These services should meet the requirements detailed above
  • These services should be fully tested

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
Resolvedkostajh
ResolvedNone
ResolvedTchanders
ResolvedDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedBUG REPORTDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedBUG REPORTDreamy_Jazz
DeclinedPRODUCTION ERRORNone
ResolvedTchanders
OpenNone
ResolvedBUG REPORTDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedNone
ResolvedDreamy_Jazz
Resolvedkostajh
Resolvedkostajh
ResolvedDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedDreamy_Jazz
ResolvedDreamy_Jazz

Event Timeline

Dreamy_Jazz set the point value for this task to 2.Nov 20 2023, 4:40 PM
Dreamy_Jazz changed the point value for this task from 2 to 3.

Change 981470 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@master] [WIP] Add service for interacting with PhotoDNA API

https://gerrit.wikimedia.org/r/981470

Change 982915 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@master] maintenance: Add debugPhotoDNA script for verifying file

https://gerrit.wikimedia.org/r/982915

Change 983180 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@master] PhotoDNAServiceProvider: Add statsd instrumentation

https://gerrit.wikimedia.org/r/983180

Change 981470 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Add service for interacting with PhotoDNA API

https://gerrit.wikimedia.org/r/981470

Change 983416 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] Add image attribution and expand tests for PhotoDNA API services

https://gerrit.wikimedia.org/r/983416

Change 983435 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] Add further tests for MediaModerationPhotoDNAServiceProvider

https://gerrit.wikimedia.org/r/983435

Change 983439 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] Follow-Up: maintenance: Add debugPhotoDNA script for verifying file

https://gerrit.wikimedia.org/r/983439

Change 982915 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] maintenance: Add debugPhotoDNA script for verifying file

https://gerrit.wikimedia.org/r/982915

Change 983180 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] PhotoDNAServiceProvider: Add statsd instrumentation

https://gerrit.wikimedia.org/r/983180

Change 983416 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Add image attribution and expand tests for PhotoDNA API services

https://gerrit.wikimedia.org/r/983416

Change 983435 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Add further tests for MediaModerationPhotoDNAServiceProvider

https://gerrit.wikimedia.org/r/983435

Change 983500 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] [WIP] Refactor PhotoDNA scanning services to split ::check

https://gerrit.wikimedia.org/r/983500

Change 983439 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Follow-Up: maintenance: Add debugPhotoDNA script for verifying file

https://gerrit.wikimedia.org/r/983439

Change 983500 abandoned by Dreamy Jazz:

[mediawiki/extensions/MediaModeration@master] [WIP] Refactor PhotoDNA scanning services to split ::check

Reason:

Parent patch no longer needs these changes.

https://gerrit.wikimedia.org/r/983500

Change 983690 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] Send PhotoDNA the mime type of the thumbnail and not original file

https://gerrit.wikimedia.org/r/983690

Change 983690 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Send PhotoDNA the mime type of the thumbnail and not original file

https://gerrit.wikimedia.org/r/983690

Change 984166 had a related patch set uploaded (by Kosta Harlan; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.10] Send PhotoDNA the mime type of the thumbnail and not original file

https://gerrit.wikimedia.org/r/984166

Change 984166 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.10] Send PhotoDNA the mime type of the thumbnail and not original file

https://gerrit.wikimedia.org/r/984166

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:17:05Z] <lucaswerkmeister-wmde@deploy2002> Started scap: Backport for [[gerrit:984166|Send PhotoDNA the mime type of the thumbnail and not original file (T351401)]], [[gerrit:984169|Add maintenance script to scan files in the mediamoderation_scan table (T351399)]]

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:18:37Z] <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde and kharlan: Backport for [[gerrit:984166|Send PhotoDNA the mime type of the thumbnail and not original file (T351401)]], [[gerrit:984169|Add maintenance script to scan files in the mediamoderation_scan table (T351399)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2023-12-19T14:24:59Z] <lucaswerkmeister-wmde@deploy2002> Finished scap: Backport for [[gerrit:984166|Send PhotoDNA the mime type of the thumbnail and not original file (T351401)]], [[gerrit:984169|Add maintenance script to scan files in the mediamoderation_scan table (T351399)]] (duration: 07m 53s)

Moving this to QA as I think further issues will be addressed in follow-ups.

I'm not sure how QA would be able to effectively QA this ticket, without essentially doing QA on T351399 which makes use of the services added in this task.

If QA Engineers think this ticket can have QA skipped, then please feel free to move to Done.


On a separate note, it may be difficult to QA T351399 on betawiki as the API key for PhotoDNA is not present on betawikis.

Djackson-ctr subscribed.

I concur with Dreamy_Jazz's comment regarding QA and since this ticket correlates to the same services thats in ticket T351399 (which I am currently testing) then this ticket can be pushed to Done.