Page MenuHomePhabricator
Create Task
Maniphest T336074

[ DEBUG ] fix or remove file hashing
Closed, InvalidPublic
Actions

Description

Right now MediaModeration appears to run into an error when hashing images. Those errors look like this:

2023-05-05 15:51:53.829913 [e0fd7bba5ba543b04bd7df62] mw1438 commonswiki 1.41.0-wmf.7 mediamoderation DEBUG: Hash check request failed for file Mikołaj_Łahodyński_(-1911).jpg.

This error is thrown here, which implies it is a problem with the extension's hashing (and not on the PhotoDNA side).

We need to do the following:

  • validate we are sending PhotoDNA a bad hash. We are receiving errors from 3206 errors from PhotoDNA ("Given file could not be verified as an image"), so we should confirm the hash is the culprit. One way to do this would be to disable hashing and re-run the script and see what happens.
  • investigate why the hashing is failing. Things to verify are (1) we are getting the thumbnail correctly and (2) the request moderation check is working
  • determine how to fix it. We should consider removing hashing all together if the problem will require a lot of time to fix, as the API does not require a hash

Related Objects
Search...

Event Timeline

• JKieserman created this task.May 5 2023, 5:00 PM
• JKieserman updated the task description. (Show Details)
• JKieserman updated the task description. (Show Details)
• JKieserman updated the task description. (Show Details)May 5 2023, 5:04 PM
kostajh subscribed.May 8 2023, 1:25 PM
Aklapper added a project: MediaModeration.May 10 2023, 4:47 PM
Aklapper removed a subscriber: MediaModeration.
• JKieserman edited parent tasks, added: T336699: Debug MediaModeration script; removed: T336069: [EPIC] Run a successful manual scan with MediaModeration's ModerateExistingFiles maintenance script.May 15 2023, 8:02 PM
• JKieserman edited parent tasks, added: T287511: Evaluate PhotoDNA MediaModeration failures; removed: T336699: Debug MediaModeration script.May 15 2023, 9:52 PM
• ERayfield subscribed.May 31 2023, 11:13 AM

FWIW, since T336205 https://phabricator.wikimedia.org/T336205 will not be using hashing, would that take care of this ticket?

kostajh added a comment.May 31 2023, 12:01 PM

Right now MediaModeration appears to run into an error when hashing images. Those errors look like this:

My understanding is that the MediaModeration extension does not do any hashing of images. It instead sends a URL to PhotoDNA, and the PhotoDNA services visits that URL. The PhotoDNA service then generates a hash of the image, which it compares to its internal database of hashes. The documentation for the endpoint also indicates that you either send the image content or a publicly accessible URL.

Hash check request failed for file Mikołaj_Łahodyński_(-1911).jpg.

I think this is just a not well-written error message. What the error message is saying is, "The HTTP request to the PhotoDNA service failed", not "Attempting to generate a hash of an image failed".

So overall, I think we can mark this task as Invalid.

• JKieserman closed this task as Invalid.Jun 1 2023, 6:21 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits