Change Details

We'll need to * create an implementation of the [[ https://apiumhub.com/tech-blog-barcelona/introduction-perceptual-hashes-measuring-similarity/ | pHash ]] algorithm that can be run inside MediaWiki (or find an open source implementation we can use) * run this for all existing images, and store the hashes * create a new hash and store it for every new upload The above is covered by T167947, and will allow the community to create tools to find duplicates. If we also gathered **deleted** images and generated and stored hashes for those this would be even more useful. Storing hashes for images on other wikis would be useful too - if those images are fair-use then they're likely to be copyrighted Once that's done we'll need to also generate the hash for stashed uploads in UW, and match that against existing stored hashes, and alert the user if there's a match (and warn them if that match has already been deleted) --- Some implementation details * Trying to figure out how to store hashes that can be searched based on hamming distance is what made a previous version of this run aground (see T121797). According to [[ https://commons.wikimedia.org/wiki/User:F%C3%A6/Imagehash | User:Fæ ]]'s work on this we'll catch ~90% of duplicates just checking for identical hashes, so let's just check for identical hashes * We can possibly store the hashes in a mysql table with a flag to indicate if the original image has been deleted (that gets updated when the image is deleted/undeleted)

We'll need to * create an implementation of the [[ https://apiumhub.com/tech-blog-barcelona/introduction-perceptual-hashes-measuring-similarity/ | pHash ]] algorithm that can be run inside MediaWiki (or find an open source implementation we can use) * run this for all existing images, and store the hashes * create a new hash and store it for every new upload The above is covered by T167947, and will allow the community to create tools to find duplicates. If we also gathered **deleted** images and generated and stored hashes for those this would be even more useful. Storing hashes for images on other wikis would be useful too - if those images are fair-use then they're likely to be copyrighted Once that's done we'll need to also generate the hash for stashed uploads in UW, and match that against existing stored hashes, and alert the user if there's a match (and warn them if that match has already been deleted) --- Some implementation details * Trying to figure out how to store hashes that can be searched based on hamming distance is what made a previous version of this run aground (see T121797). According to [[ https://commons.wikimedia.org/wiki/User:F%C3%A6/Imagehash | User:Fæ ]]'s work on this we'll catch ~90% of duplicates just checking for identical hashes, so let's just check for identical hashes * ~~We can possibly store the hashes in a mysql table with a flag to indicate if the original image has been deleted (that gets updated when the image is deleted/undeleted)~~ ** Ideally we'd store the pHash in the new `file` table, see T28741

We'll need to * create an implementation of the [[ https://apiumhub.com/tech-blog-barcelona/introduction-perceptual-hashes-measuring-similarity/ | pHash ]] algorithm that can be run inside MediaWiki (or find an open source implementation we can use) * run this for all existing images, and store the hashes * create a new hash and store it for every new upload The above is covered by T167947, and will allow the community to create tools to find duplicates. If we also gathered **deleted** images and generated and stored hashes for those this would be even more useful. Storing hashes for images on other wikis would be useful too - if those images are fair-use then they're likely to be copyrighted Once that's done we'll need to also generate the hash for stashed uploads in UW, and match that against existing stored hashes, and alert the user if there's a match (and warn them if that match has already been deleted) --- Some implementation details * Trying to figure out how to store hashes that can be searched based on hamming distance is what made a previous version of this run aground (see T121797). According to [[ https://commons.wikimedia.org/wiki/User:F%C3%A6/Imagehash | User:Fæ ]]'s work on this we'll catch ~90% of duplicates just checking for identical hashes, so let's just check for identical hashes * ~~We can possibly store the hashes in a mysql table with a flag to indicate if the original image has been deleted (that gets updated when the image is deleted/undeleted)~~ ** Ideally we'd store the pHash in the new `file` table, see T28741