Page MenuHomePhabricator

[POC] Face location recognition improvement
Closed, InvalidPublic

Assigned To
Authored By
Klapi
Oct 15 2020, 8:27 AM
Referenced Files
F32385979: Zrzut ekranu 2020-10-15 o 22.15.51.png
Oct 15 2020, 8:19 PM
F32385976: Zrzut ekranu 2020-10-15 o 22.15.33.png
Oct 15 2020, 8:19 PM
F32385421: Screenshot from 2020-10-15 11-04-11.png
Oct 15 2020, 9:06 AM
F32385401: Screenshot_20201014-084058_Wikipedia.jpg
Oct 15 2020, 8:29 AM
F32385406: Screenshot_20201014-091630_Wikipedia.jpg
Oct 15 2020, 8:29 AM
Tokens
"Love" token, awarded by scblr.

Description

Problem: some images are not centred on the face, so the thumbnail shows us the body of a person ina article. We would like to see the face.

This is proove-of-concept [POC] to examine how we can improve face regognition.

It's part of my Hacktoberfest hackaton work.

Event Timeline

Aklapper changed the task status from Open to Stalled.Oct 15 2020, 9:06 AM

Hi @Klapi, thanks for your interest in this and welcome to Wikimedia Phabricator!
What are steps to reproduce this? How to see what is shown in the screenshots? Is this about some unmentioned mobile app?
Please provide some more context for others to understand. :)

Cannot reproduce when going to https://pl.m.wikipedia.org/wiki/Iga_Świątek in mobile mode:

Screenshot from 2020-10-15 11-04-11.png (884×380 px, 139 KB)

Aklapper renamed this task from [POC] Face recognition improvement to [POC] Face location recognition improvement.Oct 15 2020, 9:06 AM

Yes, I came here from https://github.com/wikimedia/apps-android-wikipedia so I made wrong assumption it's phabricator for Android app only. Sorry.

Example with "Iga Świątek" doesn't work any more (someone has just changed photo), but you can try "LeBron James" on app.

Aklapper changed the task status from Stalled to Open.Oct 15 2020, 10:05 AM

Ah, thanks! :)

Dbrant subscribed.

Thanks for creating this task!
I'm moving it to our "Tracking" column, since it's not something for our team to do, but rather something an external volunteer (yourself) would like to do.

Just a little further background, just so you don't spend time unnecessarily:

  • The app uses the built-in Android face detection API for detecting faces in images. In a small fraction of cases, the face detection is unsuccessful, and the result is a misaligned image, as you have seen. As the Android platform evolves, the face detection logic might get improvements that we will automatically inherit.
  • The app does not support gadgets, and there is no API yet for getting structured data created by a specific gadget. To do what you are proposing, you would need to read the full body of the File page from Commons, and parse out the {{ImageNote}} templates created by the ImageAnnotator gadget. This approach would be inefficient and very brittle.
  • In the past we have discussed the possibility of letting users choose the alignment of the image from the app (basically provide an annotation that specifies the focus point of the image), but we are very far from building that kind of feature.

@Dbrant thanks for your answer. We are trying to measure how many cases are correct and how many are not.

There is a threshold in FaceDetector class, however I couldn't find any way to change it and compare the results.

public static final float CONFIDENCE_THRESHOLD = 0.4f;

Also... there is another way: maybe we could use an EXIF data of an image to send data from Commons. I agree that getting additional info from API for every image is not a good solution.

If we figure something out we let you now. It's our proof-of-concept during hackaton.

Best wishes!

Here we can see good example of curent face recognition working fine:

Zrzut ekranu 2020-10-15 o 22.15.33.png (1×744 px, 576 KB)

However it doesn't work with small foto (thumbnail):
Zrzut ekranu 2020-10-15 o 22.15.51.png (1×744 px, 262 KB)

IMHO that's the reason why we should give some tags on fotos on Commons and use this data in Android app.

@Klapi: Hi! This task has been assigned to you a while ago. Could you maybe share an update? :) Do you still plan to work on this task, or do you need any help? Thanks in advance!