Page MenuHomePhabricator

General image classifier for commons
Open, LowPublic

Tokens
"Like" token, awarded by Liuxinyu970226."Mountain of Wealth" token, awarded by Ujjwalagrawal17."Love" token, awarded by Effeietsanders."Like" token, awarded by D3r1ck01."Love" token, awarded by Capankajsmilyo."Like" token, awarded by MichaelMaggs."Haypence" token, awarded by Ricordisamoa.
Assigned To
Authored By
Halfak, Jan 17 2017

Description

What it does: Image classification for commons uploads.

Wiki thing it helps with:

  • Crude image categorization (human/selfie, dog, street, house, car) is easier than specific, biggest thing is that you need some train set (thus are not able to predict unseen categories)
  • Find uncategorized images
  • Find likely unwanted images (copyvio, etc.)
  • Estimate image quality
  • Estimate image creation date (1920s, 2000s) which could be used to verify PD-old claims
  • AI which combines image elements with articles and suggests relevant images (combine text based)
  • Automatically generate image captions and alt text

Things that might helps us get this AI built (optional):

Related efforts:

How to move forward?

  • Spec out what needs to be done. (We: Commons, WLM, WMF Research and any other community/entity that should be involved).
  • WMF Research creates a collaboration to work with one student on the research component. I estimate what we're talking about is a 8-12 week work depending on how much details we want to go into.
  • We work with a few developers to implement change. This latter can be a bit more work than usual given that the code-base of Commons will change due to Structured Data on Commons project.

See also
https://meta.wikimedia.org/wiki/WikiAI

Event Timeline

Halfak created this task.Jan 17 2017, 8:53 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 17 2017, 8:53 PM
Halfak updated the task description. (Show Details)Jan 17 2017, 8:54 PM
Halfak updated the task description. (Show Details)Jan 18 2017, 12:18 AM
Halfak updated the task description. (Show Details)Jan 18 2017, 12:20 AM
Halfak added subscribers: Lydia_Pintscher, Dbrant.
Halfak updated the task description. (Show Details)Jan 20 2017, 8:17 PM
Halfak added subscribers: leila, Sadads.
Halfak added a subscriber: ellery.

@ellery & @leila, this is something that came up quite often at the Dev. Summit. Is it something you'd be interested in looking into?

Halfak added a subscriber: merbst.Jan 20 2017, 9:12 PM
Ricordisamoa added a subscriber: Ricordisamoa.
MichaelMaggs added a subscriber: MichaelMaggs.
Halfak triaged this task as Low priority.Feb 2 2017, 3:32 PM
leila added a comment.Feb 2 2017, 4:55 PM

@Halfak I'm going to assign this task to myself just so I'm able to find it easily and as something I'm interested to work on. Realistically, I can get to this in April or May 2017 once some of the current projects are some more off the ground. Also, I'm cc-ing a few people in WLM international team as I know this is something that will be highly relevant to the work of WLM and the international team may put development resources towards it.

leila claimed this task.Feb 2 2017, 4:56 PM
leila added a project: Research-Backlog.
leila added subscribers: LilyOfTheWest, Slaporte, Yarl and 3 others.
JeanFred updated the task description. (Show Details)Feb 2 2017, 4:59 PM
leila updated the task description. (Show Details)Feb 2 2017, 5:50 PM
Basvb added a comment.Feb 2 2017, 6:27 PM

I think some first steps are to define what exactly we'd like to classify and then to tag (or better derive tags) for those things we want to classify. When I made my Proof of concept the main thing I ran into was downloading the images (preferably loads of images on small size (300x300px)) which took relatively long making it unfeasible to classify all images in the uncategorized images (and a future step could even be to reclassify loads of already classified images into new categories if relevant).

Something I thought of as a possibility was to try and train models for different classification tasks and then offer these in the pywikibot framework for others to reuse.

I'd love to continue on the models (have to find some time of course) but struggled a bit with integrating this into existing structures.

It could be possible to use this in trying to generate tags for something like T125273 after which these tags can be used for other purposes (categorisation, search).

Would it be a good idea to have a discussion somewhere in the next weeks to see what people have in mind for an image classifier?

Some questions that I think needs answering:

  • What do we want to classify?: scope of classes and images to classify.
  • In which way do we want to integrate this? e.g.: a separate tool which runs on images to be categorized, or something more native adding tags to all images.
  • What are some steps we can take to facilitate the start of this? e.g.: I think it would be good to annotate some images/collect useful categories we could use as classes.

@Basvb was just about to say this. Can you ping me off-thread and we set up a time to chat?

Hi! I was recently at this Facebook dev summit and I've come across this (possibly) useful library that (I believe) Facebook uses to do image recognition. https://github.com/facebookresearch/multipathnet.
https://caffe2.ai is another one that was widely mentioned.

They're not very openly talking about this, but it's being employed a lot at Facebook. See https://code.facebook.com/posts/457605107772545/under-the-hood-building-accessibility-tools-for-the-visually-impaired-on-facebook/

Basvb added a comment.Apr 20 2017, 9:47 PM

The recently (4-4-2017 (re)-shared message (original 11-08-2016 on commons-l) on wikitech-l about some of Google's work with Commons images might also be relevant. https://www.youtube.com/watch?v=HgWHeT_OwHc&feature=youtu.be&t=2h1m19s and https://cloud.google.com/blog/big-data/2016/05/explore-the-galaxy-of-images-with-cloud-vision-api

Strainu added a subscriber: Strainu.Aug 1 2017, 6:49 PM
Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptAug 1 2017, 6:49 PM

I'm very interested in seeing some of the work you're doing. I've personally invested some time last year to detect monument signs using opencv's templateMatch algorithms, but I haven't followed up recently on this.

Puik added a subscriber: Puik.Aug 1 2017, 7:11 PM
Basvb added a comment.Aug 1 2017, 8:28 PM

I'm very interested in seeing some of the work you're doing. I've personally invested some time last year to detect monument signs using opencv's templateMatch algorithms, but I haven't followed up recently on this.

Hi Strainu, whose work are you referring to? If there's any recent developments on this I'm very interested to hear about it. What I did hasn't continued since last year and is not much more than downloading 2 sets of images and following the https://www.tensorflow.org/tutorials/image_retraining tutorial. If we can think of some small task (not requiring to go over all images/thousands of categories) we can maybe try to get something to work. Think along the lines of predicting portrait/selfies or COM:PENIS pictures for easier clean up. Another approach can be the complete/large approach (detecting the objects depicted in every image on commons and labeling those), but then some outside help would be a good idea.

Hi Strainu, whose work are you referring to?

The good thing about images is that they don't differ much across countries, so any work here would be great to see. I would be especially interested in seeing something done over the Panoramio images, many of those are still uncategorised and unprocessed.

Basvb added a comment.Aug 8 2017, 2:28 PM

An interesting test project could be the detection of images which need to be rotated. For these images getting (endless) training data is trivial: We can assume 99,9% of the images have the correct orrientation. We can select all images as the positive class and rotate an equal amount of images and use these as the negative class.

The amount of rotated images per day (currently done by SteinsplitterBot) falls within 10-50 it seems. Some of these images have had an incorrect orrientation for years or months, others are directly marked by their uploader.

Relevant work: http://www.cs.toronto.edu/~guerzhoy/oriviz/crv17.pdf

Miriam added a subscriber: Miriam.Apr 18 2018, 6:23 PM

Any progress so far? It's been more than a year since it was proposed.

D3r1ck01 added a subscriber: D3r1ck01.
Ainali added a subscriber: Ainali.Jun 18 2018, 6:02 AM

This task looks super interesting. Can you share the link to the project codebase and documentation?