Page MenuHomePhabricator

General image classifier for commons
Open, LowPublic

Assigned To
Authored By
Jan 17 2017, 8:53 PM
Referenced Files
"Like" token, awarded by Liuxinyu970226."Mountain of Wealth" token, awarded by Ujjwalagrawal17."Love" token, awarded by Effeietsanders."Like" token, awarded by xSavitar."Love" token, awarded by Capankajsmilyo."Like" token, awarded by MichaelMaggs."Haypence" token, awarded by Ricordisamoa.


What it does: Image classification for commons uploads.

Wiki thing it helps with:

  • Crude image categorization (human/selfie, dog, street, house, car) is easier than specific, biggest thing is that you need some train set (thus are not able to predict unseen categories) T331134
  • Find uncategorized images
  • Find likely unwanted images (copyvio, etc.)
  • Estimate image quality T184739
  • Estimate image creation date (1920s, 2000s) which could be used to verify PD-old claims
  • AI which combines image elements with articles and suggests relevant images (combine text based) T236142
  • Automatically generate image captions and alt text

Things that might helps us get this AI built (optional):

Related efforts:

How to move forward?

  • Spec out what needs to be done. (We: Commons, WLM, WMF Research and any other community/entity that should be involved).
  • WMF Research creates a collaboration to work with one student on the research component. I estimate what we're talking about is a 8-12 week work depending on how much details we want to go into.
  • We work with a few developers to implement change. This latter can be a bit more work than usual given that the code-base of Commons will change due to Structured Data on Commons project.

See also

Related Objects

Event Timeline

Halfak added subscribers: leila, Sadads.
Halfak added a subscriber: ellery.

@ellery & @leila, this is something that came up quite often at the Dev. Summit. Is it something you'd be interested in looking into?

Halfak triaged this task as Low priority.Feb 2 2017, 3:32 PM

@Halfak I'm going to assign this task to myself just so I'm able to find it easily and as something I'm interested to work on. Realistically, I can get to this in April or May 2017 once some of the current projects are some more off the ground. Also, I'm cc-ing a few people in WLM international team as I know this is something that will be highly relevant to the work of WLM and the international team may put development resources towards it.

I think some first steps are to define what exactly we'd like to classify and then to tag (or better derive tags) for those things we want to classify. When I made my Proof of concept the main thing I ran into was downloading the images (preferably loads of images on small size (300x300px)) which took relatively long making it unfeasible to classify all images in the uncategorized images (and a future step could even be to reclassify loads of already classified images into new categories if relevant).

Something I thought of as a possibility was to try and train models for different classification tasks and then offer these in the pywikibot framework for others to reuse.

I'd love to continue on the models (have to find some time of course) but struggled a bit with integrating this into existing structures.

It could be possible to use this in trying to generate tags for something like T125273 after which these tags can be used for other purposes (categorisation, search).

Would it be a good idea to have a discussion somewhere in the next weeks to see what people have in mind for an image classifier?

Some questions that I think needs answering:

  • What do we want to classify?: scope of classes and images to classify.
  • In which way do we want to integrate this? e.g.: a separate tool which runs on images to be categorized, or something more native adding tags to all images.
  • What are some steps we can take to facilitate the start of this? e.g.: I think it would be good to annotate some images/collect useful categories we could use as classes.

@Basvb was just about to say this. Can you ping me off-thread and we set up a time to chat?

Hi! I was recently at this Facebook dev summit and I've come across this (possibly) useful library that (I believe) Facebook uses to do image recognition. is another one that was widely mentioned.

They're not very openly talking about this, but it's being employed a lot at Facebook. See

The recently (4-4-2017 (re)-shared message (original 11-08-2016 on commons-l) on wikitech-l about some of Google's work with Commons images might also be relevant. and

I'm very interested in seeing some of the work you're doing. I've personally invested some time last year to detect monument signs using opencv's templateMatch algorithms, but I haven't followed up recently on this.

I'm very interested in seeing some of the work you're doing. I've personally invested some time last year to detect monument signs using opencv's templateMatch algorithms, but I haven't followed up recently on this.

Hi Strainu, whose work are you referring to? If there's any recent developments on this I'm very interested to hear about it. What I did hasn't continued since last year and is not much more than downloading 2 sets of images and following the tutorial. If we can think of some small task (not requiring to go over all images/thousands of categories) we can maybe try to get something to work. Think along the lines of predicting portrait/selfies or COM:PENIS pictures for easier clean up. Another approach can be the complete/large approach (detecting the objects depicted in every image on commons and labeling those), but then some outside help would be a good idea.

Hi Strainu, whose work are you referring to?

The good thing about images is that they don't differ much across countries, so any work here would be great to see. I would be especially interested in seeing something done over the Panoramio images, many of those are still uncategorised and unprocessed.

An interesting test project could be the detection of images which need to be rotated. For these images getting (endless) training data is trivial: We can assume 99,9% of the images have the correct orrientation. We can select all images as the positive class and rotate an equal amount of images and use these as the negative class.

The amount of rotated images per day (currently done by SteinsplitterBot) falls within 10-50 it seems. Some of these images have had an incorrect orrientation for years or months, others are directly marked by their uploader.

Relevant work:

Any progress so far? It's been more than a year since it was proposed.

This task looks super interesting. Can you share the link to the project codebase and documentation?

leila removed leila as the assignee of this task.Mar 18 2020, 11:40 PM
leila removed a subscriber: ellery.

@Miriam can you look into who can be the best person to own the task? (I'm thinking the person is you but I'm cautious in assigning it to you.;)

This is probably somewhat of topic: Instead of trying to fine tune manually what goes into "depicts" (see Special:SuggestedTags), wouldn't it be more interesting to make this queryable by adding them into a new property on a large scale?

Miriam claimed this task.