@ellery & @leila, this is something that came up quite often at the Dev. Summit. Is it something you'd be interested in looking into?

@Halfak I'm going to assign this task to myself just so I'm able to find it easily and as something I'm interested to work on. Realistically, I can get to this in April or May 2017 once some of the current projects are some more off the ground. Also, I'm cc-ing a few people in WLM international team as I know this is something that will be highly relevant to the work of WLM and the international team may put development resources towards it.

I think some first steps are to define what exactly we'd like to classify and then to tag (or better derive tags) for those things we want to classify. When I made my Proof of concept the main thing I ran into was downloading the images (preferably loads of images on small size (300x300px)) which took relatively long making it unfeasible to classify all images in the uncategorized images (and a future step could even be to reclassify loads of already classified images into new categories if relevant).

Something I thought of as a possibility was to try and train models for different classification tasks and then offer these in the pywikibot framework for others to reuse.

I'd love to continue on the models (have to find some time of course) but struggled a bit with integrating this into existing structures.

It could be possible to use this in trying to generate tags for something like T125273 after which these tags can be used for other purposes (categorisation, search).

Would it be a good idea to have a discussion somewhere in the next weeks to see what people have in mind for an image classifier?

Some questions that I think needs answering:

• What do we want to classify?: scope of classes and images to classify.
• In which way do we want to integrate this? e.g.: a separate tool which runs on images to be categorized, or something more native adding tags to all images.
• What are some steps we can take to facilitate the start of this? e.g.: I think it would be good to annotate some images/collect useful categories we could use as classes.

@Basvb was just about to say this. Can you ping me off-thread and we set up a time to chat?

Hi! I was recently at this Facebook dev summit and I've come across this (possibly) useful library that (I believe) Facebook uses to do image recognition. https://github.com/facebookresearch/multipathnet.
https://caffe2.ai is another one that was widely mentioned.

They're not very openly talking about this, but it's being employed a lot at Facebook. See https://code.facebook.com/posts/457605107772545/under-the-hood-building-accessibility-tools-for-the-visually-impaired-on-facebook/

The recently (4-4-2017 (re)-shared message (original 11-08-2016 on commons-l) on wikitech-l about some of Google's work with Commons images might also be relevant. https://www.youtube.com/watch?v=HgWHeT_OwHc&feature=youtu.be&t=2h1m19s and https://cloud.google.com/blog/big-data/2016/05/explore-the-galaxy-of-images-with-cloud-vision-api

I'm very interested in seeing some of the work you're doing. I've personally invested some time last year to detect monument signs using opencv's templateMatch algorithms, but I haven't followed up recently on this.

In T155538#3490277, @Strainu wrote:

I'm very interested in seeing some of the work you're doing. I've personally invested some time last year to detect monument signs using opencv's templateMatch algorithms, but I haven't followed up recently on this.

Hi Strainu, whose work are you referring to? If there's any recent developments on this I'm very interested to hear about it. What I did hasn't continued since last year and is not much more than downloading 2 sets of images and following the https://www.tensorflow.org/tutorials/image_retraining tutorial. If we can think of some small task (not requiring to go over all images/thousands of categories) we can maybe try to get something to work. Think along the lines of predicting portrait/selfies or COM:PENIS pictures for easier clean up. Another approach can be the complete/large approach (detecting the objects depicted in every image on commons and labeling those), but then some outside help would be a good idea.

In T155538#3490759, @Basvb wrote:

Hi Strainu, whose work are you referring to?

The good thing about images is that they don't differ much across countries, so any work here would be great to see. I would be especially interested in seeing something done over the Panoramio images, many of those are still uncategorised and unprocessed.

An interesting test project could be the detection of images which need to be rotated. For these images getting (endless) training data is trivial: We can assume 99,9% of the images have the correct orrientation. We can select all images as the positive class and rotate an equal amount of images and use these as the negative class.

The amount of rotated images per day (currently done by SteinsplitterBot) falls within 10-50 it seems. Some of these images have had an incorrect orrientation for years or months, others are directly marked by their uploader.

Relevant work: http://www.cs.toronto.edu/~guerzhoy/oriviz/crv17.pdf

Any progress so far? It's been more than a year since it was proposed.

This task looks super interesting. Can you share the link to the project codebase and documentation?

This is probably somewhat of topic: Instead of trying to fine tune manually what goes into "depicts" (see Special:SuggestedTags), wouldn't it be more interesting to make this queryable by adding them into a new property on a large scale?