Page MenuHomePhabricator

Automatically propose/suggest a category for images
Open, LowestPublic

Description

Let's crush the categorisation backlog once and for all!

some categorization could be automated already.

Searching for pictures based on meta-data is called "Concept Based Image Retrieval", searching based on the machine vision recognized content of the image is called "Content Based Image Retrieval".

What I understood of Lars' request, is an automated way of finding the "superfluous" concepts or meta-data for pictures based on their content. Of course recognizing an images content is very hard (and subjective), but I think it would be possible for many of these "superfluous" categories, such as "winter landscape", "summer beach" and perhaps also "red flowers" and "bicycle".

There exist today many open source "Content Based Image Retrieval" systems, that I understand basically works in the way that you give them a picture, and they find you the "matching" pictures accompanied with a score. Now suppose we show a picture with known content (pictures from Commons with good meta-data), then we could to a degree of trust find pictures with overlapping categories. I am not sure whether this kind of automated reverse meta-data labelling should be done for only one category per time, or if some kind of "category bundles" work better. Probably adjectives and items should be compounded (eg "red flowers").

Relevant articles and links from Wikipedia:

  1. w:Image_retrieval
  2. w:Content-based_image_retrieval
  3. w:List_of_CBIR_engines#CBIR_research_projects.2Fdemos.2Fopen_source_projects

    Some demo links bawolff found:
  1. http://demo-itec.uni-klu.ac.at/liredemo/ (Lire might even be integrated with CirrusSearch because it's based on Lucene)
  2. http://image.mdx.ac.uk/time/demo.php
  3. http://mi-file.isti.cnr.it:8765/CophirSearch/
  4. http://orpheus.ee.duth.gr/anaktisi/ (not free)
  5. https://youtube.com/2eaGwk4Xhks
  • Skills: image recognition/analysis, possibly Natural language processing; language depending on implementation, e.g. python for a PWB tool, JavaScript and PHP for a MediaWiki extension, JavaScript for a tool similar to the Wikidata Game.
  • Possible mentors: Kristian Kankainen (Keeleleek); WereSpielChequers?
  • Additional info: "I like the idea of automating categorisation, but I think we are a long way from being able to do much of it. So this would be a big longterm project. One of my concerns is that we are a global site, and we are trying to collect the most diverse set of images that anyone has ever assembled. Image recognition is a good way of saying that we now have another twenty images of this person, but it could be confused when we get our first images of one of the fox subspecies that we don't yet have a picture of. Or rather it would struggle to differentiate the rare and the unique from their more common cousins. There are also some spooky implications for privacy re image recognition and our pictures of people, aside from the obvious things like identifying demonstrators in a crowd or linking a series of shots of one person in such a way as to identify that this photograph of a face belongs to the same person as this photo of pubic hair because the hand is identical; We have had some dodgy things happening on Wikipedia with people wanting to categorise people ethnically and I worry that someone might use a tool such as this to try and semi accurately categorise people as say Jewish. Another major route for improved categorisation is geodata, and I think this could be a less contentious route. Not that everything has geodata, but if things have it could be a neat way to categorise a lot of images, especially if we can get boundary data so we can categorise images as being shot from within a set of boundaries rather than centroid data with all its problems that the parts of one place maybe closer to the centre of an adjacent place than the centre of the area they belong to. WereSpielChequers (talk) 09:58, 20 June 2014 (UTC)"

Proposed in Community-Wishlist-Survey-2016. Received 33 support votes, and ranked #45 out of 265 proposals. View complete proposal with discussion and votes here.

Details

Reference
bz47492

Event Timeline

bzimport raised the priority of this task from to Lowest.
bzimport set Reference to bz47492.
bzimport added a subscriber: Unknown Object (MLST).
Mahitgar created this task.Apr 22 2013, 3:55 AM

It's not clear to me how "automatic suggestion" would work - the only sentence you provide about your *expected* outcome is "The way bugzilla suggests automated summary".

What would proposals be based on? Only checking for strings that were previously used, or do you also have other heuristics in mind?

When would this be displayed, and where? Only when using CategoryTree?

Note people have done things like this before. See CommonSense on toolserver. I have no idea how that tool works or if it could be feasible to use the same method in mw.

I agree with andre that how to display such information is very unclear.

P.s. I don't think the original poster was referring to the CategoryTree extension when s/he said category tree.

Mahitgar: Please elaborate how this woould be gathered and displayed.

Thanks for being prompt from your side and sorry for delay from my side.

*>>how this woould be gathered<<

1)Some part of info is already being gathered for "Special:Categories" ,'Special:UncategorizedPages' and other related pages.Analyse first copule of line of categorised pages and suggest categories to un categorised pages.

2)Second option is to gather information from (interwiki) Wikidata .
Example: Say page 'abc' and its category page 'xyz' both are common for language 'o' and language 'p' wikipedia + may be more other languages; say page 'efg' also is connected on more than two wikis but category 'xyz' is mentioned on some wikis and not mentioned on some wiki's

3)Third option enhancement on HotCat gadget side, when, a category is being added collect first two or three sentences of the article for statisticle comparison,or give a key line option and request editors to provide keyline

4)fourth option, having few say 5 edit filters with upto 10 keywords each and allowing edit filter manager to add category from exmine page

  1. Fifth option As and when Wikipedia goes for realtime editing analyse first couple of lines statistically.

Where to display options and How to insert

1)Vis a Vis above first option: If a page is new or uncategorised , A button can be shown before save pages to add category.

:Horizontal category option display can be provided just above save button line ,and if clicked same as in edit tools it will get added to edit window

2)For option two same as First's option

  1. For third option HotCat option display as some one clicks + button
  1. For fourth option to display and add category from exmine page please also see bug no. 47497 also.

5)For fifth option either like first option or a pop-up after firt few sentences are typed on a new or uncategorised page.

I came across one edit filter Filter no1(http://ja.wikipedia.org/wiki/特別:不正利用フィルター/1) on Japanese Wikipedia it has tagged more than 80000 new uncategorised pages , the data what is already available in special pages.But here purpose of using edit filter may be to be behind page creators to add a category.This is how an avarage wikipedian looks towards imporatance of categorisation it seems.

Gather display and add few more options

option 6)At advance search give an option to compare and suggest a category. If selected Compare the categories also when advance search is being carried out.

Have a differentlly coloured column on right hand side of search results of uncatgorised pages

Option 7)In special pages there is one tool called Special:ComparePages .The tool is currently under utilised if enhanced with feature button assking to show uncategorised pages one after another. and suggesting pages for comparison . Utilisation of this good tool will improve and also rest of the data also will get improved.

Is this bug also apropriate for mw extension VisualEditor ?

So we discuss a number of options here and it is not clear at all how to proceed. Wondering if an https://www.mediawiki.org/wiki/Requests_for_comment would be better.

Yeah, this idea is interesting but pretty much out of scope for Bugzilla as long as it's entirely unclear how to code it.

Qgil moved this task from Backlog to Need Discussion on the Possible-Tech-Projects board.
Qgil updated the task description. (Show Details)
Qgil set Security to None.
Qgil added a subscriber: Qgil.
Qgil added subscribers: Ijon, Niharika, Aklapper.
Qgil added a comment.Sep 23 2015, 9:10 AM

This is a message posted to all tasks under "Re-check in September 2015" at Possible-Tech-Projects. Outreachy-Round-11 is around the corner. If you want to propose this task as a featured project idea, we need a clear plan with community support, and two mentors willing to support it.

Qgil added a comment.Sep 23 2015, 9:36 AM

This is a message sent to all Possible-Tech-Projects. The new round of Wikimedia Individual Engagement Grants is open until 29 Sep. For the first time, technical projects are within scope, thanks to the feedback received at Wikimania 2015, before, and after (T105414). If someone is interested in obtaining funds to push this task, this might be a good way.

Hi. I think this project is very interesting. I want to work on this in the upcoming round of Outreachy and I want to know what is expected out of me in terms of the skills and background knowledge required. Please let me know how I can contribute. Thanks.

Qgil added a subscriber: WereSpielChequers.

The description says "Possible mentors: Kristian Kankainen (Keeleleek); WereSpielChequers?" CCing @WereSpielChequers, couldn't find Kristian.

I'm also adding Community-Tech, since this feature looks like within their scope of tools helping experienced editors. Please remove it if this is not correct.

It looks like this task needs a bit of discussion to confirm that it is a good fit for Possible-Tech-Projects. Is it a feature that is needed and the community would welcome? Is the estimated project time for a senior contributor around 2-3 weeks?

Hi. I am new to image recognition, so I think this could be a great learning opportunity for me. I would really appreciate it if I could get some help understanding the scope of this project and also help me find mentors, or suggest how I could get in touch with possible mentors. I am interested in applying for this project in Outreachy 11, but if that doesn't work out, I think this would still be a really great learning opportunity for me and I would like to contribute to it nonetheless. Thanks.

matmarex renamed this task from Automatically propose/suggest a category to Automatically propose/suggest a category for images.Dec 18 2015, 12:21 AM
Sumit added a subscriber: Sumit.Mar 1 2016, 5:37 PM
IMPORTANT: This is a message posted to all tasks under "Need Discussion" at Possible-Tech-Projects. Wikimedia has been accepted as a mentor organization for GSoC '16. If you want to propose this task as a featured project idea, we need a clear plan with community support, and two mentors willing to support it.
Qgil removed a subscriber: Qgil.Mar 2 2016, 10:21 AM
Niharika removed a subscriber: Niharika.Mar 2 2016, 7:00 PM
This task was proposed in the Community-Wishlist-Survey-2016 and in its current state needs owner. Wikimedia is participating in Google Summer of Code 2017 and Outreachy Round 14. To the subscribers -- would this task or a portion of it be a good fit for either of these programs? If so, would you be willing to help mentor this project? Remember, each outreach project requires a minimum of one primary mentor, and co-mentor.
Sumit added a subscriber: Halfak.Mar 1 2017, 4:29 AM

@Halfak this looks like an interesting and standard ML thing. Can this be structured as a research project to do a feasibility study of methods for image category suggestions for current round of GSoC, Outreachy, RGSoC internships?

Halfak added a comment.Mar 2 2017, 1:05 AM

I'm not sure I'd call it "standard". Image processing is a notoriously difficult problem. See https://xkcd.com/1425/ for relevant discussion.

srishakatux added a comment.EditedApr 19 2017, 11:46 PM

I'm adding this task here T154988. Let me know if it is not a good idea to do so :) Also, if it's fine as it is in the linked task, would someone be willing to be a point of contact/ mentor/ available to answer questions for this task during the hackathon?

I'm happy to mentor, though my availability is limited at present. Email me if a Skype chat would be useful.

Basvb added a subscriber: Basvb.Apr 20 2017, 12:28 PM

@WereSpielChequers That sounds great! Thanks for willing to mentor :) I'm going to link your Phabricator username in the hackathon task, and if someone wants to reach out to you for questions, they can comment here or msg you on Phabricator?

Basvb added a comment.Apr 20 2017, 8:11 PM

I'm interested in thinking along/working on image recognition (and classification) on Commons. See also T155538.

Ainali added a subscriber: Ainali.Jun 18 2018, 6:22 AM

Removing the Possible-Tech-Projects tag as we are planning to kill it soon! This project does not seem to fit in the Outreach-Programs-Projects category in its current state, so I am not adding that tag right now!

Aklapper added a subscriber: Niharika.

Someone correct me if I'm wrong: I think that @Niharika worked on this in T155538: General image classifier for commons as can be seen in https://commons.wikimedia.org/wiki/File:Wikimedia_Hackathon_2018_showcase_-_Image_labelling_and_category_suggester.jpg ?
Wondering if there is a link to a codebase to share?

Someone correct me if I'm wrong: I think that @Niharika worked on this in T155538: General image classifier for commons as can be seen in https://commons.wikimedia.org/wiki/File:Wikimedia_Hackathon_2018_showcase_-_Image_labelling_and_category_suggester.jpg ?
Wondering if there is a link to a codebase to share?

True, but as I remember @Niharika says during the presentation that she worked on a proof-of-concept. Backend is at https://github.com/Niharika29/imagery.

Why removing MediaWiki-extension-requests ? It would be great to have this feature integrated in mediawiki, no ? For example directly integrated into the UploadWizard.

Aklapper removed a subscriber: Niharika.Jul 16 2018, 1:53 PM

@Framawiki: Is there a reason why this should preferably be implemented via a MediaWiki extension (instead of any kind of implementation)?

Hi there. I did work on a proof of concept for this during the hackathon. Backend code is at https://github.com/Niharika29/imagery as Framawiki pointed out. The user script is at https://commons.wikimedia.org/wiki/User:NKohli_(WMF)/imagery.js (It needs a lot of improvements but works okay as a proof of concept). To use it, add importScript( 'User:NKohli_(WMF)/imagery.js' ); to your common.js page and then you will see a 'Detect labels' button underneath each image on the File page for the image. Clicking that button will activate it and after a few seconds it will be replaced by features it found in the image and suggested categories.

This could use a lot of feature and design improvements. I haven't heard any actual feedback on the script yet so I was waiting on that a bit. I would be more than happy to work on improving this script and eventually proposing it be a gadget, if people think that's a good idea.

As for integration into MediaWiki - that would be a challenge. This script works by querying Google's image recognition service. Anything that can do feature labelling for images needs to have a huge, good, trained dataset like Google does. We'd first have to build that in order to build the extension because we do not deploy extensions which rely on external services. That's a monumental task and it will probably be years before we can get that in place.