What should an AI do you for you? Building an AI Wishlist.
Closed, ResolvedPublic

Tokens
"Like" token, awarded by Tgr."Pterodactyl" token, awarded by bearND."Like" token, awarded by Glorian_Yapinus."Like" token, awarded by ZhouZ."Pterodactyl" token, awarded by Capt_Swing."Love" token, awarded by dr0ptp4kt."Love" token, awarded by Ladsgroup.
Assigned To
Authored By
Halfak, Oct 8 2016

Description

Type of activity: Scheduled session
Main topic: T147708: Facilitate Wikidev'17 main topic "Artificial Intelligence to build and navigate content"
Timing: Tuesday, January 10th at 11:00AM PST
Location: Room 1
Stream link: https://www.youtube.com/watch?v=j8ND7Uu4e_s via @Halfak's laptop
Back channel: #wikimedia-ai
Etherpad: https://etherpad.wikimedia.org/p/devsummit17-AI_wishlist

Historically, AI has been used to filter vandalism, facilitate searching content, and extract structured information from text. In this session, we'll explore both old and new problem spaces where artificial intelligence will be able to help Wikimedia Projects.

Problem statement

There's no good list of open projects and contact lists for AI developers to reference.

Expected outcome

Develop a list of desired projects for AI (including current ones!) and initiate a list of interested parties who want to be consulted and made aware of new developments.

Summary of discussion

We've seen a lot of interest via subscriptions and tokens (@Basvb, @Sumit, @DarTar, @ellery, @Esquivalience, @Lydia_Pintscher, @Quiddity, @Capt_Swing, @Neil_P._Quinn_WMF, @Nettrom, @Ghassanmas, @Glorian_Yapinus and @Ladsgroup) and a bit of discussion of new AIs to consider.

A few of new AIs were discussed.

  • Draft article quality models (spam/vandalism/attack)
  • Edit type models (copy-edit/elaboration/verifiability/npov)
  • Conflict of interest detection
  • Article importance modeling (general and per-wikiproject)
  • Language models for
    • Spam
    • Attack/polite language
    • Vandalism
    • Featured article content

Discussion at the summit should be focused on expanding this list and getting basic description for each proposal.

Links

Halfak created this task.Oct 8 2016, 10:10 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 8 2016, 10:10 AM
Halfak updated the task description. (Show Details)Oct 8 2016, 10:15 AM
Halfak updated the task description. (Show Details)
Ladsgroup added a subscriber: Ladsgroup.
Qgil added a subscriber: Qgil.Oct 10 2016, 12:33 PM

Do you expect this top be a discussion session with many people or more of a hands-on session by a small group? The reason to ask is to decide whether this proposals makes more sense as a pre-scheduled session in the program or as an Unconference session.

Most likely, this will not be hands-on anything. Attendance is intended to be broad. Tool devs are great for asking this question because they are already supporting workflows that might include AI components.

@DarTar expressed interest in documenting the process by which new models get deployed in ORES. This is sort of a meta-item to the wish list.

@ellery is working a recommender system to direct readers across Wikipedia.

@Esquivalience contacted me about created a model that detect assertions of important (re. CSD#A7) or lack there of in new articles.

There's interest in new AIs, but it's not showing up directly on this phabricator task.

I forgot to mention that we're working on building an edit type prediction model. See https://meta.wikimedia.org/wiki/Research:Automated_classification_of_edit_types

I'm also working on a draft quality model for detecting new pages spam/vandalism and attack pages. https://meta.wikimedia.org/wiki/Research:Automated_classification_of_draft_quality

Halfak updated the task description. (Show Details)Nov 18 2016, 10:56 PM
Sumit added a subscriber: Sumit.Nov 19 2016, 1:37 AM
Basvb added a subscriber: Basvb.Nov 19 2016, 6:48 AM

I won't be at wikidev but am: "an interested party who want to be consulted and made aware of new developments."

Good to know. Thanks @Basvb. I'll ping here with details before we have a session and I'll update with notes afterwards.

Halfak updated the task description. (Show Details)Nov 21 2016, 3:37 PM

A couple of months ago, I was thinking that it would be very useful to have a model that detects conflict-of-interest and promotional editing. This seems like it might be an area that would benefit from AI support, since it's relatively common; considerably harder to address and detect than simple vandalism; and has a substantial population of COI editors who would probably follow white-hat editing guidelines if a volunteer introduced them. However, I don't have the expertise to determine if it is actually a tractable problem for AI, so a wishlist would be a good place to register the idea.

One thing to consider, though, is that a COI model should probably be developed in private, because making it public would be a tremendous giveaway to black-hat COI editors. We've raised this concern before about vandalism models (we don't want to train better vandals), but very few vandals are likely to take the time to learn enough about the system to game it. That's not true of COI editors who have a strong financial motivation to do so. A central wishlist could also help catalog high-level concerns like this.

Esquivalience added a comment.EditedNov 21 2016, 9:13 PM

A couple of months ago, I was thinking that it would be very useful to have a model that detects conflict-of-interest and promotional editing. This seems like it might be an area that would benefit from AI support, since it's relatively common; considerably harder to address and detect than simple vandalism; and has a substantial population of COI editors who would probably follow white-hat editing guidelines if a volunteer introduced them. However, I don't have the expertise to determine if it is actually a tractable problem for AI, so a wishlist would be a good place to register the idea.

One thing to consider, though, is that a COI model should probably be developed in private, because making it public would be a tremendous giveaway to black-hat COI editors. We've raised this concern before about vandalism models (we don't want to train better vandals), but very few vandals are likely to take the time to learn enough about the system to game it. That's not true of COI editors who have a strong financial motivation to do so. A central wishlist could also help catalog high-level concerns like this.

This idea has been pitched before: https://meta.wikimedia.org/wiki/Grants:IdeaLab/Bot_to_detect_and_tag_advocacy_editing

I believe that it would definitely be tractable to create such a model: given that we have some promotional and non-promotional text, one could use, for example, a Naive Bayes classifier or Markov chain to classify some text added to articles.

I'm working PCFGs for this. See T148032. Using this, we should be able to highlight sentences that are particularly "spammy".

Halfak updated the task description. (Show Details)Nov 28 2016, 11:21 PM
Halfak added a subscriber: Nettrom.

I've been talking to @Nettrom about an Article Importance prediction model. This model would learn from Wikipedian assessments of importance and then predict the importance of articles that are uncategorized. It would also be able to predict the importance of an article to a specific WikiProject.

Halfak updated the task description. (Show Details)Nov 28 2016, 11:22 PM
Halfak updated the task description. (Show Details)
Halfak updated the task description. (Show Details)
bearND added a subscriber: bearND.

I'm interested in joining remotely.

Halfak updated the task description. (Show Details)Dec 8 2016, 7:34 PM
Halfak renamed this task from Building an AI wishlist & working groups for Wikimedia Projects to What should an AI do you for you? Building an AI Wishlist..Dec 16 2016, 11:16 PM
Halfak updated the task description. (Show Details)
Halfak updated the task description. (Show Details)
Basvb added a comment.Dec 17 2016, 9:02 AM

How about image recognition? https://meta.wikimedia.org/wiki/2016_Community_Wishlist_Survey/Categories/Commons#Use_computer_vision_to_propose_categories seems to gather quite some support and using retraining it is quite easy to obtain reasonable image recognition results without too much difficulty. See https://commons.wikimedia.org/wiki/User:Basvb/Deeplearning (sort on probability!) to see the results of a one day test with 2000 training images.
The main use case for image recognition looks to be (high level) categorisation of uncategorised images.
Other use cases could include:

  • Copyvio detection (maybe not so much AI)
  • Quality estimation (is this a potential QI/FP)
  • Relating images to articles (what article best describes the image/can use this image)
  • Automatic Cropping (proposals)
  • Estimate when an image was taken to verify PD-old claims.
  • Maybe something combined with geotags and recognising nearby objects.

Some of these use cases are likely unrealistic, but I believe it is is possible to have a working high-level image categorisation bot within a few months, and would be very interested to work on something like that with others.

Another idea for text could be fact checking, AI's can be used to gather knowledge of facts from texts, we could see where these facts conflict internally, use this to propose facts for wikidata or check facts against external sources.

A potential use of machine learning techniques can also be to predict user behaviour, for example when a user is about to leave/burnout from conflict or other reasons. This could then be used to give the user some positive feedback to improve the chances of staying on. This one has an ethical side as well, which should be given some consideration. I'm not sure if I'd like to be "followed" like that.

Tgr awarded a token.Dec 23 2016, 12:43 AM
Halfak updated the task description. (Show Details)Dec 28 2016, 8:11 PM
Tarrow added a subscriber: Tarrow.Jan 4 2017, 10:11 AM
cscott added a subscriber: cscott.Jan 4 2017, 11:05 PM

I'm interested in expanding our use of machine translation to aid our editors. I wonder if we might score articles arising from the Content Translation service to aid in content maintenance. As a first step, we might port the "Good edits" score to be relevant to Content Translation: ie incorporate completeness & accuracy of translation, for example, and perhaps also flagging translations that appear fragmentary or incomplete? To improve the maintenance task, we might also look at the evolution of articles over time, and try to distinguish when a translated article has drifted out of date; perhaps there is important new information added to the source article. Can we automatically classify subsequent edits to both articles re "significant additions/changes made"? If one side of the translated pair has "significant changes" and the other doesn't, that might be a cue to translators to update one side or the other of the pair.

Another aspect: can we encourage the spread of information between language-specific wikis? We know that English wikipedia's articles tend to be geotagged in North America and Europe. Can we use machine learning to identify articles in wiki A which are "most interesting" to wiki B, and target them for translation? Generally, can we use classifiers to increase the diversity of articles on our wikis?

To the owner of this session: Here is the link to the session guidelines page: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Session_Guidelines. We encourage you to recruit Note-taker(s) 2(min) and 3(max), Remote Moderator, and Advocate (optional) on the spot before the beginning of your session. Instructions about each role player's task are outlined in the guidelines. The physical version of the role cards will be made available in all the session rooms. Good luck prepping, see you at the summit! :)

Halfak updated the task description. (Show Details)Jan 7 2017, 3:20 AM
Halfak added a comment.Jan 7 2017, 4:22 AM

Just set up https://etherpad.wikimedia.org/p/devsummit17-AI_wishlist. Looking forward to seeing you all in a few days.

Basvb added a comment.Jan 8 2017, 8:38 PM

Is something already known about the stream? Would love to participate from a distance.

@Basvb looks like from the program schedule here https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Program that this session will be video recorded and available for later viewing! learn more about remote participation here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Remote_Participation

Halfak updated the task description. (Show Details)Jan 9 2017, 5:56 PM
Halfak updated the task description. (Show Details)
Halfak updated the task description. (Show Details)Jan 9 2017, 6:10 PM
Halfak added a comment.Jan 9 2017, 6:15 PM

Hi @Basvb, @Ladsgroup, @Nettrom, and anyone participating remotely. It turns out that this session will not have a room-wide stream. So I'll be setting one up on my laptop. It will allow you to sort-of follow along. See https://www.youtube.com/watch?v=j8ND7Uu4e_s

I've also explicitly made space for IRC-based participants. See https://etherpad.wikimedia.org/p/devsummit17-AI_wishlist-IRC. I'll have someone translating instructions to IRC and taking questions. We'll be meeting up in #wikimedia-ai. See you at 1900 UTC tomorrow.

Note-taker(s) of this session: Follow the instructions here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Session_Guidelines#NOTE-TAKER.28S.29 After the session, DO NOT FORGET to copy the relevant notes and summary into a new wiki page following the template here: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Your_Session and also link this from the All Session Notes page: https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/All_Session_Notes. The EtherPad links are also now linked from the Schedule page (https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Schedule) for you!

Qgil removed a subscriber: Qgil.Jan 15 2017, 5:48 PM

See https://phabricator.wikimedia.org/tag/artificial-intelligence/ for wishlist items and to track their development.

Halfak closed this task as Resolved.Feb 7 2017, 8:31 PM