Page MenuHomePhabricator

Accuracy review of Wikipedias
Closed, ResolvedPublic

Description

Create a medawiki-utilities bot to find articles in given categories, category trees, and lists. For each such article, find passages with (1) facts and statistics which are likely to have become out of date and have not been updated in a given number of years, and optionally (2) phrases which are likely unclear. Add an indication of the location and the text of those passages either to the page in question using templates, to a bookkeeping page with other page names as headings, and/or to a database local to the bot. Use a customizable array of keywords and regular expressions and measures of text comprehensibility to find such passages for review. Use Wikiwho to pre-compute the age of each word in an article (to avoid the move and blanking issues described in e.g., http://wikitrust.soe.ucsc.edu/talks-and-papers ) before processing each article of interest.

(Done:) Present flagged passages to one or more reviewers. Update html files with the reviewer(s)' response, but keep the original text in files with different suffixes. When reviewers disagree, update the files to reflect that fact, and present the question to a third reviewer to break the tie.

(TODO:) Note that the reputation scores won't be generally displayed to the reviewers but the review items that reduced the score will be displayed to them.

The project repositories contain all the information regarding the current state of the project and will be updated regularly. Check them out for the most up-to-date information.

Project repositories: https://github.com/priyankamandikal/minireview
and: https://github.com/jsalsman/arowf (merge in progress as of July 17, 2016)

Wiki page: https://www.mediawiki.org/wiki/Accuracy_review

Project Blog (update in progress as of July 17, 2016):
https://priyankamandikal.wordpress.com/wiki-accuracy-review/

Etherpad:
https://etherpad.wikimedia.org/p/accuracyreview

Primary mentor: @Jsalsman
Co-mentors: Fabian Flöck (@FaFlo) and @Maribelacosta
Skills: Python 2 and 3, natural language processing
Estimated project time for a senior contributor: 2 weeks

Tasks completed till date:
Deploying and running Wikiwho code in PythonAnywhere
Reviewer reputation database design
Login system in Python Flask for registering and logging in reviewers
SMOG Readability Testing
Implement the Flesch–Kincaid readability test
Review item queue database design
Manual list-based input
Reviewer workflow

Microtasks:
T136290
T136292

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
prnk28 added a subscriber: prnk28.Oct 7 2015, 6:38 PM

Hi there,
Is it too late to be involved in this project as an Outreachy student?
I work actively with Dbpedia and have used NLP and python to build brainforks.com. I have a lot of ideas using machine learning and other algorithms that would help with automated detection review. Let me know if this is still open!
Thanks

I don't know what the Outreachy deadlines are, but I will support all
applicants who want to work on this via Outreachy or IEG or any other means
of sponsorship, and mentor anyone who would like to work on it whether they
have sponsorship or not. Please do the research, apply, and let me know
what I need to do to support you.

01tonythomas added a subscriber: 01tonythomas.

I am shifting this to Outreachy-Round-11 as the project description has two mentors, micro-tasks and looks ready for the 11th edition of Outreachy ( Dec 2015 - Mar 2016 ) . Potential candidates should start by submitting their proposals as a blocker for this task, by November 02.

Feel free to revert it back, if this task has some relevant issues which might block its completion in this term of Outreachy.

01tonythomas removed Jsalsman as the assignee of this task.Oct 20 2015, 10:58 AM

To avoid confusion for a potential candidate, I am setting the task up for grabs to encourage participants to submit proposals for the same.

ref : https://www.mediawiki.org/wiki/Outreach_programs/Possible_mentors#Before

gda94 added a subscriber: gda94.Oct 22 2015, 9:23 PM
gda94 added a comment.Oct 22 2015, 9:53 PM

Hey,
Im majoring in Computational Linguistics pursuing projects in Sentiment Analysis. This project caught my interest and I would be more than willing to contribute to it, if possible, as an Outreachy student. I have some ideas about solving this project. Im currently working on the initial setup as you described in a previous comment. Meanwhile, it would be great if you could set me specific tasks that I could complete by the application deadline for Outreachy.
Thanks.

Hey,
Im majoring in Computational Linguistics pursuing projects in Sentiment Analysis. This project caught my interest and I would be more than willing to contribute to it, if possible, as an Outreachy student. I have some ideas about solving this project. Im currently working on the initial setup as you described in a previous comment. Meanwhile, it would be great if you could set me specific tasks that I could complete by the application deadline for Outreachy.
Thanks.

Hello @gda94 ! Great to hear for your interest in contributing to Wikimedia Community.

Please follow https://www.mediawiki.org/wiki/Outreach_programs/Life_of_a_successful_project#Submitting_your_proposal and start by submitting your idea as a project-proposal.

@Shrutika719, @Minervaxox, @gda94: there are now less than two hours
to get your applications in at
https://outreachy.gnome.org/?q=program_home&prg=5 and it looks like
none of you have started them(?)

prnk28 added a comment.Nov 7 2015, 6:55 PM

I completed the initial task and got the code up and running. I have detailed the steps in the following wiki associated to the task.
https://www.mediawiki.org/wiki/Talk:Accuracy_review

prnk28 updated the task description. (Show Details)Jan 25 2016, 7:02 PM
prnk28 claimed this task.
prnk28 updated the task description. (Show Details)Jan 25 2016, 7:25 PM
prnk28 added a comment.EditedJan 25 2016, 7:37 PM

The login system for reviewers has been completed. The code is written in Python Flask and uses SQLAlchemy ORM for storing the data. Currently, a reviewer can register and login into the system. The password functionality is fully secure through password hashing. Additional functionalities like confirmation mails, password updating and resetting, etc make the login system more robust and thorough.

I have now moved onto working on the review item queue database design and a couple of readability tests that can be run on wiki articles on a sentence-to-sentence basis. SMOG readability testing has already been implemented. I propose to work on the Flesch–Kincaid readability test that is more ubiquitous these days.

The review item queue db can be tentatively described as follows:
title of review item; article name; project e.g. enwiki or dewiki; link; type tag for whether the link is a diff, permalink, ordinary article URL, or other; pertinent excerpt text; source of review item e.g. manual entry, student editor, bias concern, SMOG flagged, etc. More details regarding this will posted later.

Note that the reputation scores won't be generally displayed to the reviewers but the review items that reduced the score will be displayed to them.

Post this, the aim is to integrate all these features into the reviewer workflow. Once the microtasks are completed, this task can be more easily dealt with.

The project repository can be found here: https://github.com/priyankamandikal/wikireview
The project blog can be viewed here: https://priyankamandikal.wordpress.com/wiki-accuracy-review/ or alternately can be viewed in the project wiki on GitHub

I have added the Flesch-Kincaid readability tests for the entire text as well as sentence-by-sentence basis. Code is available here.

Sumit added a subscriber: Sumit.Feb 19 2016, 8:16 PM
NOTE: Outreachy round 12 applications are now open and GSoC 2016 is round the corner. This project was featured for Outreachy round 11 and has a well defined scope. Are you ready to mentor the project this season? If yes, then we'll feature this for Outreachy round 12 and GSoC 2016 as well. Please reply back in comments.
Niharika removed a subscriber: Niharika.Feb 20 2016, 5:20 AM

@Sumit yes I am. What do I need to do to register as an Outreachy mentor for this?

@Sumit yes I am. What do I need to do to register as an Outreachy mentor for this?

You will have to register as a mentor over here - https://outreachy.gnome.org/ - I guess.

@Sumit I have registered and am shown as a mentor. Can you please feature this for Outreachy round 12?

Sumit added a comment.Feb 23 2016, 4:36 PM

@Sumit I have registered and am shown as a mentor. Can you please feature this for Outreachy round 12?

This project is already in the featured column for Outreachy 12. We'll showcase it for GSoC '16 as well after 1st March, if we're selected :)

Qgil removed a subscriber: Qgil.Feb 24 2016, 10:51 AM

@Jsalsman Is this project still open for new contribution for Outreachy 2016 or a potential contributor has been decided?

@Swetarani3007 I prefer projects with two mentees, and we have three mentors, but my understanding is that the other mentors get to decide whether we take one or two mentees.

I still don't see a demand for this proposed feature. The proposal therefore fails https://www.mediawiki.org/wiki/Outreach_programs/Selection_process at multiple levels.

@Nemo_bis during the pilot tests we produced https://youtu.be/30LvELic624 to explain the solution. Prior reviews have stated, "This is part of our core mission: to display accurate, updated information....
A tool for identifying out-of-date information in Wikipedia articles would have significant positive impact for both research and as a mechanism to feed on-wiki workflows...."

Jsalsman updated the task description. (Show Details)Feb 28 2016, 3:48 PM

Video is not a suitable medium for demonstrating community demand.

So is this project being considered for the coming round or not?
@Sumit @01tonythomas could you clarify?

Jsalsman added a subscriber: Qgil.Feb 28 2016, 8:07 PM

@Qgil your guidance would also be appreciated, please.

Re-adding Possible-Tech-Projects to this task. Regardless of whether someone considers the proposal useful/wanted or not, it still is a possible technical project.

So is this project being considered for the coming round or not?

Please see T89416#2055686.

prnk28 added a comment.EditedFeb 28 2016, 9:11 PM

@Aklapper I understand that the project has been featured. I have been working on it since December and would like to continue if possible. But I also presume that projects have to get approved first. Working on a proposal that is unlikely to be accepted would hinder my chances of working on other projects of my interest.

So a clarification regarding the status of this proposal at the earliest by the admins would be apt at this moment.

Nemo_bis removed a subscriber: Nemo_bis.Feb 28 2016, 9:57 PM
Qgil added a comment.Feb 29 2016, 8:52 AM

@Qgil your guidance would also be appreciated, please.

Content curation is an area identified as prioritaire in several discussions. Even if there was no specific demand for this project, I think it still makes sense to consider it as an experiment, and let it go through the same evaluation process as the rest of proposals in this round.

@Swetarani3007 I prefer projects with two mentees, and we have three mentors, but my understanding is that the other mentors get to decide whether we take one or two mentees.

@Jsalsman , sorry - but our policy at https://www.mediawiki.org/wiki/Outreach_programs/Life_of_a_successful_project#For_possible_mentors -- says that

A single project idea shouldn't be shared between two candidates, but try breaking up the large project to individual non-overlapping modules, allowing better evaluation of individual efforts.
FaFlo added a comment.Mar 2 2016, 5:00 PM

So I'm not certain how "demand" is usually measured, but based on discussions I witnessed, and my understanding of the challenges Wikipedia faces regarding quality, I would concur with @Qgil that content curation and bringing in more eyes to spot errors is something that is certainly helpful for the editor community.

Sumit added a comment.EditedMar 8 2016, 5:42 PM

@Aklapper I understand that the project has been featured. I have been working on it since December and would like to continue if possible. But I also presume that projects have to get approved first. Working on a proposal that is unlikely to be accepted would hinder my chances of working on other projects of my interest.

So a clarification regarding the status of this proposal at the earliest by the admins would be apt at this moment.

@prnk28, this is a featured project for GSoC '16/Outreachy-12, proposals for the same are most welcome!
See https://www.mediawiki.org/wiki/Outreach_programs/Life_of_a_successful_project#Coming_up_with_a_proposal

The project plan has been ready for a while, but the student is not sure
how much to allocate among about three months of work, primarily because
how long it will take to get sufficient users for validation
measurements once the initial versions system is ready is a big unknown.
So, I am advising her to be optimistic while making it clear that schedule
slippage because of insufficient utilization is not a failure condition and
will not be counted against her in any evaluations.

I am asking at this be explicitly noted in the application schedule.

There is plenty to do even if lots of early adopters don't show up. The
documentation can be made more thorough and include training materials; we
can add new modules for additional classes of article features to be
flagged for review; the pattern matching system can be extended to
allow flagging different kinds of articles and revisions, by categories and
descriptive sets, and we can add example patterns, both implemented and as
examples for things we might want to do in the future. But ideally, since
the readability level measure is already implemented, we should easily be
able to recruit early adopters from Simple English Wikipedia editors, for
example.

For reference the schedule is due in a week by March 15 (my deadline for
review) and then due in The Outreachy application a week later on the 22nd:

https://wiki.gnome.org/Outreachy/2016/MayAugust

https://developers.google.com/open-source/gsoc/timeline

The project plan has been ready for a while

Any link? (Maybe I've missed that in previous comments, sorry.)

schedule slippage because of insufficient utilization is not a failure condition and will not be counted against her in any evaluations.

Could you clarify which followup events/milestones and how their timing in the schedule would be influenced by utilization? (Answers might already be covered by the project plan.)

Jsalsman added a comment.EditedMar 9 2016, 1:18 PM

@Aklapper the plan is a little fragmented at present. I think 1-3 from https://priyankamandikal.wordpress.com/wiki-accuracy-review/project-scedule/
may be done at least partially at present. On https://www.mediawiki.org/wiki/Accuracy_review#Next_steps
Flesch–Kincaid is implemented, but the article collector for it probably isn't. Then there are the (presently four) issues at https://github.com/priyankamandikal/wikireview/issues

Once we get schedule milestones we will mark the ones which depend on external factors not entirely under our control.

The project plan has been ready for a while

Any link? (Maybe I've missed that in previous comments, sorry.)

@Aklapper , I was in constant touch with the mentor over chat. So much of that hasn't popped up here, (mostly for the reason that it might clutter this discussion).

I have now drafted my proposal for GSoC and have given a detailed schedule for all tasks that I hope to complete during the internship period.

Hi @prnk28, @Jsalsman, @Maribelacosta,

Your starting point looks good! I am a comp sci undergraduate at the Unviersity of San Francisco and I would love to join your project for GSOC '16 as a second mentee. I've been interested in NLP for a while and have some experience with building text editors, markov chains, and flesch scores and would love to participate in this project if there is room for one more!

Could you tell me how I could find out more or get involved in this project?
Thank you!
Shai

@Aklapper Hi ! I would like to be a mentee! Should I submit a proposal with Outreachy or GSOC '16? Or should I contact one of the mentors already on this project? Thanks!

I am sorry, I didn't read the previous comment correctly... :(

Regarding contacting mentors, see https://www.mediawiki.org/wiki/Outreach_programs/Life_of_a_successful_project .
If you are eligible for both GSoC and Outreachy it's recommended to apply or both, as far as I know (someone who is involved with those projects please correct me if I'm wrong.)
Also note that only an individual may work on a given project.

@Shaiwilson
I guess you can submit a proposal after working on a micro task. And as @Aklapper mentioned, you can apply to both programs if you are eligible.

@prnk28 It looks like a proposal was already submitted for this project. Are proposals still being accepted/can I partner with another student to complete the project?

Thanks!

Sumit added a comment.Mar 20 2016, 4:22 AM

Are proposals still being accepted

@Shaiwilson individual proposals may be submitted till 25th March and are evaluated independent of the others, and only one of them is accepted for a project. Please refer to GSoC guidelines here.

can I partner with another student to complete the project?

No, GSoC rules do not allow students to work in teams.

If you want to also look at our other featured projects, have a look at "Featured for GSoC/Outreachy" column in Possible-Tech-Projects to see if others interests you.

@prnk28 @Sumit @Aklapper Would it be possible for me to post my proposal for feedback?

Thank you!

Qgil removed a subscriber: Qgil.May 2 2016, 8:12 AM
prnk28 updated the task description. (Show Details)May 26 2016, 12:58 PM
Scott added a subscriber: Scott.May 26 2016, 2:20 PM
Jsalsman raised the priority of this task from Normal to High.Jul 18 2016, 3:23 AM
Jsalsman updated the task description. (Show Details)
Jsalsman set the point value for this task to 123.
Jsalsman changed the point value for this task from 123 to 1234.Jul 18 2016, 3:32 AM

@prnk28 Here is a copy of the email I just sent you which has a list of things. I thought I would paste it here because you might want to make phabricator tasks from many of these things, but many of them you have already done:

Would you please do this merge? I tried but had some issues. I am
still getting 300% agreement ratios from my test data:

https://github.com/jsalsman/arowf/compare/July10...July18

Highlights:

It's now called arowf instead of minireview. AROWF stands for Accuracy
Review of Wikipedias in Flask.

<br clear=all /> after iframes and throughout the templates. They are
right justified because they are supposed to be indented, but that
doesn't work if the next section's header flows to their left.

I started work on /inspect showing the last 20 (-d)one question file
sets, but then I wondered if that could be abused, so I didn't finish
that. Maybe it would be okay for a registration token search?

I am still getting agreement ratios > 100%. Please correct my formula
again. Sorry. You are lucky I didn't try to merge with your Bootstrap
responsive thing. I really wanted to merge your question generators,
but I trust you can complete those without my help, or just asking me
questions.

I incorporated many of your suggested changes but do not have time to
do a full merge.

See if you can get more of the #@@@ and some of the # to-do items
before you add more than a few menu items to the layout, please? I
want to keep it very clean and simple, but if you want to add links to
your documentation, then create new page(s) on mediawiki.org such as
http://mediawiki.org/Accuracy_review/docs/ask
http://mediawiki.org/Accuracy_review/docs/answer
http://mediawiki.org/Accuracy_review/docs/answer-a
http://mediawiki.org/Accuracy_review/docs/answer-i
http://mediawiki.org/Accuracy_review/docs/answer-r
http://mediawiki.org/Accuracy_review/docs/answer-t
http://mediawiki.org/Accuracy_review/docs/recommend
http://mediawiki.org/Accuracy_review/docs/inspect
etc, then you can link those from a different "Help" menu at each
point in the process, and from the index.html template.

Let me know the Google docs /edit link for the project report (I will
add an outline) and the follow-on grant application template, please.
Just start them empty for now and send Fabian and I the sharing links
please.

Thank you for the Bootstrap work you did at
https://github.com/priyankamandikal/minireview/commit/bfd624269591cc995e88ded93698d235ca4f0663
and for the scripts you have done since.

(DONE:) Use the article names to get Wikiwho dates for "recently" and make the
question like, "The article [article with link] has the word
'recently' inserted in 2006. Is it still correctly used? The context
is '...blah blah 100 characters on a word boundary blah blah recently
blah blah more words....' [iframe HTML; Don't try to pre-scroll to the
context, that is too hard.]"

(DONE?) "The article [article name with link] has a poorly scoring sentence
with Fleish-Kincade readability of X. Can you simplify it? The context
is: '....' [iframe HTML]"

Are you ready to read up on Babel internationalization for tomorrow?

(DONE) Please update the links on http://mediawiki.org/wiki/Accuracy_review
to point to the right github and add the running instance please.

How about making a new blog post tomorrow, too? Just a few paragraphs saying "we changed everything"
:) can you use the diagrams from the first and second pages of that
PDF and .odp file I sent?

Also we still need a URL regexp that handles Unicode and parens in and
around the URL correctly.... You said phabricator might have one?
Google around for that please.

We should start thinking about optional registration. We need some
form where people can sign up to get a unique string emailed to them
that they can include to get credit. It's unlikely that people will
want to try to forge them, but we can hide them from visibility unless
people know what to search on. We can also let the session key
automatically include the signature tokens so people don't have to
paste them. Let's think about that for a few more days.

Best regards,
Jim

Aklapper removed the point value for this task.Sep 12 2018, 9:01 AM
Aklapper closed this task as Resolved.

Closing this task which is tagged as Google-Summer-of-Code (2016) : https://www.mediawiki.org/wiki/Google_Summer_of_Code/Past_projects#Accuracy_review_of_Wikipedias states that the tool was created at https://tools.wmflabs.org/arowf/ as T129536 got accepted.
Specific tasks for Accuracy-Review-of-Wikipedias can be filed under that tag in separate tasks.