Page MenuHomePhabricator

Drinking and Filtering the Recent Changes Stream
Closed, ResolvedPublic

Description

Project Page https://github.com/notconfusing/WIGI
Talk Page https://wikimania2015.wikimedia.org/wiki/Submissions/The_Pleasures_and_Pains_of_Analyzing_All_the_Wikis_in_Realtime

Tutorial Abstract

What started off as the problem of tracking citations eventually lead us to develop a much more general solution - a tool to track all the edits of all Wikis in realtime. With this a new world of possibilities opens up: tracking the trends in what people are writing about, allowing users to receive alerts on edits based on custom queries on article and edit content. These ideas are far away, but we can bring them closer by joining together in building the platform. This introduction is a tutorial in what exists so far in drinking and filtering the Recent Changes (RC) Stream.

Technologies we will cover:

  1. RCstream and websockets.
  2. Wikimedia labs.
  3. Mediawiki diff API.
  4. Wikitext parsing.
  5. Stream rebroadcasting.

We also hope to brainstorm and organize future uses and development of a community platform.

Our Future Uses Brainstorm:

  1. Using the changes queue directly
  2. Trend tracking with dynamic topic modeling (More on this here)
  3. Real-time wikimedia analytics in the style of social media analytics and search
  4. Alerts based on stream queries.

Event Timeline

notconfusing raised the priority of this task from to Needs Triage.
notconfusing updated the task description. (Show Details)
notconfusing added subscribers: notconfusing, Dfko.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 15 2015, 8:06 PM

I hope http://devhub.wmflabs.org/wiki/API:Recent_changes_stream serves as a good starting point. If you come up with a cooler demo or better "sandbox" for changes, let me know or edit the original article on mw.org. I'll try to attend.

Qgil added a subscriber: Qgil.Jul 3 2015, 10:25 AM

Please confirm and promote this activity by assigning it to its owner, listing it or scheduling it at the Hackathon wiki page and by placing it in the right column at #Wikimania-Hackathon-2015. Thank you!

Restricted Application added a subscriber: Luke081515. · View Herald TranscriptJul 3 2015, 10:25 AM
Qgil added a comment.Jul 7 2015, 2:05 PM

@notconfusing, are you planning to give this training session at Wikimania? If so, please schedule it. https://wikimania2015.wikimedia.org/wiki/Hackathon#Schedule

notconfusing renamed this task from Tutorial: Drinking and Filtering the Recent Changes Stream to Drinking and Filtering the Recent Changes Stream.Jul 8 2015, 9:10 PM
notconfusing set Security to None.
notconfusing updated the task description. (Show Details)Jul 15 2015, 5:19 PM

What is the status of this task, now that Wikimania 2015 is over? Did this training session take place? If yes: Please provide an update and potentially summarize findings / potentially provide a link to anything relevant. If no: Please edit this task by removing the #Wikimania-Hackathon-2015 project from this task / potentially close this task by editing its status. Thanks for your help and keeping this task updated!

This session happened, and people were interested in it, specifically for real time plagiarism detection.

notconfusing closed this task as Resolved.Jul 23 2015, 6:09 PM