Page MenuHomePhabricator

Progress: Build a data visualization tool for the evolution of Wikipedia articles maintained by WikiProjects
Closed, ResolvedPublic

Description

Task to track progress for T356498: Outreachy Project: Build a data visualization tool for the evolution of Wikipedia articles maintained by WikiProjects

Profile

Name: Mahima Singhal
Email: agarwalmahima01@gmail.com
Github: mahi01agarwal
LinkedIn: agarwalmahima01
Location: New Delhi, India.
Time Zone: UTC+5:30
Typical Working Hours: 9 AM to 2 AM (UTC+05:30)

Synopsis

Using this model to predict the quality of Wikipedia articles, the WMF's Research team built a dataset with feature values and predicted quality scores for all revisions of all articles in more than 300 language editions of Wikipedia. Thus, this dataset provides key information on the expansion and quality of Wikipedia articles over time. In this project we will build on this novel data asset to develop a visualization tool that will allow anyone to explore the evolution of quality in articles maintained by WikiProjects.

Specifically, the work will the following phases:

  • Becoming familiar with the dataset, i.e. how to pre-process the data, what is the data structure, what other features could be added (e.g. pageviews), etc.
  • Selection of specific Wikiprojects of interest
  • Analysis and visualization of language-agnostic features of revisions of articles from Wikiprojects (over time)
  • Build an interactive web interface for users to explore the data visualizations

Mentors

Timeline

PeriodTask
May 27 to June 02Onboarding and readings
Jun 03 to June 09Start working with the dataset; Exploring WikiProjects
Jun 10 to June 16Continue working with the dataset; Exploring article quality
Jun 17 to June 23Visualization of User Journey #1
Jun 24 to June 31Visualization of User Journey #2
Jul 01 to July 07Visualization of User Journey #3
Jul 08 to July 14Review of visualizations #1 #2 #3 (Wikiproject expert)
Jul 15 to July 21Development of an MVP following the reviews of User Journeys #1 #2 #3
Jul 22 to July 28Development of an MVP following the reviews of User Journeys #1 #2 #3
Jul 29 to Aug 04Development of an MVP following the reviews of User Journeys #1 #2 #3
Aug 05 to Aug 11Final Adjustments and Enhancements
Aug 12 to Aug 18Final Adjustments and Enhancements (Write Documentation)
Aug 19 to Aug 23Finalize and Review

Deliverables

TBD

About the mentee

I’m Mahima, a curious mind and tech enthusiast currently on my B.Tech journey in Information Technology at the Cluster Innovation Centre, University of Delhi, India. My passion? Diving deep into the mesmerizing worlds of Artificial Intelligence, Machine Learning and Data Science.

Issues

none

Event Timeline

@Aklapper because we have been overwhelmed by the many Outreachy applicants subscribed and commenting to T356498 and want to have a new clean subtask for coordination with the selected intern.

Pablo updated Other Assignee, added: MahimaSinghal.
Pablo updated the task description. (Show Details)
Pablo updated Other Assignee, removed: MahimaSinghal.
Pablo updated the task description. (Show Details)

Week 1 : May 27 - May 31

Onboarding and Readings

Overview of Tasks Completed:

Learnings:

  • Knowledge about the Language-Agnostic Quality Model.
  • Knowlege about WikiProject, WikiProject Council and Knowlege Gaps.
  • Knowledge about Wikidata Query Service.
  • SPARQL syntax.

Week 2 : June 3 - June 7

Start working with the dataset

Overview of Tasks Completed:

  • Conducted research on existing tools .
  • Formulated User Stories for the tool.
  • Completed my second blog : Everybody Struggles
  • Documented the work I've done here : Week 2

Learnings:

  • Creating User Stories.
  • Exploratory Data Analysis(EDA).
  • Gained knowledge about existing tools and bots, which will serve as benchmarks for our tool.

Week 3 : June 10 - June 14

Continue Working With Dataset; Exploring Article Quality

Overview of Tasks Completed:

  • Explored various visualization techniques.
  • Created a draft visualization for different user stories.
  • Link to the Visualization Draft : Visualization Draft

Learnings:

  • Gained knowledge about data visualization methods.
  • Acquired insights into Wikimedia Quarry.

Week 4: June 17 - June 21

Visualization of User Stories: Journey #1

Overview of Tasks Completed:

  • Collected data for User Story 1 using Wikimedia Quarry and API.
  • Created colab notebooks for visualizations of User Story 1 for the WikiProjects: Climate Change, Science, and Caribbean.
  • Link to the document for Visualization for User Story 1: Week 4
  • Attended WikiWorkshop 2024.

Learnings:

  • Gained more knowledge about WikiProjects and the categories within a WikiProject.
  • Learned about additional Wiki technologies.

Week 5: June 24 - June 28

Visualization of User Stories: Journey #2

Overview of Tasks Completed:

  • Completed my third Blog Post: Think about your audience
  • Created a blog post explaining Wikimedia Quarry: Wikimedia Quarry
  • Collected data for User Story 2 using Wikimedia Quarry and API.
  • Created Colab notebooks for visualizations of User Story 2 for the WikiProjects: Climate Change, Science, and Caribbean.
  • Created PAWS notebooks for my work.

Learnings:

  • Gained more knowledge about Data Visualization tools

Week 6: Jul 01 to July 07

Visualization of User Stories: Journey #3

Overview of Tasks Completed:

  • Completed my blog post: Progress Report: My Journey So Far in the Outreachy Internship
  • Modified visualizations for User Story 1 and User Story 2 based on feedback received.
  • Created visualizations for User Story 3.
  • Selected Cloud VPS as the final tool for deploying our application after evaluating various Wikimedia cloud services.

Learnings:

  • Expanded my knowledge of data visualization tools and techniques.
  • Acquired insights into Cloud VPS, including its setup and management.
  • Explored and understood various cloud services offered by Wikimedia, aiding in the decision-making process for deployment.

Quick spark-sql query to get link changes where someone tags a new wiki project on the talk page:

select meta.dt,
       page_id,
       transform(filter(added_links, l -> not l.external and rlike(l.link, '/wiki/Template:WikiProject_')), e -> substring(e.link, 28))
  from event_sanitized.mediawiki_page_links_change
 where year = 2024 and month = 7 and day = 10 and hour = 10
   and page_namespace = 1
   and cardinality(filter(added_links, l -> rlike(l.link, '/wiki/Template:WikiProject_'))) > 0

Week 7: Jul 08 to July 12

Review of visualizations #1 #2 #3 (Wikiproject expert)

Overview of Tasks Completed:

  • Completed my blog post: Career goals after Outreachy
  • Had a meeting with the Wikiproject Expert about our User Stories.
  • Modified my visualizations according to his reviews.

Learnings:

  • Learned about how PetScan and PagePile works.
  • Learned how users can use data from our tool in PetScan.

Week 8: Jul 15 to July 19

Development of an MVP following the reviews of User Journeys #1 #2 #3

Overview of Tasks Completed:

  • Started working on the prototype of our tool.
  • Using Flask for the backend of our database; currently focusing on the Caribbean dataset.
  • Using Flask for the frontend; began developing the frontend of our tool.
  • Using Wikimedia/research-api-interface-template for CSS.

Learnings:

  • Learned about React and Flask as I am very new to development.
  • Learned how to integrate Flask and React to improve the tool.

Week 9: Jul 22 to July 26

Development of an MVP following the reviews of User Journeys #1 #2 #3

Overview of Tasks Completed:

  • Initiated the backend development of our tool.
  • Created a Toolforge account.
  • Successfully connected to the Toolforge server using SSH.
  • Set up a personal database within the Wikimedia Tools DB.

Learnings:

  • Understood the fundamentals of SSH and how it ensures secure remote connections.
  • Acquired skills in connecting MySQL to a server via SSH tunnels.
  • Gained knowledge about managing databases.

Frontend Prototype:

Week 10: Jul 29 to Aug 02

Development of an MVP following the reviews of User Journeys #1 #2 #3

Overview of Tasks Completed:

  • Enhanced the frontend of the tool by adding more features.
  • Set up a database for the tool on the toolsDb server.
  • Integrated accessibility-related features.

Learnings:

  • Gained a deeper understanding of making graphs interactive and visually appealing using React features.

Week 11: Aug 05 to Aug 09

Development of an MVP following the reviews of User Journeys #1 #2 #3

Overview of Tasks Completed:

  • Re-organized the dashboard into two tabs that appear after the Wikiproject dropdown menu.
  • Focused on re-organizing the UI to enhance user-friendliness.

Learnings:

  • Learned how to use react-tabs and implemented them effectively to improve the user interface.

Week 12: Aug 12 to Aug 16

Final Adjustments and Enhancements (Write Documentation)
As I near the end of my Outreachy internship, I've focused on finalizing tasks and preparing documentation.

Overview of Tasks Completed:

  • Documentation: Given that this was the second-to-last week of my internship, I prioritized creating comprehensive documentation. This documentation will serve as a guide for future users and developers, detailing how the tool functions.
  • Project Status: While the project is not fully deployed yet, it is nearly complete. All core functionalities have been implemented, and the tool is functioning as expected in a test environment. The final deployment is pending, but preparations for this are well underway.

Week 13: Aug 19 to Aug 23

Finalize and Review

Overview of Tasks Completed:

  • Frontend Design: Implemented the final changes to the frontend design, ensuring everything is polished and aligned with the project's goals.
  • Documentation: Focused on refining and finalizing project documentation.
  • Project Status: The MVP is now complete. The next step is to deploy it on Toolforge. Since my project involves both a React frontend and a Flask backend, deploying on Toolforge may be a bit complex, but I'll continue working on this even after my internship ends.
  • Blog Posts: Completed my final Outreachy blog posts, covering informal chats and wrapping up my experience.

Wrapping Up Blogs:
Learning from Tech Industry Professionals
Wrapping Up

https://medium.com/@agarwalmahima01 : See all my blogs here.

The deployment of the tool on Toolforge is still in progress and, unfortunately, won't be completed today. Although today is officially the last day of my Outreachy internship, I want to let you know that I will continue volunteering until the final deployment is complete and the tool is fully ready for use, along with all the necessary documentation.

The reason for the delay is that deploying a project that involves both a React frontend and a Flask backend on Toolforge is a bit complicated for me. This complexity is partly due to the lack of documentation specific to this kind of deployment. However, I am committed to figuring out a solution and will continue working on it until it's done.

Closing this task out, as Outreachy Round 28 has ended. Thanks for all your contributions to this project, @MahimaSinghal, and thanks to your mentors for being a fountain of knowledge - @Pablo, @CMyrick-WMF, and @Isaac! If there is additional work to do, please open another task and tag this one as reference.

Outreachy Round 29 is currently seeking projects and mentors, if there are remaining tasks on this project that you'd like to submit for Outreachy Round 29, please add them to T372834 and to the Outreachy site before September 11, 2024. Feel free to reach out with any questions, thanks!