Page MenuHomePhabricator

Build new scraper to get better bias data
Closed, ResolvedPublic

Description

Currently, the MediaBiasFactCheck data we are getting from another scraper (https://github.com/drmikecrowe/mbfcext) distinguishes sites into "left", "left-center", "center", "right", and "right-center". However, this scraper is missing MediaBiasFactCheck's extreme-left and extreme-right ratings.

We need to build a new scraper that pulls the extreme-left and extreme-right sources so that we can distinguish them from the rest. This can be done by analyzing the URL of each source's scale image (which includes a rating and number, such as left2 or extremeright3).

This new scraper will also make our script more efficient as we won't be pulling down extra data we don't need.

Event Timeline

SuperHamster moved this task from Backlog to In Progress on the Cite-Unseen board.
SuperHamster updated the task description. (Show Details)
SuperHamster triaged this task as Medium priority.
SuperHamster moved this task from In Progress to Done on the Cite-Unseen board.