Page MenuHomePhabricator

Definition of knowledge integrity risks
Closed, ResolvedPublic

Description

While the Wikipedia Knowledge Integrity Risk Observatory already contains several metrics following the categories of the proposed taxonomy, there is an interest to build (one/few) risk metrics to allow observatory stakeholders to take preventive actions when Wikimedia projects are under risk in the knowledge integrity space.

This take will focus on defining specific risks that will then be measured quantitatively.

Event Timeline

Weekly updates

  • Initial brainstorming on risks to capture with quantitative metrics.
  • Call with UW researchers: they have started building a theoretical framework of disinformation risks for small Wikipedias that largely overlaps with the Knowledge Integrity Risk Taxonomy. Commo interest in a collaboration on a quantitative empirical study using the result of the frawework + knowledge integrity risk metrics.

Weekly updates

  • To identify possible preventive workflows, academic literature on knowledge integrity issues on Wikipedia has been mapped through the Wikimedia Research Newsletter, the feed of Wikimedia Research Twitter account and Google Scholar. The collection contains ~100 papers grouped into the following categories: vandalism, non-verifiability, biases, promotion, sockpuppetry, false information.

Weekly updates

  • Fruitful meeting with the Disinformation Manager in which we identified the workflows of his team in which it is more relevant to provide a Knowledge Integrity Risk Metric. Quarterly coordination meetings will start to be held between both teams.
  • Progress of Q1 was shared in the Disinfo & Government Regulation Quarterly Learning Session and the Disinformation Working Group Call.

Weekly updates

  • Meeting with the PI of the Community Health Metrics project to discuss community metrics to reflect structural issues that might affect knowledge integrity.
  • Progress of Q1 was shared internally to be considered in Metrics that Matter.
  • Work on the draft of the position paper, that has resulted into a first definition of knowledge integrity

Weekly updates

  • Quaterly neeting with all the members of the Disinformation team to discuss how to integrate the risk observatory on their deep-dive workflow. It was decided that a tutorial session will be scheduled in order to provide them with skills to operate the existing Superset dashboard.
  • Quaterly meeting with the UW researchers working on disinformation risks. They shared results from their last interviews and discussed next steps (including the mutual interest in a collaboration).
  • Work to complete the dataset of spambot deleted revisions to be able to create a specific spambot detection model under the new version of ORES

As later reported in T321947, knowledge integrity risks will be captured through the volume of high-risk revisions (based on the language-agnostic revert risk model) and their revert ratios.