Page MenuHomePhabricator

[FY24/25] WE.2.1/2.5 KR evaluations and measurements
Closed, ResolvedPublic

Description

Description

This task captures the effort of evaluating the outcomes of the Content Coverage key result, which is part of the Knowledge Equity objective area for FY24/25.

Key Result (WE.2.5/KE 1): By the end of Q4, support organizers, contributors, and institutions to increase the coverage of quality content in key topic areas, i.e. Gender (women's health, women's biographies), and Geography (biodiversity) by 138 articles through experiments.

Other Details

The initial measurement of success was identified as below;
1. Increasing the coverage of content.

  • Number of articles/pages translated through topic selection, collections & list-building service experiments.
  • Number and % of successful (i.e., non-deleted) translations.
  • Number of key topic areas of articles translated.
  • Distribution of key topic areas in articles translated.

2. Improving the quality of content.

  • Number of edits made through topic selection, collections & list-building service experiments.
  • Number of articles/pages that are edited or improved.
  • Number of bytes: words, references, links added.
  • Number of images added.
  • Number of key topic areas improved/ edited.
  • Distribution of key topic areas in articles improved/edited.

3. Other metrics:

  • Number of editors that translate are generated through topic selection, collections & list-building as a service experiments.
  • Number of pageviews/readership of content through topic selection, collections & list-building as a service experiments.
Baseline Calculation

While doing the baseline calculations, T362212, we determined the count of 138 quality articles by examining individual wikis. For the KR evaluation, we need to check the number of translated articles generated through topic and collection suggestions, then identify the quality levels, impacted wikis and topics, involved editors, and readership characteristics of these translated articles.

Omissions

As of May 2025 (Q4), list-building as a service experiments that was an experiment under 2.1/2.5 KR did not have a direct contribution to the collections added to the CX dashboard. This aspect should be omitted from the analysis; it will be included in the overall KR learnings at the end of the 6 months.

Event Timeline

PWaigi-WMF moved this task from Backlog to Prioritized on the LPL Hypothesis board.
KCVelaga_WMF changed the task status from Open to In Progress.May 23 2025, 5:55 AM
KCVelaga_WMF moved this task from Prioritized to In-progress on the LPL Hypothesis board.
KCVelaga_WMF moved this task from Incoming to In progress on the LPL Analytics board.
KCVelaga_WMF moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

Interim stats

  • ~200 articles were translated using the custom suggestions feature
    • arwiki has the highest usage (~40% of the translations)
    • translations across 29 different languages
  • Primary topics of translated articles
    • Culture-misc: 67 articles (34.4%)
    • Geography: 43 articles (22.1%)
    • Culture-biography: 33 articles (16.9%)
    • STEM: 29 articles (14.9%)
    • History_and_Society: 23 articles (11.8%)
  • Top 3 sub-topics
    • Culture.Philosophy_and_religion: 43 articles
    • Culture.Biography.Biography*: 33 articles
    • STEM.STEM*: 21 articles
  • 25 unique daily users on average
  • Topic vs Collection distribution (for the published articles)
    • Topic areas: 58%
    • Search result (within custom suggestions menu): 24%
    • Collections: 17%

We are most likely under-counting. Last week, when I started data gathering, we identified that the some of the major events were not being logged after the unified dashboard was released to desktop in March (T395493). We capture the target page ID and title for the translation, at the publish_success event (i.e. when an article is published, it would have an identifier, but not when the translation in progress). As that event was not being logged, we couldn't exactly get the published pages. The workaround for now is, we know source article, language and along with target language when users start the translation. So we are looking at the Wikidata item of the respective articles and check if an article is linked to the target language, if it was created post the start date and has cx tag. While this is a good workaround, if the target article is not linked Wikidata item for any reason, we will miss those.

Once the issue is fixed, we can re-run the numbers at the end of the quarter. However, the old data is lost, these is no way to backfill that. We will combine any stats from the new data with above counts.