As a product manager, I want to define measurable success criteria for improved changelogs so that we can track the impact of our changes.

Context
This epic is one of the initiatives required for eventually turning on Wikidata changes in Wikipedia by default. Therefore, it is important for us to have some definitions of metrics to help us monitor the comprehension and engagement of the new changelogs.

We want to qualitatively and quantitatively measure whether editors have found the new changelogs better than the old ones.

Acceptance Criteria:

1. A list of key metrics to track comprehension and engagement of editors encountering changelogs overtime

Metric 1: % Editors who click on the P and Q numbers in their Wikipedia watchlist. This means that we figure out the number of people who click on P and Q numbers. These should decrease if we have made the edit summaries and changelogs more comprehensive. Compare the % before and after implementation. There should be a decrease over time.
Metric 2: % increase in the overall number of editors who turn on the Wikidata changes in watchlist functionality over time. We already have a list of the state of the function from March 2025, and Andrew will automate this over time, but we can always ask for monthly numbers.

2. A plan for gathering user feedback on the proposal for improving changelogs now and in future.

Checklist

• Work with UX to determine how to collect qualitative feedback from editors.
• Identify key user behaviours that indicate improved comprehension (e.g., comprehension of edit summaries). Which of these methods are feasible, given our tracking infrastructure?
  • The number of people who turn on Wikidata changes in their Wikipedia preferences over time. OKR for Collab agreement 25/26 - 3 wikis should turn these on by default by June 2026.
• Work with UX to design usability testing sessions to compare current vs. improved changelogs.
• Develop a tracking plan to monitor the effectiveness of changes over time.
  • How can we monitor the effectiveness of our changes over time?
  • Select pilot wikis
  • Confirm pilot wikis
• Book in a knowledge sharing session on the engineering side of tracking
  • Create tracking ticket for Engineers

*Note: we cannot use A/B testing because one database holds all the changes. Therefore, it is impossible for different editors to see different types of change formats.