Wikidata Analytics Request
This task was generated using the Wikidata Analytics request form. Please use the task templates linked on our project page to create tasks for the team. Thank you!
Purpose
Please provide as much context as possible as well as what the produced insights or services will be used for.
The assumption is that ReferenceCheck has created a demand for references that the current tooling for the citation experience cannot support cleanly. So in essence, ReferenceCheck worked and now we owe newcomers intuitive tools to actually follow through. This will inform opportunities TechWish will work on.
To validate this assumption we want to first look at the numbers for the behavior of 3 groups of newcomers:
- Newcomers who keep adding references via ReferenceCheck prompts
- Newcomers who have seen ReferenceCheck at least once and now add references independantly
- Newcomers who have NEVER seen ReferenceCheck and have always added references independantly.
As a second step we want to identify where in the user flow do newcomers fail when attempting to add a reference and what are the failure reasons.
We will want to distinguish between:
- Failures that lead to NO net-new-reference in a session : 'We expected a reference, but there was no net-new-reference and this was the problematic step.'
- Friction signals in a session: 'We found a net-new-reference, but we also saw the user struggle to create it'
- ideally we also expand this analysis to editing references not just creating new references.
Criteria:
- Newcomer : edit count < 100
- Surface: Mobile vs Desktop
- Mobile using mobile UI, mobile using desktop UI, desktop using desktop UI and other
- Scope is for all citation-related events within a single VisualEditor editing session. Out of scope is sub-referencing
Desired Outputs
The desired outputs of this task are listed as check boxes and confirmed as being finished below.
- A dashboard or single charts that cover the aspects above.
Deadline
Please make the time sensitivity of this request clear with a date that it should be completed by. If there is no specific date, then the task will be triaged based on its priority.
15.04.2026
Information below this point is filled out by the task assignee.
Assignee Planning
Sub Tasks
A full breakdown of the steps to complete this task.
- Check process needs with stakeholders
- Monthly as we're using mediawiki_history
- 3 groups of newcomers
- We want the counts and percents of these groups per wiki
- We need some kind of boolean flags table that has whether the editor has seen ReferenceCheck
- Two failure/friction criteria:
- We expected a new reference via editcheck-references from (Reference_check_Tags)
- Failures that lead to NO net-new-reference in a session [and the steps that determined failure]
- Friction signals [steps] in a session
- Ideality: User does an edit that requires a reference and adds it or is prompted by ReferenceCheck and adds it
- How many users are actually on these ideal paths
- If a large proportion are following an ideal path, then we could do a direct comparison between those that do follow the ideal path and those that don't
- Where from the ideal paths do they drop off, and what's the percentage chance that a drop off at a particular step means that it's a failure where there is no new reference vs. a friction where we do end up getting a new reference
- Prepare presentations of steps and drop off/friction behavior in preparation for discussion about what has value to track long term
- Investigating events via the playground
- Write needed queries to derive metrics
- For groups
- For funnel
- Write DAG to compute metrics
- For groups
- For funnel
- Test queries and DAGs
- Deploy DAGs
- Create Superset dashboard
- Charts for 3 groups of newcomers (one aggregate and and percent)
- Ideally with the ability to compare wikis (maybe two charts that can be filtered for other wikis - for example)
- Aggregate and percent failed new references based on expectation
- WIP - to be discussed in April after getting initial results: Charts to display the proportion of a user being able to add a new reference after dropping off at any given step in the process
- Charts for 3 groups of newcomers (one aggregate and and percent)
Estimation
Estimate:
Actual:
Data
The tables that will be referenced in this task and the samples from them that will be used.
- link_to_table
- sample_size
Notes
Things that came up during the completion of this task, questions to be answered and follow up tasks.
- Note