WMDE Analytics Request

As we work on improving the citation experience for contributors, we want to set up additional metrics to track how well we're serving them. This work is part of Technical Wishes' annual KRs to increase newcomer retention.

Why (Context & Decision)

Citoid's automatic reference maker currently lacks support for some source types, which leads users to create a citation manually. When this happens, users, and especially newcomers, are faced with a difficult, unintuitive experience. We want to decrease the number of users going that deep into the funnel.

An example of missing capabilities for Citoid's automatic reference maker is the creation of properly formatted citations from PDF sources. We do not have full support at the moment, and thus links to PDF sources where the hosting site does not return metadata as HTML will return an error and then prompt the user to 'Add manually'.

Logstash shows an average of 1.3K failed attempts to create an automatic citation—this is the equivalent to 'Add manually' in the Visual Editor UI. The majority, if not all, are links to PDF sources.

With that said, our hypothesis is:

• If we build full support for PDF sources we can decrease the number of manual creations to increase nr of reference edits AND newcomer retention. (the last two metrics are already available via airflow dags)

What (Scope & Output)

Describe the specific question(s), metrics, segments, or deliverables (e.g., dashboard, deep dive, experiment analysis), including any relevant definitions or constraints.

• Number of manual creations via Visual Editor over time.
  • This can be by clicking on the 'Add manually' button, leading up to a citation creation
  • Clicking on the manual tab in Citoid, leading up to a citation creation
  • events from visualeditorfeatureuse (feature.action)
    • Add manually: cite . automatic-generate-manual-fallback
    • Selection in manual tab: citoid . manual-choose
    • Insert citation: cite . dialog-done
    • see patch demo for all instrumentations : https://796f874949.catalyst.wmcloud.org/w/index.php?title=Main_Page&veaction=edit

• Number of manual creations via Visual Editor by user type over time.
• Proportion of manual citations (choose, fallback) vs automatically generated
  • events for automatic generations:
    • citoid . automatic-generate
    • citoid . automatic-insert
    • citoid . dialog-automatic-insert

Criteria:

• Wiki scope: all wikis
• data retention: 90 days
• user types:
  • logged-in and temp
  • newcomers (<100 edits), experienced editors (>= 100 edits)

• Output
  • superset charts

By When (Timing & Priority)

Provide a clear deadline, any key milestones (e.g. launch, leadership review), and note if timing is flexible or fixed.

Deadline: 24.08.2026. Timing is fixed as we need this data for potential KR adjustments and for annual planning that starts in September.

---

Information below this point is filled out by the analyst.

Sub Tasks

A breakdown of the steps to complete this task.

• Check scope of the task with stakeholder
• Write job queries
  • Remove unused condition flags from wikipedia_newcomer_ve_reference_funnel_monthly query
• Create an exploration notebook for @catherine.kelsey.wmde and @AndrewTavis_WMDE to use in exploring the potential paths of the funnel and other aspects of the queries
• Write Airflow DAG
• Test job queries
• Test and deploy Airflow DAG
• Create Superset dashboard

Data

The tables that will be referenced in this task and the sample sizes from them that will be used.

• event.visualeditorfeatureuse