Data QA summary of logged-in retention instrument
Table: event.mediawiki_product_metrics_reader_retention_logged_in

I’ve reviewed and revised the measurement plan and spec for article preview. The major changes are:

1. Metrics and their categorization, based on the discussion with Sherry
2. Stream and event names, updated to follow the experiment platform team’s guidelines

I have added my comments and updates in the share feature measurement plan and instrumentation spec. A few open questions:

@SherryYang-WMF , Just to confirm, are we planning to measure Session length? It’s not mentioned in the measurement plan but is listed in the ticket. If so, what do we expect to see?

For documentation purposes, we ultimately decided to set up the dashboard in GrowthBook with support from the Experiment Platform team. Here is the dashboard link: https://growthbook.wikimedia.org/experiment/exp_6v4betpmmnu69x2#results

Here is the measurement plan and instrumentation spec for the account creation Reading List CTA. Please review and feel free to add your input.

@HFan-WMF , The list includes the top wikis other than enwiki, and looks good to me. One thing to note is that ptwiki requires users to be logged in to edit, so its registration rate may differ from others. It’s worth reviewing the results separately.

Please find the draft analysis of the second baseline. It is ready for review.

Does this include temp accounts? I see performer_is_temp in the spec as a contextual attributes.

We want to analyze users with logged_in = true and performer_is_temp = false. It’s unclear whether temporary accounts will be bucketed into the experiment. If they are, we will filter them out in the analysis using the performer_is_temp field.

Sorry for the confusion. To clarify:
For the Analytics_sampling:
Sample_unit: performer_session_id
Sample rate: 100%

If we use test kitcken, the Sample unit should be wm-user , not  session .

Here is the draft of the instrumentation spec. Pass it to the engineer for review.

Here is the draft of the analysis report , ready for review.

Status update:
Here are the Mobile Web ToC Measurement Plan and Instrumentation Spec. Ready for product team review.

Both skin preference and global preference reflect the status as of the data collection date, which is January 28, 2026.

Status update:
Here is the initial draft of the image browsing usage analysis. Have shared with the team for review.

• Defined metrics in metrics_catalog.yaml

Have defined retention metrics for desktop web using the same definition as mobile web retention. They passed the test in local notebook.
Merge request link.

It has been completed in T406013 and T410794.

Here are the QA results based on the data collected between Dec 12 and Dec 18.

{F71132115}

Status update:
Analytics dashboard has been set up for tier 1 AB test.
Metrics include:

• Mobile web session length (seconds)
• Mobile web retention

Please find the instrumentation spec for mobile in the sub-ticket T411658. Feel free to update it if engineer has any new proposals.

@aude, thank you for answering the open questions.

@HFan-WMF , I have added a table for mobile in instrumentation spec, which covers the five measurement questions in the scope for T411430. I tried to maintain consistency between mobile and desktop, so if something is not instrumented on desktop, I assumed we would not instrument it on mobile either.
Please also refer to the instrumentation and workflow to help map the events to the buttons or links in the interface design.
Let me know if you have any questions or if anything is missing.

Status update:
Dashboards for both tier runs have been enabled with dummy metrics.
Here are the dashboard links

• 1st tier on Arabic, French, Vietnamese, Chinese, Indonesian wikipedia: https://superset.wikimedia.org/superset/dashboard/p/RLqO3MMoBYE/

Snapshot

• 2nd tier on English wikipedia: https://superset.wikimedia.org/superset/dashboard/p/5eEB8kkev9Y/

Merge request
Snapshot

Status update:
Discovered that 2nd round AB test on enwiki changed the treatment group name from image-browsing-test to treatment. Updated the SQL so that it correctly pulls from both tiers of group results. Merge request.

Status update
Here are the drafts of the measurement plan and the instrumentation spec. For the user flow, please refer to Mobile section header Instrumentation and workflow.

Status update: Task is completed.

• The metrics are now enabled on the analytics dashboard on xlab superset. Metrics include:
  • Mobile web image browsing clickthrough rate per user
  • 2nd week retention rate ( both 7-day cohort and full experiment cohort)
  • 2nd day retention rate (both 7-day cohort and full experiment cohort)
  • 2nd day and 3rd day retention rate (both 7-day cohort and full experiment cohort)
  • 21-day cumulative retention (both 7-day cohort and 14-day cohort)
• The bug in CTR was fixed by merge

Here are the QA results based on the data collected between Nov 12 and Nov 19.

Status update:
Have prepared the configuration code (merge request). It's under review now.

Measurement plan and instrumentation spec will be tracked in T409163

Here are the QA results based on the data collected between Nov 3 and Nov 7. Everything PASSED the test.

As we discussed, I re-pulled a list of eligible users for the ReadingLists experiment using new criteria:

• User touch: within 3 months, between '20250801' and '20251101' (excluded)
• 0 watchlist from namespace 2 or 3
• 0 reading lists table across wikis
• 0 edits

Here is my code for your review.

The following analysis is based on data collected from English Wikipedia over a three-week period (October 11–31, 2025). The data represents a 1% sample of English Wikipedia traffic.

I have verified the events in the event.mediawiki_web_ui_actions schema. Here’s a summary:

• We started receiving events for 'ca-watch', 'ca-unwatch', 'menu.watch', and 'menu.unwatch' on Oct. 10.
• Events are being received from logged-in users on both mobile web and desktop.
• On enwiki, the average daily click event count is below 50, generated by fewer than 20 sessions. (sample rate 1%) I recommend starting the analysis after we’ve accumulated more data.
• Events are fired from both the article pages (namespace = 0) and the talk pages (namespace = 1). cc @HFan-WMF

Following the instrumentation info that @Jdlrobson-WMF mentioned in T401972#11096518, I didn’t find the event from logged-in users on enwiki in web_ui_actions schema. Instead, I only see menu.watch events from logged-out users on mobile web. Could anyone confirm whether the instrumentation is enabled as expected?

I have shared the lists of local user ids for arwiki, frwiki, viwiki, zhwiki, idwiki, enwiki at stat1010: /home/jiawang/share/T406388_reading_list_test_id/abtest_local_user_id .

Does this mean pages in the article namespace? and excludes the user and user talk pages?

Yes，namespace=0. the user and user talk pages are excluded.