Please find the instrumentation spec for mobile in the sub-ticket T411658. Feel free to update it if engineer has any new proposals.

@aude, thank you for answering the open questions.

@HFan-WMF , I have added on table for mobile in instrumentation spec, which covers the five measurement questions in the scope for T411430. I tried to maintain consistency between mobile and desktop, so if something is not instrumented on desktop, I assumed we would not instrument it on mobile either.
Please also refer to the instrumentation and workflow to help map the events to the buttons or links in the interface design.
Let me know if you have any questions or if anything is missing.

Status update:
Dashboards for both tier runs have been enabled with dummy metrics.
Here are the dashboard links

• 1st tier on Arabic, French, Vietnamese, Chinese, Indonesian wikipedia: https://superset.wikimedia.org/superset/dashboard/p/RLqO3MMoBYE/

Snapshot

• 2nd tier on English wikipedia: https://superset.wikimedia.org/superset/dashboard/p/5eEB8kkev9Y/

Snapshot

Status update:
Discovered that 2nd round AB test on enwiki changed the treatment group name from image-browsing-test to treatment. Updated the SQL so that it correctly pulls from both tiers of group results. Merge request.

Status update
Here are the drafts of the measurement plan and the instrumentation spec. For the user flow, please refer to Mobile section header Instrumentation and workflow.

Status update: Task is completed.

• The metrics are now enabled on the analytics dashboard on xlab superset. Metrics include:
  • Mobile web image browsing clickthrough rate per user
  • 2nd week retention rate ( both 7-day cohort and full experiment cohort)
  • 2nd day retention rate (both 7-day cohort and full experiment cohort)
  • 2nd day and 3rd day retention rate (both 7-day cohort and full experiment cohort)
  • 21-day cumulative retention (both 7-day cohort and 14-day cohort)
• The bug in CTR was fixed by merge

Here are the QA results based on the data collected between Nov 12 and Nov 19.

Status update:
Have prepared the configuration code (merge request). It's under review now.

Measurement plan and instrumentation spec will be tracked in T409163

Here are the QA results based on the data collected between Nov 3 and No 7. Everything PASSED the test.

As we discussed, I re-pulled a list of eligible users for the ReadingLists experiment using new criteria:

• User touch: within 3 months, between '20250801' and '20251101' (excluded)
• 0 watchlist from namespace 2 or 3
• 0 reading lists table across wikis
• 0 edits

Here is my code for your review.

The following analysis is based on data collected from English Wikipedia over a three-week period (October 11–31, 2025). The data represents a 1% sample of English Wikipedia traffic.

I have verified the events in the event.mediawiki_web_ui_actions schema. Here’s a summary:

• We started receiving events for 'ca-watch', 'ca-unwatch', 'menu.watch', and 'menu.unwatch' on Oct. 10.
• Events are being received from logged-in users on both mobile web and desktop.
• On enwiki, the average daily click event count is below 50, generated by fewer than 20 sessions. (sample rate 1%) I recommend starting the analysis after we’ve accumulated more data.
• Events are fired from both the article pages (namespace = 0) and the talk pages (namespace = 1). cc @HFan-WMF

Following the instrumentation info that @Jdlrobson-WMF mentioned in T401972#11096518, I didn’t find the event from logged-in users on enwiki in web_ui_actions schema. Instead, I only see menu.watch events from logged-out users on mobile web. Could anyone confirm whether the instrumentation is enabled as expected?

I have shared the lists of local user ids for arwiki, frwiki, viwiki, zhwiki, idwiki, enwiki at stat1010: /home/jiawang/share/T406388_reading_list_test_id/abtest_local_user_id .

Does this mean pages in the article namespace? and excludes the user and user talk pages?

Yes，namespace=0. the user and user talk pages are excluded.

Here are the drafted measurement plan and instrumentation spec for the team to review and discuss. For a visualized workflow, you can also refer to the instrumentation and workflow.

Here are the statistics for other 5 wikis.

• The percentage of unique active users with 0 articles on their watchlist and 0 items in their reading list is lower than that of enwiki.

Findings:
1. Number of users with 0 edits but at least one article on their watchlist

• About 5% of active users with 0 edits had at least one article added to their watchlist.
• No significant difference was observed when changing the definition of “active users” (based on having a user_touched event within 3 months, 6 months, or 1 year).

Summary of Analysis
Within 3 months

• Among logged-in users who had a user_touched event within the last 3 months on English Wikipedia:
  • Only 0.95% of users had an empty watchlist. The majority had their user page and user talk page on their watchlist.
  • Most users watched both the article page and its talk page.
  • 94.91% of users had 0 article on their watchlist.
  • We found a similar percentage (~95%) if the condition was defined as:“0 articles” , or “0 article or article talk page on watchlist,” or “0 watchlist items other than their user page and user talk page,” or “no more than 2 items on watchlist.”
  • 94.55% of users had 0 article on their watchlist and no reading list.

@HFan-WMF, Here are the summarized measurement questions based on the discussions on Slack or in google doc. Does this cover everything correctly?

• Number of users who have 0 edit but have at least one article added to their Watchlist
• How many sessions are using the watchstar button directly on articles on desktop?

Thanks, @Jdlrobson-WMF , for connecting me with the previous related research.

The experiments_registry.yaml file has been updated by the merge below.
Merge: https://gitlab.wikimedia.org/repos/product-analytics/experimentation-lab/experiment-analytics-configs/-/merge_requests/15

Hi, I have drafted the measurement plan and instrumentation spec for the reading list. Now passing this to the team to review and decide.
For a visualized workflow, you can also refer to the instrumentation and workflow.

We deployed the AA test on the targeted 50 wikis (T399486) on July 29. Due to a backport issue on August 1, traffic dropped significantly. The issue was fixed later that day. As a reminder, future analyses should use data starting from August 2.

Summarize the discussions about 3 observed limitations. We will accept as is.

1. Below fields are NULL

-performer.is_bot :
From Sam: Currently, performer.is_bot refers to whether the user (performer) is in the bot MediaWiki group. See https://www.mediawiki.org/wiki/Manual:Bots. With this definition in mind, it's likely that the field will be null for most visitors in your experiment.
-sample.rate
-sample.unit
Note: Current default analytics sample rate is 100% by default across all wikis.

Has configured the metrics in metrics_catalog.yaml and registration in experiments_registry.yaml.
Gitlab link: https://gitlab.wikimedia.org/repos/product-analytics/experimentation-lab/experiment-analytics-configs/-/commit/17d10bb5ae405b887f5cd8c187146be4c92dafad

@KSarabia-WMF has deployed AA test on testwiki (T399484) on July 24. Here are the initial data QA result.

• Net non-reverted edits

The data is presented in Table 5. Net Non-reverted Edits Year-over-Year Comparison in the ptwiki report.

Hi @egardner, I have drafted the instrumentation spec and measurement plan for our discussion next week.

I drafted a Presto SQL framework for CTR calculation per wiki in case the team wants to generate an initial result for the project's key metric while I’m out.

Here is the query framework for QAing the instrumentation deployment. Please replace the string_values with the values planned in column E of the instrumentation spec.

SELECT year,month, day,  COUNT(distinct performer.session_id) AS sessions,COUNT(1) AS events
FROM {table}
WHERE year=2025 and month={month} 
AND action='{string_value1}'
AND action_subtype = '{string_value2}'
AND action_source='{string_value3}'
AND NOT performer.is_logged_in
AND NOT performer.is_bot
AND mediawiki.skin='minerva'
AND agent.client_platform_family='mobile_browser'
AND mediawiki."database"='{wiki}'
GROUP BY year,month, day
ORDER BY year,month, day
LIMIT 10000

@ovasileva, I have completed the pre-post analysis of the donate banner on mobile. Here is the report draft for review. Let me know if you have any questions or suggestions. Happy to go over them in our next meeting as well.

Here is the investigation result: Metric summary 2025: non_reverted_edits.
In summary, a similar decline in net non-reverted content edits is still observed in FY2024/25.