Page MenuHomePhabricator

Data Validation and Analysis for ABC Test
Closed, ResolvedPublic

Description

  • Validate incoming data for ABC test
    • Data in schema event.MobileWikiAppNavMenu
  • Analyze results of ABC test data
    • Results for target user populations
    • Results for overall population

Related Objects

Event Timeline

Dashboard for preliminary data validation. Charts query event.MobileWikiAppNavMenu for show and select events and count unique users daily.

android-notification-abc-test-2021-09-15T16-03-00.793Z.jpg (1×1 px, 134 KB)

LGoto triaged this task as Medium priority.Sep 15 2021, 4:06 PM

Dashboard is updated with production data; events and menuItem data is showing up as expected. Will collect data and document analysis ~2021-10-01.

android-notification-abc-test-2021-09-17T19-21-56.652Z.jpg (1×1 px, 181 KB)

ABC test results are charted in deck. Additional analysis looks at NotificationInteraction increases and edit counts by test group.

Distribution of Notification Icon interaction frequencies by group:

Test Groupbellinboxmore
English Wikipedia editors in India and Nigeria32.33%37.63%30.05%
Arabic and French Wikipedia edits from Morocco, Egypt, DR Congo and Mali34.10%35.34%30.56%
Hindi Wikipedia edits30.67%37.52%31.81%
Indonesian Wikipedia edits31.56%34.81%33.63%
Japanese Wikipedia Edits34.07%44.99%20.95%
All Test Users33.68%37.90%28.43%

Difference between Bell vs Inbox Frequency ranges from 1 to 10%, depending on test group, difference for all users in test was 4.2%

SNowick_WMF raised the priority of this task from Medium to High.Oct 14 2021, 9:09 PM
SNowick_WMF moved this task from Doing to Needs Review on the Product-Analytics (Kanban) board.

Added baseline user vs test user count differences to deck: link

In summary:

All Users
Baseline users, % of DAU.19%
Test users, % of DAU.77%
% Difference:121.4%

Breakdown by target user groups in slide.

Also added editor notification interaction data by sub-group. Data, slide.

SNowick_WMF lowered the priority of this task from High to Medium.Oct 25 2021, 4:38 PM