Dashboard is updated with production data; events and menuItem data is showing up as expected. Will collect data and document analysis ~2021-10-01.
Wed, Sep 15
Dashboard for preliminary data validation. Charts query event.MobileWikiAppNavMenu for show and select events and count unique users daily.
Tue, Sep 14
Mon, Sep 13
Can we add analytics for interactions with Talk pages and Watchlist notifications? Can add to the MobileWikiAppNotificationInteraction schema. Let's discuss if this is possible.
Can we add analytics for interactions with these settings? We can add events to [[ `https://meta.wikimedia.org/wiki/Schema:MobileWikiAppNotificationPreferences | MobileWikiAppNotificationPreferences ]] schema. Let's discuss if this is possible,
Fri, Sep 10
In order to collect edit/editor browser usage we need to use 2 tables, neither contains ALL data but we can approximate using proportions for iOS/Mac OS Safari users and use counts for all Safari users as baselines. Using sampled data (1:16) from EditAttemptStep we can get the proportion of edits and editors using Safari on iOS and OS X for the past 90 days. Actual edit/editor counts requires data from wmf_raw.mediawiki_private_cu_changes table, which does not have wiki edit data. I'm verifying the data I pulled with my team but can answer @Maryana's questions with what I have now.
Tue, Sep 7
Fri, Sep 3
Not related to ABC test but I have a question about non-Notifications setting items that are on the Settings 3 on the Figma diagram. There are options to change settings for "Syncing Reading Lists" and "Downloading articles", these settings are also accessible from the Settings Menu. Will need to confirm that we can track when they are accessed from the new Nav Bar UI but this may not be the case.
Wed, Sep 1
Tue, Aug 31
Verified that version 2.7.50372-beta-2021-08-26 has resolved errors previously seen in MobileWikiAppNotificationInteraction data. Only errors in Logstash are only for version 2.7.50370.
Thu, Aug 26
Wed, Aug 25
We will need to add the following hooks for analytics to the MobileWikiAppNotificationInteraction and MobileWikiAppNotificationPreferences schemas. (I'm still working on a new schema set for this but it may not be ready by the time we roll out the test).
Tue, Aug 24
Taking a look at Logstash for other schema errors, as @cooltey noted above, there are quite a few in different schemas: (Links to Logstash for each schema below):
Mon, Aug 23
Sat, Aug 21
Aug 18 2021
For data QA will be comparing directional trend between MEP and Legacy data using sample daily retention rates, distribution of user`appInstallAgeDays` over time and distribution of is_anon true/false over time.
Aug 13 2021
Aug 11 2021
Note: A change was made in app code (version 2.7.50370-r-2021-08-03) that looks to have been effective in fixing an issue where users were not seeing notifications as expected. A clear increase in notification events post-release can be seen as of 2020-08-03.
|90 Day Count Unique Editors per Group:|
|English Wikipedia editors in India and Nigeria||3974|
|Hindi Wikipedia editors||718|
|Arabic, Moroccan Arabic and French Wikipedia editors in Morocco, Egypt, DR Congo and Mali||350|
|Indonesian Wikipedia editors||627|
|Japanese Wikipedia Editors||695|
Aug 10 2021
Following discussion with @Sharvaniharan and @Dbrant the plan for this ABC test is to use the MobileWikiAppNavMenu schema to track views and clicks for each variant Notification icon. Since we can't target UI by geolocation/wiki specifically, we will filter the results by wiki and geocoded_data based on the feedback criteria groups at the tail end.
This issue was due to a low row limit setting in Superset. I increased the row limit to 50,000 in the Chart settings, data counts seem consistent now.
Did not see any data errors in Logstash
Aug 6 2021
Aug 2 2021
Jul 28 2021
Verified we are seeing data for app version 2.7.50366-beta-2021-07-26. Note that there is an artifact column that we aren't using incoming_type, the data should be, and is, populated in notification_type instead. Intitial data.
Jul 26 2021
Done. MobileWikiAppNotificationInteraction is updated.
Jul 21 2021
Jul 20 2021
Jul 16 2021
The MobileWikiAppNotificationInteraction schema was designed to only measure counts of notification interactions, not all notifications sent. The counts we are seeing (90 day sample) are also concerning because they are lower than expected. In order to get baseline data we need, the following actions are being planned with @Sharvaniharan and @Dbrant:
Jul 13 2021
Jul 2 2021
Current Notification Preference Settings (need to be added to new UserSetting schema):
- Poll notifications (allow the app to use data to check for new notifications in the background)
- System messages
- Milestone (Certain edit counts are reached)
- Thanks (Someone thanks you for an edit)
- Revert (One of your contributions was reverted)
- Talk page (Messages from talk pages)
- Login (Your login activities)
- Mention (Someone mentions you in a page)
@SLien_WMF Yes a registered editor is an editor who has created an account and made an edit.
Current instrumentation is missing a number of setting/preference events, we will add these events to the new SettingAction schema and track user state on the new UserState schema to be detailed in https://phabricator.wikimedia.org/T285779. That ticket will track schema planning and instrumentation, this ticket will track Baseline metric questions/answers.
Current schema MobileWikiAppNotificationPreferences is outdated and missing current Notification Preference options in tracking.
Jul 1 2021
This was resolved by @mpopov's Baseline Metrics reporting which is now deprecated. We will revisit the need for the metrics listed in this request when we develop a reporting replacement in the future.
Note for item : We want to see the user name if they opt-in to showing it
Out of curiosity I ran user retention analysis on active users from each table (date range 2021-05-06 to 2021-06-24). Directionally the results are fairly close. Keep in mind that the Image Recs task has a lower return rate than regular editing tasks.
Jun 30 2021
Jun 29 2021
Jun 25 2021
- The suggestion sources in Legacy are:
Jun 22 2021
Jun 17 2021
Jun 15 2021
Thanks for this. @Mholloway I did notice some anomalies using query  for event counts previously, I use that as a first pass query but it may be that it needs to be refined. Looking at the parity between app_install_ids in both tables is a much more reliable way to assess if data is missing so I generally don't rely on results from query .
Jun 9 2021
Hi all - Please hold off on deleting the schema until we are done analyzing the data, just want to emphasize I am still using the schema data.
Jun 8 2021
For tables event.android_image_recommendation_interaction and event.mobilewikiappimagerecommendations. Seeing event count discrepancies and missing data from both tables, significantly more missing on MEP side. As of 2021-06-07:
Jun 4 2021
As of 2021-06-03 we have 3815 images with 3 or more annotations (placeholder images are excluded from this count) see spreadsheet . Please let me know stop date for experiment so I can begin answering topline questions about user engagement.