Thu, Oct 22
@schoenbaechler Hi Robin - I was able to find out why counts for results are higher than start in the MobileWikiAppSearch schema. For every start there are 2 searches initiated: one for title and and one for full text, so when results are returned they are automatically double the start events. Additionally when a user scrolls to see more results that counts as a result event as well, and this can generate multiple counts while scrolling. When we move to the MEP platform we may consider adding a singular result_returned event separate from these counts so that we are able to do a 1:1 start vs result event comparison, as it stands now the results count doesn't allow for that.
@schoenbaechler We do have a schema to track user exposure to CTAs and secondary CTAs https://phabricator.wikimedia.org/T258080 but that has been de-prioritized while we re-org the analytics workflow for Android. Agree with the assumption that CTAs are motivating returning editors.
An analysis of editor activity between 2020-08-01 and 2020-09-31 showed the following editor retention percentages.
(An editor counts as retained if they were active on or after a specific day cohort, these are averages across all active editors.)
Wed, Oct 21
- Agree we should track both of these values
@cooltey Thanks, I've updated the schema https://meta.wikimedia.org/wiki/Schema:MobileWikiAppAppearanceSettings. Please let me know when we will be ready to QA the data.
@Tsevener Those test group names are fine, if you want to shorten them we could use LivingDoc_Test and LivingDoc_Control but if length doesn't matter let's go with what you prefer.
Tue, Oct 20
Since we don't have a distinct source for image tags from Commons filepage we will count these new image tags with the regular image tags as Suggested Edit actions. Closing ticket.
Mon, Oct 19
As of 2020-10-19 we have a month's worth of users in both test groups, ~7000 users total. Can start analysis on this depending on priority.
Dashboards on Superset:
Fri, Oct 16
More findings, I checked the rest of the iOS data schemas for events and records for the 12 app_install_ids that show up on Modern and not Legacy for thank_try and found that only 4 of the 12 app_install_ids appear in other tables. I've added those ids/events to the spreadsheet but it looks like most of these users were not getting tracked on other tables and none of them show up in mobilewikiappdailystats
Thu, Oct 15
Wed, Oct 14
@Tsevener to clarify MEP meta.dt is equal to Legacy dt, MEP client_dt is equal to Legacy event.event_dt (on Android it's event.client_dt)
Tue, Oct 13
Mon, Oct 12
I made spreadsheets with events for 2 actions show_history and compare1 on both Legacy and Modern tables. First page of sheets is data from Modern and the most left column with dt from Legacy to compare events by initial timestamp. Rows with blank legacy dt field are events that only appear in Modern, rows with blank meta.dt field are events that only appear in Legacy.
Fri, Oct 9
For the events where we have anomalies I am seeing counts off on both tables, there is still not a consistent pattern I am seeing but I am preparing some analysis that I will go over with @Tsevener and @mpopov when it's ready. I also ran the validation query again and we still are seeing higher counts on legacy table.
List of pages for survey.
Tue, Oct 6
Mon, Oct 5
Sam -the closest I have gotten to finding any info on the deleted pages was using logging.log_id to query archive and revision using archive.ar_page_id = logging.log_id and logging.log_id=revision.rev_page. (I added log_id to your query and made a spreadsheet with results here.
@Tsevener Just to clarify, by modal do you mean the survey itself? So adding survey_click and survey_cancel?
Fri, Oct 2
Additional data QA process info here: https://phabricator.wikimedia.org/T264193
Thu, Oct 1
I'll pull the data and go through it to see if we can verify any of these possible scenarios, stay tuned.
Wed, Sep 30
@mpopov I think we are ok without the extra edit subdirectory for now, we have a few schemas that combine editing with other data I wouldn't want to get lost. Let's go with /analytics/mobile_app/ios/edit_history_compare
@Charlotte @Dbrant @cooltey
Looking at total accounts w/ action = success in the MobileWikiAppCreateAccount table I'm seeing a 39% drop in total accounts post 2020-08-05. Did something change with the counting mechanism. We went from and avg. 1400 success daily to avg. 842 daily. Charts Here
Tue, Sep 29
Mon, Sep 28
Confirming the latest version we are seeing in the mobilewikiappsuggestededitsfeed table is 2.7.50332-beta-2020-09-28
Was waiting for image tag data but since they are Wikidata items and are translated by Wikidata the entry language is not tracked. Dashboard ends at last month since we use mediawiki_history to collect this aggregated data.
Dashboard is here, calculations for retention and SE interactions were done off Superset.
Fri, Sep 25
Sep 22 2020
Met to discuss flow and metrics. Added back the card_editor, card_discuss, card_change and card_more metrics, added card_multiple for multiple elements shown on card.
Sep 21 2020
@cmadeo just to confirm - we want to consolidate all the links on the card (card_editor, card_discuss, card_change and card_more) to just be one metric (user clicks) - I will update docs, using card_click for metric name. Just want to be sure I'm not missing any detail we might want to keep measured separately.
Sep 14 2020
Sep 10 2020
This is resolved, data is on Dashboard Android Image Tags
Sep 1 2020
Data is updated on Registered editors and Wikipedia editors by month spreadsheet with corrected editor counts. Note the counts that changed were the Wikipedia editors, the average monthly Wikipedia editor count for the last 12 months is 285798. The counts for all project editors have not changed.
Android Build 2.7.50318-r-2020-05-18
iOS Build 6.6.1 1743 2020-06-05
Android Build 2.7.50318-r-2020-05-18
iOS Build 6.6.1 1743 2020-06-05
Aug 31 2020
Hi @nshahquinn-wmf Presto query below. It does look like my percentages are higher than your past data, there might be something missing in how I differentiated all vs Wikipedia edits. I used wikis ending with regex wiki to get Wikipedia edits.
select month, count(*) as active_editors from ( select cast(month as date) as month, user_name, sum(edits) as all_edits, max(bot_by_group) as bot_by_group, cast(date_trunc('month',min(user_registration)) as date) as registration_month from neilpquinn.editor_month where month > TIMESTAMP '2019-01-01 00:00:00.000' and user_id != 0 AND regexp_like(wiki, 'wiki$') group by month, user_name ) global_edits where not bot_by_group and not regexp_like(user_name, 'bot\\b')
Aug 27 2020
Looks good, queries are working now, thanks!
Aug 26 2020
Aug 25 2020
Aug 21 2020
Aug 20 2020
Aug 19 2020
Thanks @Dbrant, should the event data for Explore and Home be in the MobileWikiAppNavMenu table? Not seeing any events for Home or Explore. I did a SELECT distinct(event.menuItem) FROM MobileWikiAppNavMenu query and got the following:
- Write incident report, stored in Product Analytics shared drive 'Data Incident Reports"
- Documenting fix process in ticket (there were several different changes)
- Noting dates and app versions that were affected - for future data knowledge, how do we exclude/include where data was missing. Where can we put those to make sure they aren't lost?
- Estimating lost views and validating that fix is actually working (depends on new app version saturation) Estimated App Pageviews
- Who else needs that estimated data? CChen: Key product metrics, which goes into quarterly deck, publish key metrics on Commons. Also CChen for Superset. @cchen has estimated pageview counts
Aug 12 2020
If the captions are finalized can we have a flow chart of user actions - where the user could potentially encounter and what happens for each possible action? It is difficult to write a schema without clear visualization of user flow.
This looks good to me. Signing off as approved.
Use an A/B test to see if these warnings reduce the revert rate and/or retention rate and editing volume. @Dbrant let's discuss analytics for this.