Page MenuHomePhabricator

Analytics for account creation via Suggested edits
Open, MediumPublic

Description

  • # of new accounts that were created in the SE account creation workflow
  • % of new accounts that were created in the SE account creation workflow (as % of overall accounts created on Android)
  • % of new accounts that were created in the SE account creation workflow which then went on to make 1 or more edits within 3 days of account creation
  • Editor retention of accounts that were created in the SE account creation workflow 7 and 30 day

Some of these are redundant with the above, need to edit down to what metrics we most want.

  • Track how many users are creating an account via Suggested edits menu item.
    • Track how many users start editing after newly creating an account via Suggested edits menu item (new acct w/ first SE-tagged edit within 48 hrs of acct creation event).
    • Track how many users keep editing after newly creating an account via Suggested edits menu item (7-day and 30-day).
  • Track how many users are logging in via Suggested edits menu item.
    • Track how many users start editing after logging in via Suggested edits menu item (login w/ first SE-tagged edit w/in 48 hrs of login event).
    • Track how many users keep editing after logging in via Suggested edits menu item (7-day and 30-day).
  • Also:
    • Measure before and after data (session length, app usage frequency for 30 days before/after acct creation event) of app users that are newly creating an account and then start editing.
    • Measure before and after data (session length, app usage frequency for 30 days before/after acct creation event) of app users that are logging in and then start editing.

Event Timeline

Aklapper renamed this task from Analytics for account creation to Analytics for account creation via Suggested edits.May 29 2020, 11:33 AM

@SNowick_WMF moving this to investigation to disucss where/how we need to prioritize this

SNowick_WMF added a comment.EditedJun 2 2020, 7:15 PM

For the metric: Track how many users are creating an account via Suggested edits menu item

Looking at the MobileWikiAppCreateAccount and MobileWikiAppLogin schemas the source options are: "edit", "blocked", "navigation", "onboarding". The most straightforward way to track these metrics going forward would be to add "suggestededits" as a source for both.

@cooltey can you add this as a source

Charlotte raised the priority of this task from Low to High.Jun 9 2020, 6:45 PM

@cooltey @SNowick_WMF - If the source is now added and data QA has happened, I'll mark this ticket as "lowest" priority until production data has started coming in. Shay, can you verify that this is behaving as expected on the alpha?

Charlotte raised the priority of this task from Lowest to Low.Jul 14 2020, 7:22 PM

I am not seeing 'suggestededits' as a source in event.source field in MobileWikiAppCreateAccount nor in MobileWikiAppLogin, @cooltey can you please check this and make sure it's in the code? Thanks!

Hi @SNowick_WMF, here's the log in the dev version of the app, and the suggestededits source has sent successfully in both MobileWikiAppCreateAccount and MobileWikiAppLogin

2020-07-14 15:03:12.968 15797-16212/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppCreateAccount","revision":18115095,"wiki":"enwiki","event":{"action":"start","loginSessionToken":"14ef348f-d44f-4963-a2d0-ee1d96209fb6","source":"suggestededits","client_dt":"2020-07-14T15:03:12-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"bd8a7a55-92d2-469e-ba49-9d2eb76c7ad4"}}
2020-07-14 15:04:33.493 15797-16212/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppCreateAccount","revision":18115095,"wiki":"enwiki","event":{"action":"success","source":"suggestededits","client_dt":"2020-07-14T15:04:33-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"bd8a7a55-92d2-469e-ba49-9d2eb76c7ad4"}}
2020-07-14 15:04:33.544 15797-16104/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppLogin","revision":18144592,"wiki":"enwiki","event":{"action":"start","source":"suggestededits","client_dt":"2020-07-14T15:04:33-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"14ef348f-d44f-4963-a2d0-ee1d96209fb6"}}
2020-07-14 15:04:33.550 15797-16016/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppLogin","revision":18144592,"wiki":"enwiki","event":{"action":"createAccountSuccess","client_dt":"2020-07-14T15:04:33-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"14ef348f-d44f-4963-a2d0-ee1d96209fb6"}}
2020-07-14 15:04:35.290 15797-16104/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppLogin","revision":18144592,"wiki":"enwiki","event":{"action":"success","client_dt":"2020-07-14T15:04:35-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"14ef348f-d44f-4963-a2d0-ee1d96209fb6"}}

Looks great, thanks. Will follow up with reporting when release is live.

I am not seeing any entries with source = suggestededits for account creation or account login in the data for the beta release. I expected we would see some showing up by now, can we check this again? I know the app event logs show the app is writing with the proper source but it doesn't seem to be making it to Event Logging.

Data here:
https://superset.wikimedia.org/superset/dashboard/159/

Dbrant added a subscriber: Dbrant.Aug 4 2020, 1:57 PM

Hi @SNowick_WMF, here's the log in the dev version of the app, and the suggestededits source has sent successfully in both MobileWikiAppCreateAccount and MobileWikiAppLogin

Events being sent successfully does not mean that they are received successfully, or that they are formatted correctly. It looks like you never updated the schemas to support the new suggestededits source parameter.

Schemas were edited but we didn't change the revision number, missed that step, sorry.
Schema Revision Numbers:
MobileWikiAppLogin revision: 20318421
MobileWikiAppCreateAccount revision: 20318420

SNowick_WMF added a comment.EditedAug 4 2020, 8:56 PM

Success! Data with source = suggestededits is showing up in both tables. Will collect data for this ticket and submit report after alpha is released.

SNowick_WMF raised the priority of this task from Low to High.Sep 28 2020, 8:14 PM
SNowick_WMF moved this task from Next 2 weeks to Doing on the Product-Analytics (Kanban) board.

@Charlotte @Dbrant @cooltey
Looking at total accounts w/ action = success in the MobileWikiAppCreateAccount table I'm seeing a 39% drop in total accounts post 2020-08-05. Did something change with the counting mechanism. We went from and avg. 1400 success daily to avg. 842 daily. Charts Here


I looked at MobileWikiAppLogin action = createAccountSuccess and see a similar drop. I made a chart for total event counts in that table which looks like they are all lower after 2020-08-05. I also saw a lot of anomalous dates far into the future in that table, if that adds any troubleshooting info.

I checked Play Store data for DAU and New Users and don't see any significant drops. This is something we should investigate, sorry I didn't find it sooner, would like to see if this relative to these tables or something more.

@Charlotte @Dbrant @cooltey
Looking at total accounts w/ action = success in the MobileWikiAppCreateAccount table I'm seeing a 39% drop in total accounts post 2020-08-05. Did something change with the counting mechanism. We went from and avg. 1400 success daily to avg. 842 daily. Charts Here

Definitely looks like the artefact of a change rather than a natural occurrence. Any ideas @Dbrant ?

Dbrant added a comment.Oct 1 2020, 3:05 PM

@SNowick_WMF @Charlotte
I believe I've confirmed that we're logging everything correctly when creating an account and logging in.

The release that we made on Aug 4 was the one where we decided to flip around the Login and Create Account screens, so that the Create Account screen appears first. In other words, when the user goes to log in, they are presented with the Create Account screen, and if they already have an account there's a separate button to take them to the Login screen. However, I'm very skeptical that this change would result in such a radical shift in the number of accounts created.

My only other thought is that 39% is rather close to 50%, which suggests that something somewhere is being halved: either half of the success events after Aug 4 are getting lost somewhere, or half of those events aren't being sent properly from the app despite my verification so far. Therefore I'm inclined to think that there's still an issue with the logging logic that needs figuring out, and not an issue of our design/ux updates.

@SNowick_WMF @Charlotte
I believe I've confirmed that we're logging everything correctly when creating an account and logging in.

The release that we made on Aug 4 was the one where we decided to flip around the Login and Create Account screens, so that the Create Account screen appears first. In other words, when the user goes to log in, they are presented with the Create Account screen, and if they already have an account there's a separate button to take them to the Login screen. However, I'm very skeptical that this change would result in such a radical shift in the number of accounts created.

I think that's right. I would not expect such a huge step change. Suggest taking whatever steps necessary to figure out if events are indeed getting lost. If you can't find anything, then maybe an A/B test (i.e., shift the order of screens back for some % of users to verify if account creation increases) is a last resort.

SNowick_WMF updated the task description. (Show Details)Oct 2 2020, 11:02 PM
SNowick_WMF reassigned this task from SNowick_WMF to Dbrant.Oct 6 2020, 5:09 PM
SNowick_WMF moved this task from Doing to Blocked on the Product-Analytics (Kanban) board.
SNowick_WMF lowered the priority of this task from High to Medium.
SNowick_WMF moved this task from Needs Review to Doing on the Product-Analytics (Kanban) board.
SNowick_WMF reassigned this task from SNowick_WMF to Dbrant.Tue, Nov 24, 3:14 PM

There are some errors showing up in Logstash/Kibana snapshot here that I suspect to be part of the drop in Create Account events, these events won't make it to the schema, there are ~1000+ of these event per day. Kibana is a very useful tool (it is linked by default on every Schema Discussion page).

ERRORS:

2020-11-24T14:45:45 'suggestededits' is not one of ['edit', 'blocked', 'navigation', 'onboarding'] ----> This is from an older version of app because 'suggestededits' was added to the schema (see below).

The following source values were not added to the schema and because that field is set to enum the values weren't being counted.

	2020-11-24T14:45:11	'reading_lists_manual_sync' is not one of ['edit', 'blocked', 'navigation', 'onboarding', 'suggestededits']

	2020-11-24T14:44:46	'logout_background' is not one of ['edit', 'blocked', 'navigation', 'onboarding', 'suggestededits']

	2020-11-24T14:40:39	'suggestededits' is not one of ['edit', 'blocked', 'navigation', 'onboarding']

	2020-11-24T14:39:55	'reading_lists_manual_sync' is not one of ['edit', 'blocked', 'navigation', 'onboarding']

	2020-11-24T14:37:11	'settings' is not one of ['edit', 'blocked', 'navigation', 'onboarding', 'suggestededits']

	2020-11-24T14:25:02	'system' is not one of ['edit', 'blocked', 'navigation', 'onboarding', 'suggestededits']

FIX:

I have added "reading_lists_manual_sync", "settings", "logout_background", "system" to the Schema:MobileWikiAppCreateAccount schema, which must be updated to Revision 20709917 in the code for change to take affect.