@SNowick_WMF moving this to investigation to disucss where/how we need to prioritize this

For the metric: Track how many users are creating an account via Suggested edits menu item

Looking at the MobileWikiAppCreateAccount and MobileWikiAppLogin schemas the source options are: "edit", "blocked", "navigation", "onboarding". The most straightforward way to track these metrics going forward would be to add "suggestededits" as a source for both.

@cooltey can you add this as a source

@SNowick_WMF Sure, will do it.

@cooltey @SNowick_WMF - If the source is now added and data QA has happened, I'll mark this ticket as "lowest" priority until production data has started coming in. Shay, can you verify that this is behaving as expected on the alpha?

I am not seeing 'suggestededits' as a source in event.source field in MobileWikiAppCreateAccount nor in MobileWikiAppLogin, @cooltey can you please check this and make sure it's in the code? Thanks!

Hi @SNowick_WMF, here's the log in the dev version of the app, and the suggestededits source has sent successfully in both MobileWikiAppCreateAccount and MobileWikiAppLogin

2020-07-14 15:03:12.968 15797-16212/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppCreateAccount","revision":18115095,"wiki":"enwiki","event":{"action":"start","loginSessionToken":"14ef348f-d44f-4963-a2d0-ee1d96209fb6","source":"suggestededits","client_dt":"2020-07-14T15:03:12-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"bd8a7a55-92d2-469e-ba49-9d2eb76c7ad4"}}
2020-07-14 15:04:33.493 15797-16212/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppCreateAccount","revision":18115095,"wiki":"enwiki","event":{"action":"success","source":"suggestededits","client_dt":"2020-07-14T15:04:33-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"bd8a7a55-92d2-469e-ba49-9d2eb76c7ad4"}}
2020-07-14 15:04:33.544 15797-16104/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppLogin","revision":18144592,"wiki":"enwiki","event":{"action":"start","source":"suggestededits","client_dt":"2020-07-14T15:04:33-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"14ef348f-d44f-4963-a2d0-ee1d96209fb6"}}
2020-07-14 15:04:33.550 15797-16016/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppLogin","revision":18144592,"wiki":"enwiki","event":{"action":"createAccountSuccess","client_dt":"2020-07-14T15:04:33-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"14ef348f-d44f-4963-a2d0-ee1d96209fb6"}}
2020-07-14 15:04:35.290 15797-16104/org.wikipedia.dev D/org.wikipedia.analytics.EventLoggingService: lambda$log$0():54: {"schema":"MobileWikiAppLogin","revision":18144592,"wiki":"enwiki","event":{"action":"success","client_dt":"2020-07-14T15:04:35-0700","app_install_id":"a7a3e629-f881-48bf-aba3-502812e02139","session_token":"14ef348f-d44f-4963-a2d0-ee1d96209fb6"}}

Looks great, thanks. Will follow up with reporting when release is live.

I am not seeing any entries with source = suggestededits for account creation or account login in the data for the beta release. I expected we would see some showing up by now, can we check this again? I know the app event logs show the app is writing with the proper source but it doesn't seem to be making it to Event Logging.

Data here:
https://superset.wikimedia.org/superset/dashboard/159/

In T253982#6306485, @cooltey wrote:

Hi @SNowick_WMF, here's the log in the dev version of the app, and the suggestededits source has sent successfully in both MobileWikiAppCreateAccount and MobileWikiAppLogin

Events being sent successfully does not mean that they are received successfully, or that they are formatted correctly. It looks like you never updated the schemas to support the new suggestededits source parameter.

Schemas were edited but we didn't change the revision number, missed that step, sorry.
Schema Revision Numbers:
MobileWikiAppLogin revision: 20318421
MobileWikiAppCreateAccount revision: 20318420

Success! Data with source = suggestededits is showing up in both tables. Will collect data for this ticket and submit report after alpha is released.

@Charlotte @Dbrant @cooltey
Looking at total accounts w/ action = success in the MobileWikiAppCreateAccount table I'm seeing a 39% drop in total accounts post 2020-08-05. Did something change with the counting mechanism. We went from and avg. 1400 success daily to avg. 842 daily. Charts Here

I checked Play Store data for DAU and New Users and don't see any significant drops. This is something we should investigate, sorry I didn't find it sooner, would like to see if this relative to these tables or something more.

In T253982#6503791, @SNowick_WMF wrote:

@Charlotte @Dbrant @cooltey
Looking at total accounts w/ action = success in the MobileWikiAppCreateAccount table I'm seeing a 39% drop in total accounts post 2020-08-05. Did something change with the counting mechanism. We went from and avg. 1400 success daily to avg. 842 daily. Charts Here

Definitely looks like the artefact of a change rather than a natural occurrence. Any ideas @Dbrant ?

@SNowick_WMF @Charlotte
I believe I've confirmed that we're logging everything correctly when creating an account and logging in.

The release that we made on Aug 4 was the one where we decided to flip around the Login and Create Account screens, so that the Create Account screen appears first. In other words, when the user goes to log in, they are presented with the Create Account screen, and if they already have an account there's a separate button to take them to the Login screen. However, I'm very skeptical that this change would result in such a radical shift in the number of accounts created.

My only other thought is that 39% is rather close to 50%, which suggests that something somewhere is being halved: either half of the success events after Aug 4 are getting lost somewhere, or half of those events aren't being sent properly from the app despite my verification so far. Therefore I'm inclined to think that there's still an issue with the logging logic that needs figuring out, and not an issue of our design/ux updates.

In T253982#6509246, @Dbrant wrote:

@SNowick_WMF @Charlotte
I believe I've confirmed that we're logging everything correctly when creating an account and logging in.

The release that we made on Aug 4 was the one where we decided to flip around the Login and Create Account screens, so that the Create Account screen appears first. In other words, when the user goes to log in, they are presented with the Create Account screen, and if they already have an account there's a separate button to take them to the Login screen. However, I'm very skeptical that this change would result in such a radical shift in the number of accounts created.

I think that's right. I would not expect such a huge step change. Suggest taking whatever steps necessary to figure out if events are indeed getting lost. If you can't find anything, then maybe an A/B test (i.e., shift the order of screens back for some % of users to verify if account creation increases) is a last resort.