Page MenuHomePhabricator

[SPIKE] Check MobileWebMainMenuClickTracking schema is still functional and determine how we can use it for AMC
Closed, DuplicatePublic

Description

Background

We would like to add new AMC menu links to either a new schema or to expand one of the existing schemas to account for the new links to answer the following questions:

  • Which items in the new navigation are clicked most often?
  • Are there items in the navigation that we can remove based on low usage?
  • Are the majority of visits to special pages coming from the navigation or through other methods?

We initially discussed using the MobileWebMainMenuClickTracking schema for this purpose, however, since the schema has not been actively used recently, we should first do some investigation to make sure it's still working as expected.

Open questions

Event Timeline

ovasileva created this task.Mar 7 2019, 4:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 7 2019, 4:45 PM
ovasileva triaged this task as High priority.Mar 7 2019, 4:45 PM
ovasileva moved this task from Incoming to Needs Prioritization on the Readers-Web-Backlog board.
Tbayer updated the task description. (Show Details)Mar 7 2019, 5:55 PM

We should also document the sampling method and ratio on the talk page. (Apparently InitialiseSettings.php sets a default value of 0.5, but that's obviously not the actual rate.)

And once we have confirmed that the schema is working for this purpose, we need to extend it with a field indicating whether the user has enabled AMC (should probably be a separate task).

Jdlrobson closed this task as Resolved.Mar 8 2019, 1:31 AM
Jdlrobson claimed this task.
Jdlrobson added a subscriber: Jdlrobson.

This is functional.
As this graph shows it gets a huge amount of traffic:
https://grafana.wikimedia.org/d/000000566/overview?panelId=5&fullscreen&orgId=1

I can confirm the code is working and data is coming in:

use wmf; select event.name, count(*) from mobilewebmainmenuclicktracking where event.mode='stable' and year = 2019 and month=3 and day = 6 group by event.name;

name	_c1
contributions	893
home	29141
login	5465
logout	370
nearby	3118
profile	1232
random	36252
settings	17840
watchlist	1667

This is functional.
As this graph shows it gets a huge amount of traffic:
https://grafana.wikimedia.org/d/000000566/overview?panelId=5&fullscreen&orgId=1
I can confirm the code is working and data is coming in:

use wmf; select event.name, count(*) from mobilewebmainmenuclicktracking where event.mode='stable' and year = 2019 and month=3 and day = 6 group by event.name;

name	_c1
contributions	893
home	29141
login	5465
logout	370
nearby	3118
profile	1232
random	36252
settings	17840
watchlist	1667

Whoo! @Jdlrobson - a question here. Are we able to add anything outside the menu to the schema (for example the history button, things that will go into the overlay menu, etc)?

I guess we should have been more specific in the task description - "functionaI" is not quite synonymous to "sending data" ;) (I had already run basically the same query beforehand...) But presumably the above also means that the underlying instrumentation code looks solid on first glance?

Thanks for documenting the sampling on the talk page! Does "0.5 of all clicks ... based on session id" mean 0.5% of sessions or half of them? The low event rate (roughly 1 event/sec on average) would be quite surprising in the former case, considering that we have about 3500 mobile pageviews per second.

[...]

@Jdlrobson - a question here. Are we able to add anything outside the menu to the schema (for example the history button, things that will go into the overlay menu, etc)?

BTW, I'm not familiar with the history here, but from https://meta.wikimedia.org/wiki/Schema_talk:MobileWebClickTracking it seems we had a more general schema before that included clicks from the main pageview (before opening this menu), but it was split up in 2014 because of MySQL performance concerns ("The MobileWebClickTracking table is getting huge. Splitting it into several schemas will make it easier to manage").

We have since gotten the much more performant alternative Hive/Hadoop backend for EventLogging, so this concern may no longer apply.

ovasileva updated the task description. (Show Details)Mar 11 2019, 6:38 PM
ovasileva renamed this task from Check MobileWebMainMenuClickTracking schema is still functional to [SPIKE] Check MobileWebMainMenuClickTracking schema is still functional and determine how we can use it for AMC.Mar 11 2019, 6:41 PM
ovasileva reopened this task as Open.
ovasileva updated the task description. (Show Details)

Reopening and adding some additional questions

ovasileva updated the task description. (Show Details)Mar 12 2019, 5:24 PM

Can we introduce a new schema, something like MobileUIElementInteraction, and allow us to track any UI element in the future. I think it would be very useful to have something that allows us tracking clicks on elements like menu items, overlays, drawers, etc. To distinguish elements we could pass the element select/or element id. we can also pass a property type, like "menu_item" or "actions_bar_item" or "drawer".

IMHO we should pursue reusable elements instead of creating separate schemas for each type of action.

Can we introduce a new schema, something like MobileUIElementInteraction, and allow us to track any UI element in the future. I think it would be very useful to have something that allows us tracking clicks on elements like menu items, overlays, drawers, etc. To distinguish elements we could pass the element select/or element id. we can also pass a property type, like "menu_item" or "actions_bar_item" or "drawer".
IMHO we should pursue reusable elements instead of creating separate schemas for each type of action.

there is a https://meta.wikimedia.org/wiki/Schema:MobileWebUIClickTracking schema, which has been disabled for a while now

ovasileva updated the task description. (Show Details)Mar 13 2019, 10:42 AM

@Jdlrobson - I edited the description to add the questions we want to answer as well as some other notes.

...

Thanks for documenting the sampling on the talk page! Does "0.5 of all clicks ... based on session id" mean 0.5% of sessions or half of them? The low event rate (roughly 1 event/sec on average) would be quite surprising in the former case, considering that we have about 3500 mobile pageviews per second.

Just looked further into this with @Jdlrobson and also ran a query - it's indeed 50% of sessions and includes anonymous users (I clarified the documentation on the talk page).
So that means that the items tucked into this menu do in fact have a very low clickthrough ratio per pageview, e.g <0.03% for the random page button (the most popular menu item in T217851#5010368 ).

SELECT SUM(IF(event.username IS NULL, 1, 0))/SUM(1) AS anon_ratio,
COUNT(*) AS events 
FROM event.mobilewebmainmenuclicktracking 
WHERE event.mode='stable' AND year = 2019 AND month=3 AND day = 6;

anon_ratio	events
0.9345370814144908	95978
Time taken: 34.18 seconds, Fetched: 1 row(s)