Page MenuHomePhabricator

Upgrade MobileWebMainMenuClickTracking to have an AMC field
Closed, ResolvedPublic3 Story Points

Description

Clicks to items in the main menu are logged via the MobileWebMainMenuClickTracking schema for 50% of all session IDs. For AMC users we want to log all clicks and we need a way to distinguish these users from normal users.

Acceptance criteria

  • Update the schema to have an AMC flag
  • Update the schema revision number in MobileFrontend and move to Minerva codebase
  • Sampling rate should be 100% if the user is AMC, otherwise use the configurable sampling rate
  • Log whether the user is in AMC mode using the new schema revision ID

Open questions

Do we want to use the existing "mode" field in https://meta.wikimedia.org/wiki/Schema:MobileWebMainMenuClickTracking to track "amc" e.g. "amc+stable", "amc+beta" or create a dedicated field with boolean? Cf. notes at T212961

Sign off steps

  • Document the oversampling behaviour for AMC on the schema's talk page so Tilman can understand it

QA steps

This can be tested on the beta cluster using an incognito window.
50% of clicks in the main menu are tracked via the MobileWebMainMenuClickTracking schema. Please verify that you can see every scenario

Anons

Scenario 1a (50% chance)

  • Open an incognito window with https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain
  • Open developer tools and select network tab. Make sure "preserve log" is ticked and type "MobileWebMainMenuClickTracking" in the filter box
  • Click the hamburger menu
  • Open a link in a new window
  • Verify that an event shows up in the list of network requests
  • Inspect the request's "Query string parameters" and verify that "amc":false is in it

Scenario 1b (50% chance)

  • Open an incognito window with https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain
  • Open developer tools and select network tab. Make sure "preserve log" is ticked and type "MobileWebMainMenuClickTracking" in the filter box
  • Click the hamburger menu
  • Open a link in a new window
  • Verify that no event shows up in the list of network requests

AMC user

Scenario 2a (100% chance)

  • Open an incognito window with https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain
  • Login as a user with AMC mode enabled
  • Open developer tools and select network tab. Make sure "preserve log" is ticked and type "MobileWebMainMenuClickTracking" in the filter box
  • Click the hamburger menu
  • Open a link in a new window
  • Inspect the request's "Query string parameters" and verify that "amc":true is in it

non-AMC user

Scenario 3a (50% chance)

  • Open an incognito window with https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain
  • Login as a user with AMC mode DISABLED
  • Open developer tools and select network tab. Make sure "preserve log" is ticked and type "MobileWebMainMenuClickTracking" in the filter box
  • Click the hamburger menu
  • Open a link in a new window
  • Verify that an event shows up in the list of network requests
  • Inspect the request's "Query string parameters" and verify that "amc":false is in it

Scenario 3b (50% chance)

  • Open an incognito window with https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain
  • Login as a user with AMC mode DISABLED
  • Open developer tools and select network tab. Make sure "preserve log" is ticked and type "MobileWebMainMenuClickTracking" in the filter box
  • Click the hamburger menu
  • Open a link in a new window
  • Verify that no event shows up in the list of network requests

QA Results

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2019, 12:42 AM
ovasileva triaged this task as Normal priority.Mar 19 2019, 9:32 AM
ovasileva set the point value for this task to 3.Mar 19 2019, 4:46 PM
Tbayer updated the task description. (Show Details)Mar 19 2019, 5:28 PM
Tbayer added a subscriber: Tbayer.

Change 500478 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/MobileFrontend@master] Add amc flag to mw.config for client side awareness

https://gerrit.wikimedia.org/r/500478

Change 500479 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/MinervaNeue@master] Add amc field to MobileWebMainMenuClickTracking

https://gerrit.wikimedia.org/r/500479

Change 500545 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/MobileFrontend@master] Update "MobileWebMainMenuClickTracking" schema version

https://gerrit.wikimedia.org/r/500545

nray removed nray as the assignee of this task.
nray added a subscriber: nray.

Change 500610 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/MinervaNeue@master] Add MobileWebMainMenuClickTracking to Minerva EventLoggingSchemas array

https://gerrit.wikimedia.org/r/500610

Change 500545 merged by Jdlrobson:
[mediawiki/extensions/MobileFrontend@master] Remove "MobileWebMainMenuClickTracking" from MF EventLoggingSchemas array

https://gerrit.wikimedia.org/r/500545

Change 500610 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Add MobileWebMainMenuClickTracking to Minerva EventLoggingSchemas array

https://gerrit.wikimedia.org/r/500610

Change 500478 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Add amc flag to mw.config for client side usage

https://gerrit.wikimedia.org/r/500478

Change 500479 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Add amc boolean to MobileWebMainMenuClickTracking sampling

https://gerrit.wikimedia.org/r/500479

nray added a comment.Apr 2 2019, 3:59 PM

@Jdlrobson I'm trying to put QA instructions on this, but I'm not sure exactly what we should be QAing. How in-depth with this do we want to go?

Jdlrobson removed a project: Patch-For-Review.
Jdlrobson updated the task description. (Show Details)
nray added a comment.Apr 2 2019, 6:11 PM

@Jdlrobson thanks for listing the QA steps, I'm not sure that Scenario 2b is possible (nor that we want it to be possible) according to the AC:

Sampling rate should be 100% if the user is AMC, otherwise use the configurable sampling rate

So I think Scenario 2b should be removed and Scenario 2a should instead be 100% chance

Jdlrobson updated the task description. (Show Details)Apr 2 2019, 6:29 PM
ovasileva raised the priority of this task from Normal to High.Apr 9 2019, 6:17 PM
Edtadros reassigned this task from Edtadros to nray.Apr 14 2019, 10:36 PM
Edtadros added a subscriber: Edtadros.

Test Result

Status: Need More Info
OS: macOS Mojave
Browser: Chrome
Device: MBP

@nray I tried 1a, 1b, 3a, 3b about 10 times each and only saw amc:false once. Additionally when the instructions say "no event shows up" does this mean no event with the amc:false parameter, or no event at all. I get 3 beacon events every time for 1a and 1b, and two beacon events for 3a and 3b.

In the likely event I'm doing it wrong, after I click on the hamburger menu, I right click on Random and open that in a new window. I have Chrome to open dev tools automatically so it captures the traffic in the newly opened window. Given the randomness of 1a, 1b, 3a, and 3b, what sample size would you expect to see a 50/50 breakdown?

phuedx added a subscriber: phuedx.Apr 15 2019, 11:50 AM

In the likely event I'm doing it wrong, after I click on the hamburger menu, I right click on Random and open that in a new window. <snip />

The tracking code can capture the user opening the link in a new tab with ⌘+click but not clicking "Open Link in New Tab" in the context menu. IIRC capturing the latter is hard.

Change 504228 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/MinervaNeue@master] Change MinervaSchemaMainMenuClickTrackingSampleRate to default to 0.5

https://gerrit.wikimedia.org/r/504228

nray updated the task description. (Show Details)Apr 16 2019, 1:02 AM
nray added a comment.EditedApr 16 2019, 1:30 AM

=== Test Result

Status: Need More Info
OS: macOS Mojave
Browser: Chrome
Device: MBP

@nray I tried 1a, 1b, 3a, 3b about 10 times each and only saw amc:false once. Additionally when the instructions say "no event shows up" does this mean no event with the amc:false parameter, or no event at all. I get 3 beacon events every time for 1a and 1b, and two beacon events for 3a and 3b.

In the likely event I'm doing it wrong, after I click on the hamburger menu, I right click on Random and open that in a new window. I have Chrome to open dev tools automatically so it captures the traffic in the newly opened window. Given the randomness of 1a, 1b, 3a, and 3b, what sample size would you expect to see a 50/50 breakdown?

@Edtadros Ok few things:

  1. First, the last step for 2a was wrong. I just corrected it to say "Inspect the request's "Query string parameters" and verify that "amc":true is in it". Sorry about that.
  1. Instead of filtering by "beacon", filter by "MobileWebMainMenuClickTracking". I've updated the QA steps to reflect this. If you don't filter by "MobileWebMainMenuClickTracking", beacon requests might show up when the page loads that we don't care about for this ticket.
  1. I think the other reason you might be seeing multiple beacon requests is that when you click on "Random", you might be checking the chrome dev tools for the new tab (the random page). Instead, you should be checking the chrome dev tools of the tab that you clicked the menu in (on the Spain page if you testing on https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain) . When I check the beta cluster I only see 1 beacon event request (see screenshot) when I do this (and assuming other steps are followed). In addition, I ran through all of the scenarios and it seemed like I was correctly getting 50% for 1a, 1b, 3a, 3b, and 100% for 2a
  1. Just FYI but it doesn't sound like this is affecting you, you should be testing this on https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain per the QA instructions. I just noticed when I was reviewing this that local installations of mediawiki will default to a sampling rate of 0 for this which I corrected in https://gerrit.wikimedia.org/r/504228. This shouldn't affect https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain or production though because they are correctly configured.
  1. Another FYI but it doesn't sound like this is affecting you, it is crucial when testing these that you are at least opening a new tab (or a completely new window) for EACH test. You cannot simply refresh the page or else you will get the same results. So if you are testing 1a/1b, roughly 5 out of 10 tabs (or windows) should have a beacon request with 'amc: false' in the query params. The rest should not have a beacon request at all.
  1. Please ensure that you are following @phuedx 's advice as well (above)
nray updated the task description. (Show Details)Apr 16 2019, 4:30 AM

Change 504228 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Change MinervaSchemaMainMenuClickTrackingSampleRate default in README

https://gerrit.wikimedia.org/r/504228

Edtadros reassigned this task from Edtadros to nray.Apr 24 2019, 12:30 AM

@nray Thanks for the clarifications. @phuedx thanks for the mac tailored advice! That's some serious attention to detail. I've added screencasts so you can identify if I'm doing something wrong for 1a/b and 3a/b.

Test Result

Status: ❌ FAIL
OS: macOS Mojave
Browser: Chrome
Device: MBP
Emulated Device: iPhoneX

Test Artifact(s):

❌ AC1a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC1b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

✅ AC2a - The event and Query string parameter appeared 100% of the time

❌ AC3a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC3b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

Edtadros updated the task description. (Show Details)Apr 24 2019, 12:31 AM
nray added a comment.EditedApr 24 2019, 1:31 AM

@nray Thanks for the clarifications. @phuedx thanks for the mac tailored advice! That's some serious attention to detail. I've added screencasts so you can identify if I'm doing something wrong for 1a/b and 3a/b.

=== Test Result

Status: ❌ FAIL
OS: macOS Mojave
Browser: Chrome
Device: MBP
Emulated Device: iPhoneX

Test Artifact(s):

❌ AC1a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC1b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

✅ AC2a - The event and Query string parameter appeared 100% of the time

❌ AC3a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC3b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

@Edtadros I think it's because you are opening up https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain once and then clicking on the menu link multiple times. Instead, for a test of 1a/1b you should open up a tab of the Spain page, command-click once on a menu link and check for the request in dev tools from the Spain page. Then open up a NEW tab of the Spain page, command-click once on the menu link and check for the request in dev tools from the Spain page. Rinse and repeat this process as many times as needed to see if you follow the required percentages.

I've attached a video that shows the process for 1a/1b. You'll notice that the first time I open the Spain page there is a request with amc: true which meets the criteria for 1a, and the second time I open the Spain page there is not any request after clicking the menu link which meets the criteria for 1b. You should follow this process when testing for 2a/3a/3b as well. Does that make sense? Sorry for the confusion!

Edtadros reassigned this task from Edtadros to ovasileva.

@nray The new window each time did the trick. Thanks!

Test Result

Status: ✅ PASS
OS: macOS Mojave
Browser: Chrome
Device: MBP
Emulated Device: iPhoneX

Test Artifact(s):
Each AC was tried 10 times.

✅ AC1a - The event and Query string parameter appeared 50% of the time.
5 of 10 had the following result:

✅ AC1b - The event and Query string parameter appeared 50% of the time.
5 of 10 had the following result:

✅ AC2a - The event and Query string parameter appeared 100% of the time
10 of 10 had the following result:

✅ AC3a - The event and Query string parameter appeared 50% of the time.
5 of 10 had the following result:

✅ AC3b - The event and Query string parameter appeared 50% of the time.
5 of 10 had the following result:

phuedx claimed this task.Fri, Apr 26, 8:35 AM
phuedx added a subscriber: ovasileva.

I feel comfortable signing this off in @ovasileva's absence.

phuedx updated the task description. (Show Details)Fri, Apr 26, 1:29 PM
phuedx updated the task description. (Show Details)EditedFri, Apr 26, 1:33 PM

Update the schema revision number in MobileFrontend and move to Minerva codebase
Sampling rate should be 100% if the user is AMC, otherwise use the configurable sampling rate
Log whether the user is in AMC mode using the new schema revision ID

Done in https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/skins/MinervaNeue/+/517e4f537a5ba79ecd4e63478ce6e503ed5f8c05%5E%21/.

phuedx updated the task description. (Show Details)Fri, Apr 26, 1:35 PM

Document the oversampling behaviour for AMC on the schema's talk page so Tilman can understand it

Done in https://meta.wikimedia.org/w/index.php?title=Schema_talk%3AMobileWebMainMenuClickTracking&type=revision&diff=19048492&oldid=18967926.

phuedx updated the task description. (Show Details)Fri, Apr 26, 1:38 PM
phuedx closed this task as Resolved.

💪