Change 500478 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/MobileFrontend@master] Add amc flag to mw.config for client side awareness

https://gerrit.wikimedia.org/r/500478

Change 500479 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/MinervaNeue@master] Add amc field to MobileWebMainMenuClickTracking

https://gerrit.wikimedia.org/r/500479

Change 500545 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/extensions/MobileFrontend@master] Update "MobileWebMainMenuClickTracking" schema version

https://gerrit.wikimedia.org/r/500545

Change 500610 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/MinervaNeue@master] Add MobileWebMainMenuClickTracking to Minerva EventLoggingSchemas array

https://gerrit.wikimedia.org/r/500610

Change 500545 merged by Jdlrobson:
[mediawiki/extensions/MobileFrontend@master] Remove "MobileWebMainMenuClickTracking" from MF EventLoggingSchemas array

https://gerrit.wikimedia.org/r/500545

Change 500610 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Add MobileWebMainMenuClickTracking to Minerva EventLoggingSchemas array

https://gerrit.wikimedia.org/r/500610

Change 500478 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Add amc flag to mw.config for client side usage

https://gerrit.wikimedia.org/r/500478

Change 500479 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Add amc boolean to MobileWebMainMenuClickTracking sampling

https://gerrit.wikimedia.org/r/500479

@Jdlrobson I'm trying to put QA instructions on this, but I'm not sure exactly what we should be QAing. How in-depth with this do we want to go?

@Jdlrobson thanks for listing the QA steps, I'm not sure that Scenario 2b is possible (nor that we want it to be possible) according to the AC:

Sampling rate should be 100% if the user is AMC, otherwise use the configurable sampling rate

So I think Scenario 2b should be removed and Scenario 2a should instead be 100% chance

In T218627#5110077, @Edtadros wrote:

In the likely event I'm doing it wrong, after I click on the hamburger menu, I right click on Random and open that in a new window. <snip />

The tracking code can capture the user opening the link in a new tab with ⌘+click but not clicking "Open Link in New Tab" in the context menu. IIRC capturing the latter is hard.

Change 504228 had a related patch set uploaded (by Nray; owner: Nray):
[mediawiki/skins/MinervaNeue@master] Change MinervaSchemaMainMenuClickTrackingSampleRate to default to 0.5

https://gerrit.wikimedia.org/r/504228

In T218627#5110077, @Edtadros wrote:

Test Result

Status: Need More Info
OS: macOS Mojave
Browser: Chrome
Device: MBP

@nray I tried 1a, 1b, 3a, 3b about 10 times each and only saw amc:false once. Additionally when the instructions say "no event shows up" does this mean no event with the amc:false parameter, or no event at all. I get 3 beacon events every time for 1a and 1b, and two beacon events for 3a and 3b.

In the likely event I'm doing it wrong, after I click on the hamburger menu, I right click on Random and open that in a new window. I have Chrome to open dev tools automatically so it captures the traffic in the newly opened window. Given the randomness of 1a, 1b, 3a, and 3b, what sample size would you expect to see a 50/50 breakdown?

@Edtadros Ok few things:

1. First, the last step for 2a was wrong. I just corrected it to say "Inspect the request's "Query string parameters" and verify that "amc":true is in it". Sorry about that.

2. Instead of filtering by "beacon", filter by "MobileWebMainMenuClickTracking". I've updated the QA steps to reflect this. If you don't filter by "MobileWebMainMenuClickTracking", beacon requests might show up when the page loads that we don't care about for this ticket.

3. I think the other reason you might be seeing multiple beacon requests is that when you click on "Random", you might be checking the chrome dev tools for the new tab (the random page). Instead, you should be checking the chrome dev tools of the tab that you clicked the menu in (on the Spain page if you testing on https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain) . When I check the beta cluster I only see 1 beacon event request (see screenshot) when I do this (and assuming other steps are followed). In addition, I ran through all of the scenarios and it seemed like I was correctly getting 50% for 1a, 1b, 3a, 3b, and 100% for 2a
   

4. Just FYI but it doesn't sound like this is affecting you, you should be testing this on https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain per the QA instructions. I just noticed when I was reviewing this that local installations of mediawiki will default to a sampling rate of 0 for this which I corrected in https://gerrit.wikimedia.org/r/504228. This shouldn't affect https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain or production though because they are correctly configured.

5. Another FYI but it doesn't sound like this is affecting you, it is crucial when testing these that you are at least opening a new tab (or a completely new window) for EACH test. You cannot simply refresh the page or else you will get the same results. So if you are testing 1a/1b, roughly 5 out of 10 tabs (or windows) should have a beacon request with 'amc: false' in the query params. The rest should not have a beacon request at all.

5. Please ensure that you are following @phuedx 's advice as well (above)

Change 504228 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Change MinervaSchemaMainMenuClickTrackingSampleRate default in README

https://gerrit.wikimedia.org/r/504228

In T218627#5133399, @Edtadros wrote:

@nray Thanks for the clarifications. @phuedx thanks for the mac tailored advice! That's some serious attention to detail. I've added screencasts so you can identify if I'm doing something wrong for 1a/b and 3a/b.

Test Result

Status: ❌ FAIL
OS: macOS Mojave
Browser: Chrome
Device: MBP
Emulated Device: iPhoneX

Test Artifact(s):

❌ AC1a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC1b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

T218627-1ab.mp42 MBDownload

✅ AC2a - The event and Query string parameter appeared 100% of the time

T218627-2a.mp43 MBDownload

❌ AC3a - The event and Query string parameter appeared 100% of the time instead of 50% of the time.
❌ AC3b - The event and Query string parameter appeared 100% of the time instead of NOT appearing 50% of the time.

T218627-3ab.mp42 MBDownload

@Edtadros I think it's because you are opening up https://en.m.wikipedia.beta.wmflabs.org/wiki/Spain once and then clicking on the menu link multiple times. Instead, for a test of 1a/1b you should open up a tab of the Spain page, command-click once on a menu link and check for the request in dev tools from the Spain page. Then open up a NEW tab of the Spain page, command-click once on the menu link and check for the request in dev tools from the Spain page. Rinse and repeat this process as many times as needed to see if you follow the required percentages.

I've attached a video that shows the process for 1a/1b. You'll notice that the first time I open the Spain page there is a request with amc: true which meets the criteria for 1a, and the second time I open the Spain page there is not any request after clicking the menu link which meets the criteria for 1b. You should follow this process when testing for 2a/3a/3b as well. Does that make sense? Sorry for the confusion!

Update the schema to have an AMC flag

Done in https://meta.wikimedia.org/w/index.php?title=Schema%3AMobileWebMainMenuClickTracking&type=revision&diff=18984528&oldid=18947573.

Update the schema revision number in MobileFrontend and move to Minerva codebase
Sampling rate should be 100% if the user is AMC, otherwise use the configurable sampling rate
Log whether the user is in AMC mode using the new schema revision ID

Done in https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/skins/MinervaNeue/+/517e4f537a5ba79ecd4e63478ce6e503ed5f8c05%5E%21/.

Document the oversampling behaviour for AMC on the schema's talk page so Tilman can understand it

Done in https://meta.wikimedia.org/w/index.php?title=Schema_talk%3AMobileWebMainMenuClickTracking&type=revision&diff=19048492&oldid=18967926.