Note: removed requirement of sampling rate 100% for all AMC users. We're enabling AMC for all users shortly, there is no need for special handling AMC users any more.

I clarified with @pmiazga that it's expected that we're not sending seen events for UI elements yet. That is, we're tracking UI element interactions as we were before.

In T220016#5377135, @pmiazga wrote:

@MNeisler do we want to enable the schema on production with sampling rate 50% ?

@pmiazga Yes, I think we can enable with the 50% sampling rate for all clicks.

Per T220016#5377322. Alternatively, I'm happy to sign this off and create an "enable the instrumentation" task.

Change 526516 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable MobileWebUIActionsTracking schema with 50% sampling rate

https://gerrit.wikimedia.org/r/526516

Change 526516 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MobileWebUIActionsTracking schema with 50% sampling rate

https://gerrit.wikimedia.org/r/526516

Mentioned in SAL (#wikimedia-operations) [2019-07-30T23:06:53Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable MobileWebUIActionsTracking schema with 50% sampling rate (T220016) (duration: 00m 48s)

Events are sampled, but looks like sampling is not by user session or page - 50% of the time when refreshing I trigger events, 50% of the time I don't. Not sure if that was intentional.

I had to revert the deploy today - it caused this prominent spike in logstash

`
schema:EventError event_revision:19207217 level:ERROR wiki: type:eventlogging message:Additional properties are not allowed (u'modes' was unexpected) uuid:cf59c1d8b32011e981651418775b0d42 normalized_message:Additional properties are not allowed (u'modes' was unexpected) revision:14035058 tags:eventlogging_EventError, kafka, input-kafka-eventlogging, truncated_by_filter_truncate, es, normalized_message_untrimmed raw_event:{"dt": "2019-07-30T23:21:50Z", "event": {"action": "click", "destination": "/wiki/Sp%C3%A9cial:Random#/random",

Let's get the schema fixed, QA this again and make sure it gets deployed. Window is a little tight on this if we want it live for next Monday. Also note, the existing schema MobileWebMainMenuClickTracking is broken so currently we're not tracking any main menu clicks.

Mentioned in SAL (#wikimedia-operations) [2019-07-30T23:31:24Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Revert "Enable MobileWebUIActionsTracking schema with 50% sampling rate" (T220016) (duration: 00m 47s)

Change 526688 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[mediawiki/extensions/WikimediaEvents@wmf/1.34.0-wmf.15] Improved MobileUIActions tracking schema

https://gerrit.wikimedia.org/r/526688

Change 526691 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Enable MobileWebUIActionsTracking schema with 50% sampling rate

https://gerrit.wikimedia.org/r/526691

Change 526688 merged by jenkins-bot:
[mediawiki/extensions/WikimediaEvents@wmf/1.34.0-wmf.15] Improved MobileUIActions tracking schema

https://gerrit.wikimedia.org/r/526688

Change 526691 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable MobileWebUIActionsTracking schema with 50% sampling rate

https://gerrit.wikimedia.org/r/526691

Mentioned in SAL (#wikimedia-operations) [2019-07-31T16:37:52Z] <urbanecm@deploy1001> Synchronized php-1.34.0-wmf.15/extensions/WikimediaEvents/: SWAT: [[:gerrit:526688|Improved MobileUIActions tracking schema]] (T220016) (duration: 00m 54s)

Mentioned in SAL (#wikimedia-operations) [2019-07-31T16:39:32Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[:gerrit:526691|Enable MobileWebUIActionsTracking schema with 50% sampling rate]] (T220016) (duration: 00m 58s)

Looks like events are coming in: https://grafana.wikimedia.org/d/000000566/overview?orgId=1&from=now-30m&to=now&panelId=22&fullscreen !

A few quick checks of the current data:

select event.name, count(*) as event_count
from event.mobilewebuiactionstracking
where year = 2019 and month=7 and day = 31 
group by event.name
ORDER BY event_count DESC LIMIT 10000;

name	event_count
random	16237
home	10567
settings	5150
login	2092
nearby	1421
watchlist	812
profile	425
contributions	418
logout	133
homepage	9
communityportal	5
specialpages	3
preferences	2

select event.modes, COUNT(*) AS modes_count 
from event.mobilewebuiactionstracking
Where year = 2019 and month=7 and day = 31 
group by event.modes
ORDER BY modes_count DESC LIMIT 10000;

modes	modes_count
stable	36524
beta	667
beta,amc	42
stable,amc	35
desktop	6

At a glance this data is very consistent with what we were seeing in the old schema (obviously excluding AMC entries) - random was always the most popular feature there.

Re-opening this task as not all menu entries are tracked. Some UserMenu/Overflow menu elements do not have data-event-name attribute, thus interactions with those elements are not tracked.

In T220016#5403349, @pmiazga wrote:

Re-opening this task as not all menu entries are tracked. Some UserMenu/Overflow menu elements do not have data-event-name attribute, thus interactions with those elements are not tracked.

added the above to the acceptance criteria (as it was in the "background" section below)

Change 529178 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[mediawiki/extensions/WikimediaEvents@master] Track minerva.MobileWebUIActionsTracking events

https://gerrit.wikimedia.org/r/529178

Per slack conversation with @MNeisler

Because now we're using generic MobileWebUIActionsTracking schema, we should prefix event names so it's clean where those come from.
We will do following steps:

• add menu. prefix when tracking each menu entry, eg: menu.random, menu.home, menu.profile ....
• add ui. prefix when tracking menu triggers, eg ui.mainmenu, ui.usermenu, ui.overflowmenu
• watchstar icon will be tracked with menu.watch and menu.unwatch depends on the watch state. If user is not watching the page and user clicks the watchstar icon - menu.watch event should be sent.

Also, per conversation with @ovasileva there is no need to distinguish clicks from same entries but located in different menus. For example the languages menu entry can be in toolbar (regular user), or in the overflow menu (AMC user visiting user page). In both cases the Language menu entry will be tracked with menu.languages code.

Change 529180 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[mediawiki/skins/MinervaNeue@master] Track all menu interactions

https://gerrit.wikimedia.org/r/529180

Marking patch as WIP as analytics infrastructure will need to be consulted on the newly expected traffic to this schema. I'm fairly confident we haven't run this by Nuria etc.. and this increase is going to be considerable when we sample at 50%. Let's not melt the infrastructure. Alternatively we should limit this logging to AMC only to ensure traffic estimates are accurate and we don't get blocked here.

The previous MobileWebMainMenuClickTracking was tracking only main menu clicks. Because those actions are pretty rare, we were able to track such actions with 50% sampling rate.
The new schema MobileWebUIActionsTracking will track much more click actions, not only clicking menu elements, but also clicking different menus, icons on toolbar, tabs. Therefore the 50% sampling rate might be a bit too much for our analytics server.
It's very difficult to estimate how many events per minute we will get, instead of spending too much time on trying to come up with some numbers, I propose

• change the sampling rate to something very small, like 1% just before the train (Thursday, EU mid day swat)
• when train hits all wikis, and analytics servers are doing fine and we can handle 10 times more, bump it to 10% in the first available SWAT window, Thursday evening sf ?)

@Nuria are you ok with us taking this path?, First tring 1%, and then bumping to 10%?
@MNeisler is 10% sampling rate for you ok? Do you want to go higher?
/cc @ovasileva

@pmiazga Looks like this schema is to be deployed to all wikis. It has some unexplained peaks that do not look too good, as volume spikes for too short of a period (this could be an issue on the collection end if peaks were to appear in other schemas but they do not).
If the number of elements we are tracking now is an order of magnitude bigger than before the peaks might be quite significant. I suggest a very cautious sampling rate of 0.01% for about a day.

In T220016#5429419, @pmiazga wrote:

@MNeisler is 10% sampling rate for you ok? Do you want to go higher?

I'm fine with increasing to a 10% sampling rate once we confirm it's working ok.

Change 532422 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Drop MobileWebUIActionsTracking sampling rate to 0.01%

https://gerrit.wikimedia.org/r/532422

Change 529178 abandoned by Pmiazga:
Track minerva.MobileWebUIActionsTracking events

Reason:
we decided to fix the MainMenu so it doesn't call stopPropagation(). This patch is not needed atm.

https://gerrit.wikimedia.org/r/529178

Change 532427 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/extensions/MobileFrontend@master] Edit icon should trigger UI click tracking events

https://gerrit.wikimedia.org/r/532427

Change 532429 had a related patch set uploaded (by Jdlrobson; owner: Jdlrobson):
[mediawiki/skins/MinervaNeue@master] Avoid unnecessary stopPropagation usage so event click tracking can work

https://gerrit.wikimedia.org/r/532429

Change 532427 merged by jenkins-bot:
[mediawiki/extensions/MobileFrontend@master] Edit icon should trigger UI click tracking events

https://gerrit.wikimedia.org/r/532427

Change 532429 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Avoid unnecessary stopPropagation usage so event click tracking can work

https://gerrit.wikimedia.org/r/532429

Change 529180 merged by jenkins-bot:
[mediawiki/skins/MinervaNeue@master] Track all menu interactions

https://gerrit.wikimedia.org/r/529180

Change 532422 merged by jenkins-bot:
[operations/mediawiki-config@master] Drop MobileWebUIActionsTracking sampling rate to 0.01%

https://gerrit.wikimedia.org/r/532422

Mentioned in SAL (#wikimedia-operations) [2019-08-27T11:11:24Z] <pmiazga@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:532422|Drop MobileWebUIActionsTracking sampling rate to 0.01% (T220016)]] (duration: 00m 46s)

Blocked on 1.34-wmf.20 release.

Now we're tracking all clicks on menu elements (Main Menu, Toolbar, Overflow, User) with 0.01% sampling rate.
Previously, for the MainMenu only, we were getting ~80-100 events per 10 minutes, now we're getting less than 1 even per 10 minutes (see https://grafana.wikimedia.org/d/000000018/eventlogging-schema?orgId=1&var-schema=MobileWebUIActionsTracking&from=now-7d&to=now ). I'll bump the sampling rate by 100 (to 1%).

/cc @Nuria @MNeisler

Sounds good, +1

Change 533930 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Bump MobileWebUIActionsTracking sampling rate to 1 percent

https://gerrit.wikimedia.org/r/533930

Change 533930 merged by jenkins-bot:
[operations/mediawiki-config@master] Bump MobileWebUIActionsTracking sampling rate to 1 percent

https://gerrit.wikimedia.org/r/533930

Mentioned in SAL (#wikimedia-operations) [2019-09-03T11:10:59Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:533930|Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016)]] (duration: 00m 53s)

Mentioned in SAL (#wikimedia-operations) [2019-09-03T11:25:44Z] <ladsgroup@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:533930|Bump MobileWebUIActionsTracking sampling rate to 1 percent (T220016)]] (duration: 00m 52s)

We're receiving ~15 events per minute (previously we were receiving ~100 events), I think that we can still safely bump the sampling rate to ~10%, but first, we need to understand why do we get an events spike from time to time.

https://grafana.wikimedia.org/d/000000018/eventlogging-schema?orgId=1&var-schema=MobileWebUIActionsTracking&from=now-10d&to=now

/cc @Nuria @MNeisler

we need to understand why do we get an events spike from time to time.

Agreed, I bet if you look at the data on hadoop it will have a loads of clues. Let us know if you need help.

I spent a little time digging into the data that occurs around some of the event spikes. I specifically looked at the spikes seen at 2019-08-20 on 12:00pm and 2019-09-04 at 17:30pm.

These spikes seem to be due to a large number of events recorded for a single ip address and session token at these times likely indicating an unflagged bot (It was not flagged as useragent.is_bot).

On August 20th, there were a total of 2636 events associated with a single ip address and session token between 11:00 and 12:00, compared to around 1 to 40 events recorded for all other ips during that hour. All the events recorded from this ip address during this time were clicks to the random feature and came from a Chrome Mobile browser.

SELECT ip, event.token,
COUNT(*) as events
FROM event.mobilewebuiactionstracking
WHERE year = 2019 and month =08 and day = 20
AND dt > '2019-08-20T11:00:00Z'
AND dt < '2019-08-20T12:00:00Z'
GROUP by ip, event.token
ORDER BY events DESC LIMIT 100

On September 4th, there were a total of 373 events associated with a single ip address and session token between 16:30 and 17:30, compared to around 1 to 8 events recorded for all other ips during that hour. It looks like all the events recorded from this ip address during this time were clicks to the menu.edit feature and came from a Chrome Mobile browser.

SELECT ip, event.token,
COUNT(*) as events
FROM event.mobilewebuiactionstracking
WHERE year = 2019 and month =09 and day = 04
AND dt > '2019-09-04T16:30:00Z'
AND dt < '2019-09-04T17:30:00Z'
GROUP by ip, event.token
ORDER BY events DESC LIMIT 100

Let me know if there's anything else you think we should look into further.

Bumping sampling rate to 10%

Change 535536 had a related patch set uploaded (by Pmiazga; owner: Pmiazga):
[operations/mediawiki-config@master] Bump MobileWebUIActionsTracking sampling rate to 10 percent

https://gerrit.wikimedia.org/r/535536

Change 535536 merged by jenkins-bot:
[operations/mediawiki-config@master] Bump MobileWebUIActionsTracking sampling rate to 10 percent

https://gerrit.wikimedia.org/r/535536

Mentioned in SAL (#wikimedia-operations) [2019-09-10T11:08:59Z] <urbanecm@deploy1001> Synchronized wmf-config/InitialiseSettings.php: SWAT: c780fa4: Bump MobileWebUIActionsTracking sampling rate to 10 percent (T220016) (duration: 00m 55s)