Now this is quite the hilarious bug. When looking at the survey data, I noticed that on September 20th, Russian Wikipedia NavTiming hits started getting the survey 100% of the time, when the survey sampling rate didn't change, it was still 2 (i.e. only 50% of NavTiming hits are supposed to get the survey).

The root cause turns out to be https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/NavigationTiming/+/458081/

What's happening is that once we've passed the initial mw.eventLog.inSample that determines whether NavTiming should trigger (1 in 1000 chance), we call it again to see if the survey should trigger (1 in 2 chances). But mw.eventLog.inSample works on a sticky value, looking at its modulo. If a number can be divided by 1000, it can be divided by 2...

Essentially the stickiness of mw.eventLog.inSample is making it unsuitable for nested inSample logic.

Russian Wikipedia is the only wiki where this bug manifested itself, because it's the only one configured with an even survey sampling rate!

Not having a local copy of inSample in NavigationTiming is a good idea, but the local copy still had the old property of not being sticky.

The simplest fix is for sub-sampling logic in NavigationTiming to call mw.eventLog.randomTokenMatch (non sticky) instead of mw.eventLog.inSample;