Dang, I wish there was a way to set a seed in JS.

Okay, so there are a few components being tested here:

• A session ID can be generated (by this algorithm) and accessed
• That session ID can be turned into a number between 0 and 1 by normalizeId() (which is just parseInt( id.slice( 0, 8 ), 16 ) / UINT32_MAX and has been confirmed to have the desired properties)
• We can do a basic inequality check against the rate
• If we repeat this process multiple times hopefully not all are in/out of sample

I'm wondering if this unit test has much value and should be removed. (Gasp!)

The alternative would be to use statistics since this is a stochastic process. We could implement a confidence interval-based solution (happy to elaborate on that if needed) but it would be VERY computationally expensive – essentially requiring 20K session resets per run.

From @phuedx on Slack:

I wonder how valuable a test it is. Perhaps we could break it out into a couple of unit tests with stubs for the session ID generator and an (expensive?) test that tests it end-to-end that can be run manually?

Yeah, it's not a valuable test as-is. I also was thinking of providing some session IDs that we know to be:

• in-sample at 50%
• out-of-sample at 50%
• in-sample at 1% and double checking that the same ID is also in-sample at 5%, 10%, 50% (I have previously verified that the algorithm behaves in a way that this property is satisfied)

Because yeah, and end-to-end test tests:

• ID generation algorithm produces an ID that has the properties we desire / behaves in the way that we intend
• ID processing algorithm works with the generated ID in the way we desire/intend

And the properties/behavior we want to test something that we need a lot of simulated sessions per run and a lot of simulated runs to verify. Do we have that option? To have tests that aren't run by CI, only manually?

In T304379#9398410, @mpopov wrote:

Do we have that option? To have tests that aren't run by CI, only manually?

We do! We can write a QUnit test in its own file and mark it as skipped. If we want to run it, then we remove the mark, e.g.

QUnit.test.skip( 'mw.user.generateUserSessionID(): ...', ( assert ) => {
} );

// And if want to run it...

QUnit.test( 'mw.user.generateUserSessionID(): ...', ( assert ) => {
} );

Alternatively, we can write a test in Node to be run via the CLI.

Awesome! Okay, when I played around originally to get a sense of how expensive of a procedure it might need to be I generated random numbers, obtained proportions "in-sample", calculated confidence intervals, and checked how many contain the true proportion. I just thought of a much less expensive procedure that might actually be feasible in a CI setup! (Although it might still be expensive enough to make manual.)

Properties/behavior we care about:

1. If a session is in-sample at 1% rate, it should be in-sample at 5%, 10%, 25%, etc.
2. If rate is p, approx p of all sessions would be in-sample. This won't be exact but the more sessions there are (n → ∞), the sample proportion → p

• Generate N session IDs (e.g. N=1000)
• Convert them to decimals via normalizeId, yielding N numbers between 0 and 1
  • Because our algorithm uses <, property #1 will always hold.
  • Now we just need to worry about property #2
• We don't actually need to test if ~10% of sessions are determined to be in sample when sample rate is 0.1, etc. – INSTEAD we just need to verify that the distribution of those N numbers (obtained from "normalizing" the randomly generated session IDs) is approximately Uniform(0,1).
  • Fortunately there's a statistical test for that – the Kolmogorov–Smirnov test. Actually, pages 139-140 of https://www.johndcook.com/Beautiful_Testing_ch10.pdf describe the procedure and suggests thresholds to use.
  • We can even use an existing library to accomplish this: https://www.npmjs.com/package/@stdlib/stats-kstest
    • PASS if pValue >= 0.001 (fail to reject the null hypothesis that the two distributions are the same)
    • FAIL if pValue < 0.001 (reject the null in favor of the alternative hypothesis that the two distributions are different)

We can use https://commons.apache.org/proper/commons-math/javadocs/api-3.6.1/org/apache/commons/math3/stat/inference/KolmogorovSmirnovTest.html in the Java implementation

We can create our own in the Swift implementation and use SciPy's implementation (or the aforementioned Java or the JS versions) for reference.

In T304379#9400845, @mpopov wrote:

Properties/behavior we care about:

1. If a session is in-sample at 1% rate, it should be in-sample at 5%, 10%, 25%, etc.

This isn't tested in either the JS nor the Java Metrics Platform Client. I'll create a task.

2. If rate is p, approx p of all sessions would be in-sample. This won't be exact but the more sessions there are (n → ∞), the sample proportion → p

• Generate N session IDs (e.g. N=1000)
• Convert them to decimals via normalizeId, yielding N numbers between 0 and 1
  • Because our algorithm uses <, property #1 will always hold.
  • Now we just need to worry about property #2
• We don't actually need to test if ~10% of sessions are determined to be in sample when sample rate is 0.1, etc. – INSTEAD we just need to verify that the distribution of those N numbers (obtained from "normalizing" the randomly generated session IDs) is approximately Uniform(0,1).
  • Fortunately there's a statistical test for that – the Kolmogorov–Smirnov test. Actually, pages 139-140 of https://www.johndcook.com/Beautiful_Testing_ch10.pdf describe the procedure and suggests thresholds to use.
  • We can even use an existing library to accomplish this: https://www.npmjs.com/package/@stdlib/stats-kstest
    • PASS if pValue >= 0.001 (fail to reject the null hypothesis that the two distributions are the same)
    • FAIL if pValue < 0.001 (reject the null in favor of the alternative hypothesis that the two distributions are different)

We can use https://commons.apache.org/proper/commons-math/javadocs/api-3.6.1/org/apache/commons/math3/stat/inference/KolmogorovSmirnovTest.html in the Java implementation

We can create our own in the Swift implementation and use SciPy's implementation (or the aforementioned Java or the JS versions) for reference.

I'll create a task (maybe an epic with a subtask per implementation for this). For the JS implementation, we should also be wary of the execution environment, i.e. we should have a test for Node and a test that can be run in a browser.

Once these tasks are Done™, then we can remove this test.

☝️ ☕ required…