15:47:49 Minerva AB-test 15:47:49 ✖ Bucketing test 15:47:49 HeadlessChrome 73.0.3683 (Linux 0.0.0) 15:47:49 test control group is about 25% (30.8%) 15:47:49 Expected: true 15:47:49 Actual: false
This seems to be doing something stochastic, which is not cool in a test and especially not in a gated repo. From tests/qunit/skins.minerva.scripts/AB.test.js,
assert.strictEqual( ( userBuckets.unsampled / maxUsers > 0.4 ) && ( userBuckets.unsampled / maxUsers < 0.6 ), true, 'test unsampled group is about 50% (' + userBuckets.unsampled / 10 + '%)' ); assert.strictEqual( ( userBuckets.control / maxUsers > 0.2 ) && ( userBuckets.control / maxUsers < 0.3 ), true, 'test control group is about 25% (' + userBuckets.control / 10 + '%)' ); assert.strictEqual( ( userBuckets.treatment / maxUsers > 0.2 ) && ( userBuckets.treatment / maxUsers < 0.3 ), true, 'test new treatment group is about 25% (' + userBuckets.treatment / 10 + '%)' );
The right approach is to manipulate the random number generator, mock the internals of the class, whatever it takes, but failing some percentage of the time is a problem for everyone.