@mpopov Is this still important given that we check for sample ratio mismatch in automated analytics? Or is it adding a check further upstream?

It's adding a check upstream, but specifically for the MW central ID enrollment.

Hm… Thinking about @phuedx's T408133: [Spike] Explore Generalizing Enrollment Authorities it would be nice to have a generalized mechanism for verifying that an enrollment authority has the desired bucketing behavior.

Two things that an enrollment authority should be good at are:

• randomly generating and managing an identifier
• converting that identifier into an enrollment/assignment determination that respects the desired split (whether it's 50/50, 90/10, or 25/25/25/25)

I think it would be good if we required that any additional enrollment authorities had to prove themselves that they are up to the task.

If we did this, I can imagine us being able to generalize that so that more enrollment authorities could be tested.

Thank you for tagging this task with good first task for Wikimedia newcomers!

Newcomers often may not be aware of things that may seem obvious to seasoned contributors, so please take a moment to reflect on how this task might look to somebody who has never contributed to Wikimedia projects.

A good first task is a self-contained, non-controversial task with a clear approach. It should be well-described with pointers to help a completely new contributor, for example it should clearly point to the codebase URL and provide clear steps to help a contributor get set up for success. We've included some guidelines at https://phabricator.wikimedia.org/tag/good_first_task/ !

Thank you for helping us drive new contributions to our projects <3

Hi! I’m interested in working on this task. I’m a new contributor and would like to take it. Please let me know if it’s still available.

Change #1303909 had a related patch set uploaded (by Pushpaktiwari; author: Pushpaktiwari):

[mediawiki/extensions/TestKitchen@master] Add test for uniform bucket assignments

https://gerrit.wikimedia.org/r/1303909

I have submitted a patch for this task.

Gerrit change: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/TestKitchen/+/1303909

The patch adds a unit test for UserSplitterInstrumentation that generates many user hashes and verifies that bucket assignments remain approximately balanced between the control and treatment variants.

I also ran PHPUnit locally and all tests pass (33 tests, 61 assertions).

From my review of the patch:

Thank you for attempting to write this test. Unfortunately, it's not actually testing what we're looking to test.

This test is an alterative way of testing that UserSplitterInstrumentation#getUserHash() is deterministic – we send in the same 10000 IDs and get the same hashes back. However, we already have a test that proves that UserSplitterInstrumentation#getUserHash() is deterministic, #testGetUserHash() above. We know this because the #provideGetUserHash() data provider always provides the same fixtures.

So how do we test that the output of UserSplitterInstrumentation#getUserHash() is uniform? For this we need to:

1. Provide a random sample of central user IDs
2. Use the central user IDs from (1) to get a sample of hashes from UserSplitterInstrumentation#getUserHash()
3. Get a random sample of the values from the reference distribution (the uniform distribution)
4. Use the two-sample Kolmogorov-Smirnov test to accept or reject the null hypothesis that the sample in (2) and the sample in (3) are from the same distribution (the uniform distribution)
5. In practice, (1) and (2) can be combined and will look like:

$MAX_CENTRAL_USER_ID = PHP_INT_MAX; // TODO
$EXPERIMENT_NAME = __METHOD__;

$hashes = [];

for ( $i = 0; $i < 10000; ++$i ) {
 $centralUserID = random_int() / $MAX_CENTRAL_USER_ID;
 $hashes[] = $subject->getUserHash( $centralUserID, $EXPERIMENT_NAME );
}

And (3) will look the same:

$uniform = [];

for ( $i = 0; $i < 10000; ++$i ) {
 $uniform = random_int() / PHP_INT_MAX;
}

However, as far as I can tell, there is no PHP implementation of the two-sample Kolmogorov-Smirnov test that can be used easily. We might have to implement it ourselves.

and then:

Sorry. This comment wasn't quite correct.

We can use the one-sample Kolmogorov-Smirnov test to accept or reject the null hypothesis that the sample in (2) is from the uniform distribution. That way, we don't need to generate a set of random numbers (which may have a bias of their own).

Implementing the one-sample Kolmogorov-Smirnov test is non-trivial and so I've removed the good first task tag.