Page MenuHomePhabricator

Add test for uniform distribution of bucket assignments in Metrics Platform
Open, LowPublic3 Estimated Story Points

Description

As a follow up to T375900: Allow users to override experiment enrollment, we should prove uniform distribution of user enrollment assignments in experiments managed by Metrics Platform.

Description

Add another test to the UserSplitterInstrumentation class in the Metrics Platform extension by injecting randomness into the user ID value (user hash) and checking that resultant variant values have set membership instead of evaluating to a deterministic value when there are multiple variant values.

See https://gitlab.wikimedia.org/phuedx/id-distribution-test as an example of an ID distribution test.

Acceptance Criteria

  • Relevant test is added and passes

Required

  • Unit/Integration tests?
  • Documentation?
  • Passed QA?

Event Timeline

cjming set the point value for this task to 3.
cjming moved this task from Incoming to BACKLOG on the Test Kitchen board.

@mpopov Is this still important given that we check for sample ratio mismatch in automated analytics? Or is it adding a check further upstream?

It's adding a check upstream, but specifically for the MW central ID enrollment.

Hm… Thinking about @phuedx's T408133: [Spike] Explore Generalizing Enrollment Authorities it would be nice to have a generalized mechanism for verifying that an enrollment authority has the desired bucketing behavior.

Two things that an enrollment authority should be good at are:

  • randomly generating and managing an identifier
  • converting that identifier into an enrollment/assignment determination that respects the desired split (whether it's 50/50, 90/10, or 25/25/25/25)

I think it would be good if we required that any additional enrollment authorities had to prove themselves that they are up to the task.

If we did this, I can imagine us being able to generalize that so that more enrollment authorities could be tested.