We would like to run a controlled experiment to see how section editing affects editing behavior. To that end, we would like to instrument as follows:
- Split registered users 50%-50% between section editing and the current experience. Unregistered editors can get either experience, but I would recommend sticking with the current experience until we are done with the test. The bucketing should be deterministic based on the user ID (e.g user ID mod 2) so that I can later identify data corresponding to each group based on the user ID.
- Avoid bouncing users between multiple experiences (as @Esanders reminded me). Since section editing is already rolled out to all users on Hebrew, Cantonese, and Bengali Wikipedias, the bucketing should be kept off for these wikis. This also helps us get good qualitive feedback from these wikis without the confusion of half the users not getting the feature. On the other hand, the bucketing should start the moment the feature is deployed on all other wikis. The first additional deployment is scheduled for Monday, March 25. @ppelberg and I agree that we should delay this if necessary to allow time for instrumentation.
There are some things I have previously suggested, but I don't think are useful right now:
- Bucket based on a hash of the user ID and the experiment name. This feels like a best practice so that we aren't using the exact same bucketing logic as any concurrent experiments, but our current analysis database version doesn't support any widely-used hash functions (T203498). I'm pretty sure there aren't any concurrent experiments going on, so it doesn't seem worth the effort.
- Create a bucket field in the EditAttemptStep schema and send a group name along with any events. The user ID is permanently kept in our dataset, so I can just reapply the bucketing logic when fetching data rather than doing the extra instrumentation work.