Change Details

We need to record when a user is enrolled in an experiment in order to know the total number of users enrolled in the experiment and also the rate at which users were enrolled. We could: 1. Store enrollment status (id, date, bucket) in the `user_options` table 1. Submit an Event Platform event to a stream and store those events in a Hive table 1. … The `user_options` table is familiar and available to both developers and analysts. However, there's no equivalent for logged-out users, the next logical user group that we wish to enable experimentation for. Further, since experiment enrollment will be deterministic, the Enrollment Log can be write-once-read-never (WORN). We can safely discard option 1. Using the Event Platform, therefore, is the solution. However, it also has limitations – namely that it necessarily increases pressure on Varnish, EventGate, and the network; and that submitting > 1000 events/sec per event stream requires special attention from Data Engineering. === Prior Art 1. Growth's `ExperimentUserManager` service class: https://gerrit.wikimedia.org/g/mediawiki/extensions/GrowthExperiments/+/85114e98ce8d59b9475518538009edcddd5eb23f/includes/ExperimentUserManager.php 1. Readers Web's webABTestEnrollment instrument: ** Source: https://gerrit.wikimedia.org/g/mediawiki/extensions/WikimediaEvents/+/3fed0809f97152ec0302eaf51be5a6d9c4a218f7/modules/ext.wikimediaEvents/webABTestEnrollment.js ** Schema: https://schema.wikimedia.org/repositories//secondary/jsonschema/analytics/mediawiki/web_ab_test_enrollment/current.yaml 3. … === Notes 1. We don't necessarily need to record which user saw which treatment. We could use a unique token that represents the user across the lifetime of the experiment, e.g. ```lang=php $uniqueUserID = hash( 'sha256', implode( ':', [ $userID, $experimentID ] ); ```

We need to record when a user is enrolled in an experiment in order to know the total number of users enrolled in the experiment and also the rate at which users were enrolled. We could: 1. Store enrollment status (id, date, bucket) in the `user_options` table 1. Submit an Event Platform event to a stream and store those events in a Hive table 1. … The `user_options` table is familiar and available to both developers and analysts. However, there's no equivalent for logged-out users, the next logical user group that we wish to enable experimentation for. Further, since experiment enrollment will be deterministic, the Enrollment Log can be write-once-read-never (WORN). We can safely discard option 1. Using the Event Platform, therefore, is the solution. However, it also has limitations. Using it necessarily increases pressure on Varnish, EventGate, and the network. Regardless of the fact that submitting > 1000 events/sec per event stream requires special attention from Data Engineering, we cannot submit an event per pageview nor an event per 4 pageviews (?). === Prior Art 1. Growth's `ExperimentUserManager` service class: https://gerrit.wikimedia.org/g/mediawiki/extensions/GrowthExperiments/+/85114e98ce8d59b9475518538009edcddd5eb23f/includes/ExperimentUserManager.php 1. Readers Web's webABTestEnrollment instrument: ** Source: https://gerrit.wikimedia.org/g/mediawiki/extensions/WikimediaEvents/+/3fed0809f97152ec0302eaf51be5a6d9c4a218f7/modules/ext.wikimediaEvents/webABTestEnrollment.js ** Schema: https://schema.wikimedia.org/repositories//secondary/jsonschema/analytics/mediawiki/web_ab_test_enrollment/current.yaml 3. … === Notes 1. We don't necessarily need to record which user saw which treatment. We could use a unique token that represents the user across the lifetime of the experiment, e.g. ```lang=php $uniqueUserID = hash( 'sha256', implode( ':', [ $userID, $experimentID ] ); ```

We need to record when a user is enrolled in an experiment in order to know the total number of users enrolled in the experiment and also the rate at which users were enrolled. We could: 1. Store enrollment status (id, date, bucket) in the `user_options` table 1. Submit an Event Platform event to a stream and store those events in a Hive table 1. … The `user_options` table is familiar and available to both developers and analysts. However, there's no equivalent for logged-out users, the next logical user group that we wish to enable experimentation for. Further, since experiment enrollment will be deterministic, the Enrollment Log can be write-once-read-never (WORN). We can safely discard option 1. Using the Event Platform, therefore, is the solution. However, it also has limitations – namely that. Using it necessarily increases pressure on Varnish, EventGate, and the network. and the network;Regardless of the fact that submitting > 1000 events/sec per event stream requires special attention from Data Engineering, and that we cannot submitting > 1000t an events/sec per event stream requires special attention from Data Engineeringpageview nor an event per 4 pageviews (?). === Prior Art 1. Growth's `ExperimentUserManager` service class: https://gerrit.wikimedia.org/g/mediawiki/extensions/GrowthExperiments/+/85114e98ce8d59b9475518538009edcddd5eb23f/includes/ExperimentUserManager.php 1. Readers Web's webABTestEnrollment instrument: ** Source: https://gerrit.wikimedia.org/g/mediawiki/extensions/WikimediaEvents/+/3fed0809f97152ec0302eaf51be5a6d9c4a218f7/modules/ext.wikimediaEvents/webABTestEnrollment.js ** Schema: https://schema.wikimedia.org/repositories//secondary/jsonschema/analytics/mediawiki/web_ab_test_enrollment/current.yaml 3. … === Notes 1. We don't necessarily need to record which user saw which treatment. We could use a unique token that represents the user across the lifetime of the experiment, e.g. ```lang=php $uniqueUserID = hash( 'sha256', implode( ':', [ $userID, $experimentID ] ); ```