Background
GrowthExperiments currently sends a Getting Started notification 48 hours after an user account is registered, assuming the user did not make enough(*) edits yet. This is technically implemented by a delayed job, using the jobReleaseTimestamp option. The delay is configurable in server configuration (GELevelingUpGetStartedNotificationSendAfterSeconds), but it is the same for all users at a given wiki.
(*) No suggested edits and less than GELevelingUpGetStartedMaxTotalEdits total edits.
Problem
In the parent task, we want to A/B test a shorter delay of the notification. To be able to do that, the job delay needs to support A/B testing.
Proposed solution
We can create a new user variant for this experiment, which would be factored for computing the delay. We can restructure the GELevelingUpGetStartedNotificationSendAfterSeconds option (currently an integer) by making it a map from user variant to delay (+default key when nothing matches). For example, the following configuration would delay the notification by 20 hours for users assigned to quick-gettingstarted and 48 hours for any other user.
$wgGELevelingUpGetStartedNotificationSendAfterSeconds = [ 'quick-gettingstarted' => 20 * 3600, 'default' => 48 * 3600, ];
While at this, it would probably make sense to move the actual enqueueing call to a levelling-up specific place (LevelingUpManager?), instead of HomepageHooks (which is already quite overcrowded).
Potential challenges
- No reserved variant names: technically, we could create an user variant called default. This would then clash with the default key proposed above. Given we already have a default variant (called control), I consider introduction of a similarly-named variant like default to be a relatively unlikely event. For this reason, I don't think we need to do anything special about this.
- Variant querying is error-prone in onLocalUserCreated: The notification is enqueued within the onLocalUserCreated hook. Sometimes, accessing the central user IDs in this hook (which we would have to do) fails and returns a zero, which would break any bucketing we try to do here. See T380500: CentralAuthUser returning outdated data after user creation for more details.