Background
GrowthExperiments currently sends a Getting Started notification 48 hours after an user account is registered, assuming the user did not make enough(*) edits yet. This delay is implemented using a delayed job. Since T394957, the delay of the job can be configured per wiki and user variant, allowing an experiment to be held.
There are several configuration options that need to be changed to make this possible:
- wgGELevelingUpGetStartedNotificationSendAfterSeconds, to request the job to be executed with an appropriate jobReleaseTimestamp,
- reenqueue_delay in changeprop, to ensure the jobs are actually executed at a time reasonably matching the requested jobReleaseTimestamp
Because of uncertainities regarding changeprop's behaviour (which were not sufficiently clarified during the T393955 research spike), the Growth engineers are unsure about the appropriate value of reenqueue_delay. This means making this change more riskier than normal deployments.
Checklist
- Set reenqueue_delay for notificationGetStartedJob to 30 minutes
- Complete T394957: Support delaying NotificationGetStartedJob differently based on user variant
- For users in the get-started-notification variant on pilot wikis (eswiki, arwiki, frwiki), shorten the notification delay to 20 hours
- Observe the notification behaviour (done largely in T395260: Measure the difference between the intended and actual execution of delayed notification jobs)
Acceptance Criteria
- No change is observed outside of the pilot wikis (eswiki, arwiki)
- On the pilot wikis, 50% of new users are assigned to the control variant and 50% to the get-started-notification variant.
- Users in the get-started-notification variant receive the Getting started notification between 19 and 21 hours since their registration (desired delay is 20 hours, with an error margin of 1 hour on both sides).
- Users in the control variant receive the Getting started notification after 48 hours since their registration
- The conditions for the notification to be fired remain unchanged (no suggested edits + less than GELevelingUpGetStartedMaxTotalEdits edits).
Possible risks
The risks of this change are contained to the job queue. The following (undesired) behaviours have a particularly significant chance of happening (ordered by impact from lowest to highest):
- the getting started notification is being delivered, but not at the right time (see A/C for details on when the notification is supposed to arrive)
- the getting started notification is being delivered, but only on some wikis (it should be sufficient to test this at both pilot wikis and an arbitrary non-pilot wiki)
- the getting started notification is not being delivered at all
- jobs stop getting executed reliably