In T257490, the Russian Wikipedia community explained that they would be interested in a report like this one, which would hopefully show that the welcome survey does not negatively impact newcomers. Therefore, we'll deploy the welcome survey as an A/B test there.
After about 6 weeks (or whatever time period @nettrom_WMF recommends) we'll want to analyze the test.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Goal | Trizek-WMF | T228980 [goal] Deploy Growth features on Wikipedias | ||
| Resolved | Catrope | T257490 Deploy Growth experiments at Russian Wikipedia | |||
| Resolved | MMiller_WMF | T259768 Scale: analyze Russian welcome survey |
@Iniquity -- thanks for bringing this back up. We do intend to do this. I will schedule a time for this work with our team's data analyst and let you know.
We are on it. :) It will take more time then expected since we are doing it for all wikis where the survey is deployed.
@Iniquity -- we plan to do this during the month of December. I'll let you know if the date changes.
This analysis is now ready for review. I've uploaded the notebook with the analysis to GitHub.
We've never had randomization fail, and when we ran Help Panel and Homepage experiments at the same time randomization across experiment groups worked flawlessly. Yet, I decided to run a sanity check on the survey group assignment against Growth feature group assignment, confirming that randomization was functioning correctly and that the split was 20%/80% Control/Survey (as described in T257490#6395284) and 20%/80% Control/Growth features. This means that the effect of the Growth features is expected to distribute evenly across the survey groups and allows us to compare them.
I then define the start and end time periods to be used based on deployment (ref T257490#6416979) and the most recently available MediaWiki history snapshot, because MW history is our preferred source of truth. We only use on-wiki registrations because other accounts aren't expected to see the Welcome Survey, and exclude bots. Correlating with the survey data we find 15.5% of registrations don't have survey group assignment. Based on previous analysis of registrations, I expect these to be app registrations. Since they do not see the Welcome Survey, we exclude them from the analysis.
The dataset consists of 30,475 users, and if a user edits within 24 hours of registration they are defined as "activated". We get the following counts and proportions of users:
| Survey Group | Activated? | N users | Percent of survey group |
|---|---|---|---|
| Control | No | 3, 882 | 63.6 |
| Control | Yes | 2,220 | 36.4 |
| Survey | No | 15,300 | 62.8 |
| Survey | Yes | 9,073 | 37.2 |
A chi-square test of proportions using these counts finds that the differences are not significant (X^2 = 1.45, df = 1, p = 0.23). In other words, the Welcome Survey does not appear to have a significant impact on activation in the Russian Wikipedia.
@Iniquity : let me know if you or the Russian community has questions about the analysis and I'll do my best to answer.
Handing this off to @MMiller_WMF for review and sign-off.
This work is completed. @Iniquity -- we'll be working on other aggregates in a separate task.