For this trial run in particular, we're interested in:
- Conversions for each bucket, as a percent of impressions
- Number of users indefintiely blocked from each bucket n days after conversion
- What are the IP ranges used for registration, and are there any egregious sockpuppeteers?
- What is the rate of email authentication for control and test?
Detailed description
- Funnel analysis
To measure CTR/CNV rates we will deduplicate all events by token. Every token will be associated with a vector with a boolean value for each event type (impression, submit, create):
- token,1,0,0 => user landed at least once on the page but didn't click on submit
- token,1,1,0 => user landed at least once on the page and clicked at least once on submit, but never got a successfully created account
- token,1,1,1 => user successfully completed the whole funnel
Log clean up requires:
- discarding all tokens with no impression
- discarding all account_create events with self_made=0
- discarding all bucket-less or token-less account_create events
- discarding multiple account_create events from same token
- Error estimates
Even without error logging, we can get a rough estimate of whether users in a given bucket experience a higher rate of errors at any stage of the funnel by plotting the distribution of event counts per token. We won't be able to tell whether multiple impressions are due to reloading the landing page or producing several errors but at least we will get a sense of the volume of multiple events per bucket.
- Time to threshold
Only for those users who completed the funnel, we will look at the time it takes to complete the first edit (or how many users complete an edit at all), as long as the other metrics originally planned (block rate and email authentication rate).
Trello card: QNygrDIf
- column: Done
- due: 2012-10-12T00:00:00.000Z