Background

Currently, we have one agreed-upon standard retention metric for new users, called second-month retention and defined as "of the new users who edited within the first 30 days after their registration, how many edited in the second 30 days after their registration?"

This is based on the definition of surviving new editor, which one change: rather than selecting from new users who edited within the first day after their registration, it changes the period to 30 days.

I (@Neil_P._Quinn_WMF) made that change so the metric would more closely parallel the new active survival rate from the editor model. I consulted with @Halfak about the change and he responded:

The only reason I set up that metric with a 1 day activation is so that the metrics (surviving new editor and new editor) were strictly hierarchical. I don't think it is important. If we do want to do the editor model thing again, then it'll be useful to keep.

When defining metrics for the 2018-19 Audiences annual plan, a meeting of analysts and product managers settled on a slight variation of this metric for retention of editors within individual features:

We are comfortable with adopting the "feature-based" retention definition, which takes the user's first edit on the corresponding platform as point of origin, as opposed to the date of registration. (Platform being one of mobile web / iOS app / Android app, or all three together for [overall mobile retention])

There was some conversation about the choice of retention period (activity between 1-2 months after the first edit), considering that e.g. for app user retention, a shorter limit of 7 days after install is a more common. But we are comfortable with continuing to use the 1-2 month choice, which appears to go back to Aaron's standardization work at https://meta.wikimedia.org/wiki/Research:Surviving_new_editor .

One concern about using such a feature-based retention metric is that it might not sufficiently focus on new editors (as in new to Wikipedia overall), for those teams who are targeting such users next FY. However, this can be addressed by restricting the metric to recent signups. (This corresponds to choosing the parameter t1 in https://meta.wikimedia.org/wiki/Research:Surviving_new_editor .)

Standardization and improvement

We should revise and expand as necessary in order to turn this single metric into a coherent suite of retention metrics.

Potential areas for improvement include (please edit!):

• Complementary, shorter-term metrics: Two months is too long to wait for results when a team is doing a product experiment, so one or two standard short versions would be valuable.
• Complementary, longer-term metrics: The case for a long version is less clear (for example, @Halfak's sensitivity analysis concluded there wasn't much need for it), but there is probably still value. For example, a longer-term metric was one of the things requested in T203138.
• Extension for existing editors: Computing retention among existing users is also valuable, and it would be ideal to make the metrics as analogous as possible to the metrics for new users so end users can easily understand them.
• Aligning with industry standards. It seems like our second-month retention definition may not align with the standard web analytics practices for computing retention, and it could be an advantage to move closer to those standard practices (for example, so we could more easily benchmark our own retention to similar platforms like StackExchange or Reddit).
• Better defining "user birth". General second-month retention takes a user's registration as their birth, while feature-specific retention takes a user's first use of a feature as their birth. The metrics might be better matched if general retention took the user's first edit as their birth, since many users sign up without any intention to edit.