@Samwalton9-WMF Some initial guidance about sample size needed below:

Sample Size Estimate

• In general, the sample size requirements for establishing a baseline are less stringent than the requirements for detecting a change in an AB test. However, we need a sufficient number of distinct users to achieve a representative sample so the data doesn't just reflect a few power users.
• Establishing a baseline for click-through rates on the homepage will be much easier than establishing a baseline for retention rates. With simple engagement, such as clicks, we will get data immediately for everyone who visits the homepage. With retention rates, we expect only a small proportion (~10%) of those users to return, which reduces the effective sample size.
• Based on this, I estimate we'd need a minimum of 100 distinct users across all pilot wikis that engage with the homepage. Note: This is the absolute minimum sample size we'd need to be able to establish baseline estimates. This will provide us with the data needed to confirm interaction rates and retention rates; however, there will be higher margin of error for retention rates. Additional users (ideally up to 200) will help increase confidence in these numbers.

Required visits to moderator homepage
The available data for baseline analysis will also depend on the proportion of invited users that click through to visit the homepage.

If we estimate that we will invite 500 users to the homepage over an 8 week time period (as indicated in T402802), then we would need at least 15% of those users to visit the homepage to achieve 100 distinct users, which is very high click through rates for the current invite approach. I'm currently checking if there is some existing research to confirm click through rates on menus invited by blue dot notification but I'd estimate we'd typically see rates around 5 to 10%.

Based on this, I'd recommend either (1) exploring approaches to more actively get users to the homepage and/or (2) roll out to more pilot wikis if feasible, to increase the number of users invited. We also discussed these options in the WE 1.3 Steering Commitee Meeting on 13 Jan (notes).

@Samwalton9-WMF:

Quick clarifying question: The user criteria specified in T402647 states:

To be opted in to the experiment, registered accounts must have made 100 edits. We have decided not to include an upper limit on edit count or tenure.

To confirm, does this mean we are removing any limits on user tenure? If so, this increases the sample size significantly from the original estimate of 500 users. I rechecked the number of editors at the pilot wikis (idwiki, thwiki, trwiki, simplewiki) that met the following criteria:

Criteria 1:

• Made at least one edit in the last 8 weeks.
• Have made 100+ cumulative edits
• No tenure requirement

Number of Users: 2800 users

Criteria 2:

• Made at least one edit in the last 8 weeks.
• Have made 100+ cumulative edits
• Registered their account no more than 3 years ago

Number of Users: 570 users

If we remove the tenure requirements and estimate that we will invite 2800 users over an 8-week time period, then we need only about 5% of users to visit the homepage to achieve our goal, which seems more realistic.