@Samwalton9-WMF Are we still aiming to have a sufficient sample size to establish a baseline by 31 March? I wanted to confirm the number of weeks available for data collection to recommend some weekly visitor targets.

Based on current rates, we will likely have sufficient data to establish baselines for clickthrough rate metrics; however, the retention rate metric currently requires a 14-day observation window. As a result, the initial cohort for retention would be limited to users who have visited the dashboard prior to this week to be included in a retention rate calculation by end of March. That would mostly be users who visited the dashboard before the "loud" deployment.

In T418810#11744524, @MNeisler wrote:

@Samwalton9-WMF Are we still aiming to have a sufficient sample size to establish a baseline by 31 March? I wanted to confirm the number of weeks available for data collection to recommend some weekly visitor targets.

Because we launched slightly late we anticipate running the KR into mid-April, so we could set the end date as ~14th April instead?

Based on current rates, we will likely have sufficient data to establish baselines for clickthrough rate metrics; however, the retention rate metric currently requires a 14-day observation window. As a result, the initial cohort for retention would be limited to users who have visited the dashboard prior to this week to be included in a retention rate calculation by end of March. That would mostly be users who visited the dashboard before the "loud" deployment.

Yeah I think we should avoid this, since the 'real' invites started on the 19th, 14 days would take us to 2nd April.

Below are some proposed weekly visit targets based on the new KR end date (30 April):

Current Status
There have been 874 unique visitors to the dashboard from 19 March (when the real invites were sent out) through today (26 March).

While this is a large sample size for the initial first-week cohort, it's also heavily weighted by an initial spike after the real invites were initially sent out (390 visitors on 19 March).

Rates have decreased to around 70 unique visitors per day (which appears stable at the moment). We want to monitor these rates to ensure a sufficient sample of users visit during this steady-state time period (at least 200 new users per week), as they will be more representative of how the dashboard will perform in the long run.

Proposed Weekly Visitor Targets:

200 additional users will keep the margin of error low for any single cohort and provide a total sample size of close to 1500 users. This is a statistically high-powered sample size, which will allow us to effectively measure impacts from future iterations.

If any week falls below 200 users, then the daily visitor rate has decreased significantly, and we may want to consider ways to increase visitor rates.

@MNeisler Thanks for this analysis! I believe we decided on 23rd April rather than 30th April, to aim for 5 weeks since 19th March? Does that change the numbers substantially?

Hi! Ah yes sorry I was going off this slack update but we did discuss ending data collection on 23rd April.

No, an April 23rd end date shouldn't change the proposed weekly targets substantially. Based on current daily visitor rate (~50 to 60 users per day), we should still have a sufficient sample size to establish a baseline for retention rate by that time as long as there are no significant declines in visitors.

The primary change is the date of the final cohort. To meet the 23 April end date, our final acquisition window to include users in the retention rate metric will be 9 April (allowing for the 14-day observation window).

Below are some slightly adjusted proposed weekly targets to ensure that we have a sufficient sample size and a representative sample for each weekly cohort.

Proposed Weekly Visitor Targets:

Thanks! These seem achievable, especially given our impending launch on enwiki, so I'm not going to worry too much about preparing alternative paths to get users to visit the dashboard.