Details of the Reply Tool AB test set-up for reference:

• Run from 11 February 2021 through 10 March 2021 (4 weeks)
• Included logged-in users that have not previously interacted with the reply tool (defined as users whose discussiontools-editmode preference is empty)
• Included 22 Wikipedias
• During this test, 50% of users included in the test had the Reply tool automatically enabled, and 50% did not. Users at these Wikipedias were still able to turn the tool on or off the tool in Special:Preferences.
• Data logged during that timeframe:
  • About 1300 edits attempts by Junior Contributors in each test group
  • A little over 2500 edit attempts across all experience levels in each test group

I will use this information to help determine the AB test set-up requirements for the New Discussion Tool

Here are my current recommended estimates:

The number of wikis the A/B test would need to be run on

I'd recommend running the test on 15+ wikis at minimum including a combination of large and medium-sized wikis. Small wikis can also be included as well but this should be in addition to the 15 large and medium-sized wikis to ensure we have a sufficient number of wikis with a representative sample of data.

Rationale
(1) Size of the wikis included: In this proposed AB test, each user would serve as our observational unit. As a result, we would need a sufficient number of distinct users across all participating wikis to calculate the metrics above. If we included larger wikis in this test such as English Wikipedia or Spanish Wikipedia then we'd need a fewer number of total wikis to obtain a sufficient sample of users. If we included only smaller wikis, we'd likely need to increase this number to get a sufficient number of users.
(2) Accounting for the effect of the wiki on the metrics. In the Reply Tool AB test, we accounted for the effect of the wiki on the success probability of a Junior Contributor completing an edit, which allowed us to more accurately understand the effect of the reply tool. I'd recommend we do the same in this test. The more distinct wikis we are able to include in the test, the more data we will have to account for the effect of each wiki.
(3) Obtaining a representative sample for each wiki In the Reply Tool Test, there were 22 participating wikis but only about 15 of those wikis including the larger and mid-sized wikis included sufficient events to be included in the analysis. Smaller wikis such as Swahilli and Afrikanns had only a couple of events logged during the Reply Tool test and as a result we were not able to make any conclusions about these wikis.

For how long the A/B test would need to be run on

I'd recommend running the test for a minimum of 4 weeks similar to the Reply Tool. After the 4 week period, I'd recommend reviewing the number of events logged to confirm it is sufficient for the analysis prior to ending the test. Note: This assumes we are running the test on 15+ large and small wikis. The test might need to be run longer if a smaller set of wikis are included.

Rationale:
A review of daily edit attempts indicates that there are roughly twice as many daily reply tool edit attempts compared to new discussion tool edit attempts (looking at wikis where both are deployed as opt-in features). If we assume the same ratio, a 4 week test on a similar set of participating wikis would give us around 600 edit attempts by junior contributors and 1250 edit attempts across all experience levels. While this is less than the Reply Tool, it would be sufficient to complete the analysis and provide data on weekday/weekend trends but more data is always helpful if possible.