@ppelberg (+ @MNeisler?): I added T387918: [MILESTONE] Run an A/B test to evaluate impact of Tone Check as a parent to this because we (Growth) are blocked on crucial decisions on the outcome of this task.

Please let me know if there is anything we can do to support your team working on this and getting the scaffolding ready for the A/B test.

We're looking to learn both about the volume of the rejected Tone Checks, but also their quality. Would it be possible to record somehow also the paragraph (literal text) of the tone checks that were rejected / not acted upon?

Once the a/b test is running you'd be able to query recent changes for ones that're tagged with editcheck-tone ("this edit has tone issues") and editcheck-tone-shown ("the user was shown a tone check") which should mean edits which rejected a tone check. You'd need to do a bit of manual examination of the revision to guess the quality -- but most of them are going to be edits within a single paragraph so it'll be obvious.

In T397885#11043823, @DLynch wrote:

Once the a/b test is running you'd be able to query recent changes for ones that're tagged with editcheck-tone ("this edit has tone issues") and editcheck-tone-shown ("the user was shown a tone check") which should mean edits which rejected a tone check. You'd need to do a bit of manual examination of the revision to guess the quality -- but most of them are going to be edits within a single paragraph so it'll be obvious.

That makes sense, thank you! Then we'll aim to be ready for this when the A/B tests starts.

In T397885#10999900, @Michael wrote:

@ppelberg (+ @MNeisler?): I added T387918: [MILESTONE] Run an A/B test to evaluate impact of Tone Check as a parent to this because we (Growth) are blocked on crucial decisions on the outcome of this task.

Thank you for letting us know, @Michael !

FYI: I made T387918 a sub-task of this one seeing as how, it seems to be, blocking y'alls (Growth) work her.