Change #1180215 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1180215

That patch, as well as setting up the tone a/b test, lets us run a/b tests of any current experimental check. Just one per wiki though.

In T402195#11100251, @DLynch wrote:

That patch, as well as setting up the tone a/b test, lets us run a/b tests of any current experimental check. Just one per wiki though.

To confirm: as currently implemented, this patch enables us to run a/b tests of different Edit Checks (e.g. Paste Check, Reference Check, and Tone Check) concurrently provided there are no plans for two different Checks to be tested on the same wiki(s) at the same time.

@DLynch: to what extent – if any – is the above inaccurate?

Assuming it's aligned with what's currently implemented, I think this sounds good. [i] cc @MNeisler

i. Reference Check a/b test is scheduled to run on en.wiki
ii. Tone Check a/b test is scheduled to run, for now, on fr.wiki, ja.wiki, es.wiki (maybe), pt.wiki (maybe)
iii. Paste Check a/b test can be run on wikis besides en.wiki, fr.wiki, ja.wiki, es.wiki, and pt.wiki

That's accurate!

We can set the VisualEditorEditCheckABTest site config to the name of an experimental check. It will then run an a/b test where 50/50 of both logged in and logged out users are persistently given that check enabled/disabled. For @MNeisler, those will have the bucket field in the schema set to 2025-09-editcheck-[checkname]-[test/control]

It'd be trivial to expand this to turning on several checks at once, but making it so that we can independently a/b test different checks simultaneously would require a different approach.

I will need to amend the patch slightly when we get to that enwiki reference test, because it didn't consider the possibility of being asked to handle a a test like that which needs a slightly different setup. (It's *not* an experimental check -- the enwiki a/b test here is essentially "edit check is completely turned off" vs "edit check is turned on normally".)

Change #1180215 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1180215

Change #1184115 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.16] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1184115

Change #1184115 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@wmf/1.45.0-wmf.16] Edit check: set up the tone check a/b test

https://gerrit.wikimedia.org/r/1184115

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:18:36Z] <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]]

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:24:49Z] <kemayo@deploy1003> kemayo: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-09-02T20:31:11Z] <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1184115|Edit check: set up the tone check a/b test (T389231 T402195)]], [[gerrit:1184121|Edit check: log to VEFU if a tone check would have been shown if not for the a/b test (T394952)]] (duration: 12m 34s)

@DLynch: what – if anything – is left to be done here?

Note: I'm stepping through all of the sub-tasks of T399669 to identify what's left to be done before we can begin the test.

@ppelberg we've only implemented the first option ("Different checks being tested at the same time on different wikis"), so it's down to whether we want any of the others. I don't believe we do currently, as I think the paste wikis will not be overlapping with the tone wikis.

In T402195#11227210, @DLynch wrote:

@ppelberg we've only implemented the first option ("Different checks being tested at the same time on different wikis"), so it's down to whether we want any of the others. I don't believe we do currently, as I think the paste wikis will not be overlapping with the tone wikis.

Agreed! I'll create a new placeholder task for the remaining two.

Thank you, David.