AB test was deployed on 19 Feb 2024. See patches noted in T342930

I will QA aggregate data logged to date to confirm data appears as expected based on bucketing requirements.

I reviewed the Edit Check (references) AB test data logged in EditAttemptStep and VisualEditorFeatureUse to confirm it appears expected based on bucketing requirements. Based on my findings (see details below), the data appears as expected and I don't believe any further checks or changes are needed.

Bucketing on ready vs init:
We are logging more ready AB test events than init AB test events. Per discussions with @DLynch, this is because the init event frequently fires before the article target loader can decide whether we’re being bucketed. As a result, I will need to use the ready and subsequent editing events for the analysis.

Our KPIs for the AB test do not depend on init or init_timing so this should be fine for the purposes of this analysis. Edit completion rate, one of the identified guardrails, will need to be calculated based on ready instead of init; however, we're looking for large spikes or drops in that metric and I don't expect that change would have a significant impact.

Summary of Other Checks (Data reflects events logged from 19 Feb 2024 through 21 Feb 2024)

• Confirmed AB test events start logging on 19 Feb 2024 and we are logging events for all expected AB test wikis.
• AB test buckets are assigned to the events so we can decipher each test group: 2024-02-editcheck-reference-control and `2024-02-editcheck-reference-test. Note: We decided to assign more specific strings to the bucket field to specify the AB test. This will help avoid any future hardships in case we need to run multiple AB tests at the same time.

• Bucket Balance: Numbers appear as expected based on a 50/50 per wiki split. See number of sessions and distinct users per test group and wiki below for desktop:

wiki	experiment_group	n_sessions	n_users
<chr>	<chr>	<int>	<int>
afwiki	2024-02-editcheck-reference-control	468	218
afwiki	2024-02-editcheck-reference-test	645	243
arwiki	2024-02-editcheck-reference-control	6611	2925
arwiki	2024-02-editcheck-reference-test	6156	2858
eswiki	2024-02-editcheck-reference-control	44046	26520
eswiki	2024-02-editcheck-reference-test	44812	26010
frwiki	2024-02-editcheck-reference-control	41149	21549
frwiki	2024-02-editcheck-reference-test	40552	21564
itwiki	2024-02-editcheck-reference-control	22484	11620
itwiki	2024-02-editcheck-reference-test	24373	11915
jawiki	2024-02-editcheck-reference-control	51167	32382
jawiki	2024-02-editcheck-reference-test	51779	32489
ptwiki	2024-02-editcheck-reference-control	10055	5911
ptwiki	2024-02-editcheck-reference-test	9864	5809
swwiki	2024-02-editcheck-reference-control	52	20
swwiki	2024-02-editcheck-reference-test	57	13
viwiki	2024-02-editcheck-reference-control	7485	5028
viwiki	2024-02-editcheck-reference-test	7756	5012
yowiki	2024-02-editcheck-reference-control	10	4
yowiki	2024-02-editcheck-reference-test	19	7
zhwiki	2024-02-editcheck-reference-control	31681	20291
zhwiki	2024-02-editcheck-reference-test	31296	20302

• Events are logging for both registered and unregistered users.
• An anonymous user token is applied to all AB test events by unregistered users so we can track distinct unregistered users
• Confirmed we are logging bucket data for all desktop skin types.
• Events are logging on both desktop and mobile

Desktop AB Test Events (Based on ready):

experiment_group	n_sessions	n_users
<chr>	<int>	<int>
2024-02-editcheck-reference-control	215208	126462
2024-02-editcheck-reference-test	217309	126215

Mobile AB Test Events (Based on ready)

experiment_group	n_sessions	n_users
<chr>	<int>	<int>
2024-02-editcheck-reference-control	132639	109030
2024-02-editcheck-reference-test	133389	109821

• Confirmed that VEFU features associated with edit check engagement only appear for the test group
  • Edit checks have been shown for zhwiki,viwiki ptwiki, jawiki, itwiki, frwiki, eswiki and arwiki
  • No edit checks have been shown yet for afwiki, swwwiki, yowiki but this not unexpected as these are smaller wikis.

• Per the default configuration, edit check features and actions have only been logged for users with 100 edits or fewer in the AB test.

Note: This analysis will need to be completed within 90 days. Metrics for this analysis rely on revision_id and anonymous_user_token which are scrubbed after 90 days per data retention guidelines.