Identify participating wikis for Edit Check (references) A/B test
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• ppelberg
	Aug 30 2023, 9:26 PM

Description

This task is about identifying the Wikipedias that will participate in the A/B test of the Edit Check (references) feature.

There are two parts to this task:

Identifying potential wikis for inclusion in the A/B test
Confirming with these wikis that they are comfortable participating in the A/B test

Selection requirements

This section will eventually contain the requirements the wikis participating in the A/B test will need to collectively meet.

Participating Wikis

Key:
Ready for deployment
No opposition

Wiki	Link to discussion	Feedback/Notes
ar.wp	They were informed of a by-default deployment by @Dyolf77_WMF, we change our strategy.
afwiki	discussion
eswiki	discussion
frwiki	discussion	The idea of using EC to tell users about quality of citations came up.
igwiki		Maybe not the best candidate, as 80% of new contents are translated.
itwiki	discussion	T356519: Clarify the call to action on Edit check (references)
jawiki	from T327707: Identify partner wikis for Edit Check project
ptwiki	discussion
swwiki	discussion
yowiki	discussion
viwiki	from T327707: Identify partner wikis for Edit Check project
zhwiki	discussion	The idea of using EC to tell users about quality of citations came up. Twice.

Per the "Edit Check Configuration" section below, we'll need to propose a set of Edit Check configuration values we'd need participating wikis to adopt. If/when we come to learn the policies/linguistic conventions at individual projects make setting the same configuration values infeasible, we'll need to name the implications on the analysis and decide if we're willing to accept the tradeoffs inherent with configuration values varying across participating wikis.

Edit Check Configuration

For the purposes of this A/B test, we're going to strive for the wikis that will participate to configure Edit Check in a consistent way. Doing so will enable us to draw wiki-agnostic conclusions about the impact(s) Edit Check causes.

Prompted by the question @MNeisler posed in T346837#9180760 and what we talked about offline on 20 Sep,

Related Objects
Search...

Status	Assigned	Task
Open	None	T265163 Create a system to encode best practices into editing experiences
Open	None	T331946 [RELEASE TICKET] Make Edit Check (references) available to all newcomers at all Wikipedias
Open	MNeisler	T342930 [MILESTONE] Run an A/B test to evaluate Edit Check (references) impact
Open	• ppelberg	T221185 Prepare visual editor to be a default mobile editor
Open	• ppelberg	T351778 [MILESTONE] Offer mobile VE as default for at Edit Check partner + test wikis
Resolved	Trizek-WMF	T345298 Identify participating wikis for Edit Check (references) A/B test
Resolved	MNeisler	T352129 Verify proposing A/B test wikis will provide sufficient edit volume to draw conclusions from

Event Timeline

• ppelberg created this task.Aug 30 2023, 9:26 PM

• ppelberg updated the task description. (Show Details)

• ppelberg added a subscriber: • Whatamidoing-WMF.

MNeisler mentioned this in T346837: Implement bucketing for Edit Check (references) A/B test.Sep 19 2023, 7:54 PM

• ppelberg updated the task description. (Show Details)Sep 20 2023, 9:54 PM

VPuffetMichel moved this task from To Triage to Triaged on the VisualEditor board.Sep 21 2023, 3:18 PM

• ppelberg mentioned this in T351778: [MILESTONE] Offer mobile VE as default for at Edit Check partner + test wikis.Nov 22 2023, 2:39 AM

• ppelberg mentioned this in T352122: [Config] Start Edit Check (references) A/B test.Nov 28 2023, 12:37 AM

• ppelberg added a parent task: T351778: [MILESTONE] Offer mobile VE as default for at Edit Check partner + test wikis.Nov 28 2023, 1:13 AM

• ppelberg mentioned this in T352126: Share mobile VE A/B test results and deployments plans with participating wikis.Nov 28 2023, 1:38 AM

• ppelberg mentioned this in T352127: [Config] Enable the mobile visual editor by default for the initial set of wikis.Nov 28 2023, 1:43 AM

• ppelberg mentioned this in T352129: Verify proposing A/B test wikis will provide sufficient edit volume to draw conclusions from.Nov 28 2023, 2:00 AM

Trizek-WMF subscribed.Dec 6 2023, 4:18 PM

Assigning to me to provide some data to help inform the selection of wikis for this task, including the following:

Average monthly newcomers at wikis
Review the proportion of users shown edit check at wikis where edit check is currently deployed to get a sense of how frequently newcomers are encountering this feature.

(edit conflict)

@Trizek-WMF

Here are some data and resources to help inform the selection of wikis

This Superset chart shows the average number of monthly editors with 100 edits or less. I specifically reviewed editors with 100 edits or less that made an edit to the main namespace to align with the default configuration for Edit Check.
Wikis with high number of average monthly new active editors can be found in the wiki comparison sheet.
Below are the number and proportion of users shown edit check at wikis where edit check is currently deployed as default (excludes unregistered users). Data reflects all events logged since 11 Oct 2023 when deployed at initial set of partner wikis. At these smaller wikis, we're seeing between 5 to 19% of users being shown edit check at least once.

Partner wiki	Number of distinct users shown edit check	Proportion of users shown edit check out of all that that made an edit attempt
dagwiki	6	12.2%
eewiki	1	5.3%
fatwiki	2	11.1%
fonwiki	3	10.3%
gpewiki	6	19.4%
gurwiki	5	16.7%
hawiki	16	5.4%
kgwiki	1	12.5%
lnwiki	3	17.6%

Number and proportion of distinct editing sessions where edit check is shown at partner wikis (includes unregistered users):

Partner wiki	Number of distinct editing sessions where edit check was shown	Proportion of editing sessions where edit check was shown
dagwiki	24	0.47%
eewiki	2	1.34%
fatwiki	5	2.50%
fonwiki	11	0.42%
gpewiki	11	0.99%
gurwiki	36	0.87%
hawiki	23	0.14%
kgwiki	1	0.79%
lnwiki	3	0.43%

Some selection criteria recommendations

Mix of small, medium, and at least a few large wikis to review a diverse set of communities. Per the practice used in past AB tests, we like to include small wikis to review a diverse set of communities but can exclude them from the per-wiki analysis if we don't get a representative sample.
At least some wikis with a high proportion of editors from SSA. Editors from Sub-Saharan Africa were identified as one of the primary target audiences for this project. frwiki, igwiki, ptwiki, arwiki, yowiki, swwiki, afwiki are some wikis with a decent number of distinct editors from that region per data on this supserset dashboard.
15 to 20 different wikis. Here's the list of wikis used in the New Topic tool AB test if helpful
Should not include partner wikis where it is currently enabled by default. https://www.mediawiki.org/wiki/Edit_check/Deployment_status.
Recommend as many wikis as possible that meet the following requirements below to ensure we get a large enough sample for review:
- Wikis that have 40 or more monthly new active editors per wiki comparison sheet.
- Wikis that have an average of at least 30 or more junior contributors per day (per Superset chart)
Wikis that are comfortable with the default edit check configurations. Note: Per previous team discussions, we don’t currently have a way to override the on-wiki config for the A/B test; however, it would be useful if the default settings were not radically changed in the AB test so we can be more confident on reporting the impact caused by Edit Check across all wikis.

MNeisler moved this task from Triage to Tracking on the Product-Analytics board.Dec 12 2023, 8:54 PM

Trizek-WMF updated the task description. (Show Details)Dec 18 2023, 6:10 PM

Restricted Application added a subscriber: Stang. · View Herald TranscriptDec 18 2023, 6:10 PM

I made a first pass of selection, based on the criteria. We have to refine it as most wikis listed are the "big" ones. However, smaller wikis from the geographical we are supposed to target don't pass the requirement listed.

I haven't added English Wikipedia. By it size, it would imbalance the data, and the possible requests and feedback from that community is often too specific to fit an A/B test perspective.

Thanks @Trizek-WMF The initial selections look great except I believe Swwiki already has edit check enabled by default according to this list: T327707. If so, it should be removed from the list of AB test wikis.

However, smaller wikis from the geographical we are supposed to target don't pass the requirement listed.

I'm comfortable including a few smaller to mid-size wikis as long the majority of selected wikis pass the requirements listed. As mentioned above, it would be useful to get data from a diverse set of communities but I can exclude them on any per wiki analysis if we don't end up getting enough data.

I confirmed the initial selection proposed will likely provide a sufficient sample size for the AB test analysis pending confirmation they are comfortable participating in the AB test.

Trizek-WMF mentioned this in T327707: Identify partner wikis for Edit Check project.Feb 1 2024, 3:24 PM

Trizek-WMF raised the priority of this task from Medium to High.Feb 1 2024, 4:35 PM

Trizek-WMF updated the task description. (Show Details)

Trizek-WMF updated the task description. (Show Details)Feb 2 2024, 4:48 PM

Trizek-WMF updated the task description. (Show Details)Feb 5 2024, 6:17 PM

Trizek-WMF updated the task description. (Show Details)Feb 6 2024, 4:30 PM

MNeisler closed subtask T352129: Verify proposing A/B test wikis will provide sufficient edit volume to draw conclusions from as Resolved.Feb 8 2024, 5:14 PM

Trizek-WMF updated the task description. (Show Details)Feb 9 2024, 9:44 AM

Trizek-WMF updated the task description. (Show Details)

Trizek-WMF updated the task description. (Show Details)Feb 9 2024, 10:48 AM

Trizek-WMF updated the task description. (Show Details)Feb 12 2024, 3:24 PM