Page MenuHomePhabricator
Create Task
Maniphest T291306

Identify wikis for New Discussion Tool A/B test
Closed, ResolvedPublic
Actions

Description

This task is about identifying the Wikipedias that will participate in the A/B test of the New Discussion Tool.

There are two parts to this task:

  1. Identifying potential wikis for inclusion in the New Discussion Tool A/B test
  2. Confirming with these wikis that they are comfortable participating in the New Discussion Tool A/B test

Selection requirements

The "Candidate wikis" below, collectively, met the following requirements:

  1. There is a large enough number of people editing talk pages each month for the A/B test to reach "significance" in a relatively short amount of time
  2. Wikis whose editors are likely to come from a variety of different locations
  3. Wikis whose editors use a variety of languages and scripts to communicate
  4. Wikis whose monthly talk page editors are comprised of significant numbers of Junior and Senior Contributors
  5. The Reply Tool is already available as an opt-out feature

Candidate wikis

IDWikiCodeLink to on-wiki conversationStatus (e.g. participation confirmed)
1.Amharic WikipediaamwikimessageNo objections
2.Bengali WikipediabnwikimessageNo objections (discussion archived)
3.Chinese WikipediazhwikimessageNo objections (discussion archived)
4.Dutch WikipedianlwikimessageSupportive of test (discussion archived)
5.Egyptian WikipediaarzwikimessageNo objections (discussion archived)
6.French Wikipediafrwikidiscussion, reminderNo objections
7.Hebrew WikipediahewikimessageNo objections (discussion archived)
8.Hindi WikipediahiwikimessageNo objections
9.Indonesia WikipediaidwikimessageNo objections (discussion archived)
10.Italian WikipediaitwikimessageNo objections
11.Japanese WikipediajawikimessageNo objections
12.Korean WikipediakowikimessageNo objections
13.Oromo WikipediaomwikimessageNo objections
14.Persian WikipediafawikimessageNo objections
15.Polish WikipediaplwikimessageNo objections (discussion archived)
16.Portuguese WikipediaptwikimessageNo objections
17.Spanish WikipediaeswikimessageNo objections (discussion archived)
18.Thai WikipediathwikimessageNo objections
19.Ukrainian WikipediaukwikimessageNo objections (discussion archived)
20.Vietnamese WikipediaviwikimessageNo objections (discussion archived)

Note: you might notice this list is identical to the wikis that participated in the Reply Tool A/B test (T267379) with the following exception: the Afrikaans and Swahili Wikipedias are not included because the Reply Tool A/B test data [i] leads us to think there are not enough people using talk pages at these wikis to draw conclusions from the limited usage data. [ii]

Done

  • @MNeisler verifies the ===Candidate wikis will produce a sufficient amount and diversity of usage data for us to draw conclusions from
  • @Whatamidoing-WMF consults with the ===Candidate wikis and documents the wikis that are and are not open to participating in the A/B test in this ticket
  • @ppelberg updated T291306 with the list of wikis participating in the A/B test

i. See Reply Tool AB Test Report
ii. T290204#7356556

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
ResolvedDLynchT291308 Make config change to start New Discussion Tool A/B Test
Resolved• Whatamidoing-WMFT291306 Identify wikis for New Discussion Tool A/B test
· · ·

Event Timeline

ppelberg renamed this task from Identify wiki candidates for New Discussion Tool A/B test to Identify participating wikis for New Discussion Tool A/B test.Sep 17 2021, 10:56 PM
ppelberg created this task.
ppelberg renamed this task from Identify participating wikis for New Discussion Tool A/B test to Identify wikis for New Discussion Tool A/B test.Sep 17 2021, 11:23 PM
ppelberg updated the task description. (Show Details)
ppelberg added a subscriber: MNeisler.
Restricted Application added subscribers: Huji, Stang, Base, revi. · View Herald TranscriptSep 17 2021, 11:23 PM
ppelberg assigned this task to MNeisler.Sep 17 2021, 11:23 PM
ppelberg moved this task from Backlog to Community Engagement on the Editing-team (Tracking) board.
ppelberg reassigned this task from MNeisler to • Whatamidoing-WMF.Sep 17 2021, 11:26 PM
ppelberg moved this task from Backlog to Triaged on the DiscussionTools board.
ppelberg updated the task description. (Show Details)
ppelberg mentioned this in T290204: [SPIKE] Estimate New Discussion Tool A/B Test Feasibility.Sep 17 2021, 11:30 PM
ppelberg mentioned this in T277825: Run A/B test to evaluate impact of New Topic Tool.Sep 17 2021, 11:45 PM
ppelberg mentioned this in T291308: Make config change to start New Discussion Tool A/B Test.Sep 22 2021, 1:14 AM
• Whatamidoing-WMF added a comment.Oct 5 2021, 4:40 PM

The Reply tool is not already deployed to the French Wikipedia. We were delaying that because the wiki is so large. However, I think it is likely to be deployed there fairly soon (before the A/B test is likely to start).

Shizhao added a project: Chinese-Sites.Oct 8 2021, 8:13 AM
Shizhao moved this task from Backlog to Extensions/Skins on the Chinese-Sites board.Oct 8 2021, 8:16 AM
ppelberg added a comment.Oct 13 2021, 12:03 AM
In T291306#7402915, @Whatamidoing-WMF wrote:

The Reply tool is not already deployed to the French Wikipedia. We were delaying that because the wiki is so large. However, I think it is likely to be deployed there fairly soon (before the A/B test is likely to start).

I'm glad you noted this, @Whatamidoing-WMF; I've added this information to the task description.

ppelberg updated the task description. (Show Details)Oct 13 2021, 12:06 AM
• Whatamidoing-WMF updated the task description. (Show Details)Oct 14 2021, 3:52 PM
• Whatamidoing-WMF updated the task description. (Show Details)Oct 18 2021, 11:14 PM
• Whatamidoing-WMF updated the task description. (Show Details)Oct 21 2021, 6:36 PM
• Whatamidoing-WMF updated the task description. (Show Details)Oct 21 2021, 7:04 PM
• Whatamidoing-WMF updated the task description. (Show Details)
• Whatamidoing-WMF updated the task description. (Show Details)Oct 26 2021, 7:22 PM
• Whatamidoing-WMF added a comment.Oct 26 2021, 7:24 PM

@MNeisler:

Both ptwiki and fawiki have stopped IPs from editing (or maybe only in the mainspace?). I wonder whether that would complicate the analysis. Would you prefer that we exclude those wikis?

VPuffetMichel added a project: Goal.Nov 8 2021, 11:44 PM
ppelberg mentioned this in T290502: Identify partner wikis for offering automatic topic subscriptions as a beta feature.Nov 9 2021, 11:37 PM
ppelberg mentioned this in T290500: [Config Change] Deploy Automatic Topic Subscriptions as Beta Feature at Initial Wikis.Nov 9 2021, 11:40 PM
ppelberg updated the task description. (Show Details)
ppelberg mentioned this in T295424: [Release Ticket] Offer Automatic Topic Subscriptions as Beta Feature at Remaining Wikis.Nov 9 2021, 11:44 PM
ppelberg mentioned this in T295425: [Config Change] Deploy Automatic Topic Subscriptions as Beta Feature at Remaining Wikis.Nov 9 2021, 11:48 PM
ppelberg updated the task description. (Show Details)Nov 18 2021, 9:27 PM
• Whatamidoing-WMF updated the task description. (Show Details)Nov 24 2021, 1:34 AM
ppelberg added a parent task: T291308: Make config change to start New Discussion Tool A/B Test.Nov 25 2021, 12:53 AM
ppelberg removed a parent task: T277825: Run A/B test to evaluate impact of New Topic Tool.
• Whatamidoing-WMF updated the task description. (Show Details)Dec 3 2021, 8:14 PM
ppelberg updated the task description. (Show Details)Dec 4 2021, 1:03 AM
MNeisler added a comment.Dec 7 2021, 4:19 PM

@Whatamidoing-WMF

Both ptwiki and fawiki have stopped IPs from editing (or maybe only in the mainspace?). I wonder whether that would complicate the analysis. Would you prefer that we exclude those wikis?

Thanks for raising this!
I don't think this would complicate the analysis and am fine with continuing to include them in the AB test as planned. Current thinking:

  • While we would not have IP-edit data for those two wikis, we should have sufficient IP edit data from the other wikis for the logged-out user analysis.
  • In the analysis, I plan to review results on a per wiki basis and can account for any potential impacts from the individual wiki in the impact analysis model. As a result, I'm not concerned about the results of these two wikis skewing overall results.
  • The IP editing change on those wikis is not being introduced at the same time as the deployment of the New Discussion Tool AB test so it shouldn't be a confounding factor and we should still be able to isolate changes to the usage of the New Discussion Tool.
ppelberg added a comment.Dec 7 2021, 9:22 PM
In T291306#7553319, @MNeisler wrote:

@Whatamidoing-WMF

Both ptwiki and fawiki have stopped IPs from editing (or maybe only in the mainspace?). I wonder whether that would complicate the analysis. Would you prefer that we exclude those wikis?

Thanks for raising this!
I don't think this would complicate the analysis and am fine with continuing to include them in the AB test as planned. Current thinking...

This additional context is helpful – thank you, @MNeisler.

I've added a note about the considerations you raised above to the analysis ticket (T277825) so that we can be sure to remember them when reviewing the A/B test results. See: T277825#7554521.

ppelberg closed this task as Resolved.Dec 7 2021, 9:22 PM
Stang unsubscribed.Dec 7 2021, 9:25 PM
Shizhao moved this task from Extensions/Skins to Closed on the Chinese-Sites board.Dec 8 2021, 3:09 AM
ppelberg mentioned this in T297847: Ensure New Discussion Tool Hint Is Fully Translated.Dec 15 2021, 10:50 PM
• Whatamidoing-WMF updated the task description. (Show Details)Jan 27 2022, 9:04 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL