vi experiments
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	HFan-WMF
	Oct 27 2025, 6:33 PM

Description

Background

Because we are configuring two separate experiments for English Wiki and the Arabic/French/Chinese/Indonesian/Vietnamese within Test Kitchen (fka xLabs), we should make sure that the code point to both experiments.

Acceptance criteria

All code that points to an experiment within Test Kitchen/xLabs points to en OR ar/fr/zh/id/vi experiment

Requirement

Scope: Desktop web (Vector 2022).

Update experiment configuration for Reading Lists to support both experiment IDs:
- we-3-3-4-reading-list-test1 (for Arabic, French, Chinese, Indonesian, Vietnamese wikis)
- we-3-3-4-reading-list-test1-en (for English Wikipedia)
All references to the experiment in Test Kitchen/xLabs must recognize either experiment ID to enable the Reading List feature.
Users bucketed into either experiment must be able to activate and use Reading Lists.
QA should confirm that enabling either experiment ID activates the same Reading List behavior.

BDD

Feature: Support multiple experiment IDs for Reading List feature

  Scenario: User enrolled in English experiment
    Given I am enrolled in the `we-3-3-4-reading-list-test1-en` experiment
    When I visit an article using Vector 2022
    Then the Reading List feature (bookmark icon and Special:ReadingLists) is enabled

  Scenario: User enrolled in AR/FR/ZH/ID/VI experiment
    Given I am enrolled in the `we-3-3-4-reading-list-test1` experiment
    When I visit an article using Vector 2022
    Then the Reading List feature (bookmark icon and Special:ReadingLists) is enabled

Test Steps

Test Case 1: Verify Reading List activation for English experiment

Log in and use the browser console to run: mw.xLab.overrideExperimentGroup('we-3-3-4-reading-list-test1-en', 'treatment')
Refresh the page.
AC1: The Reading List bookmark icon appears.
AC2: Special:ReadingLists is accessible.

Test Case 2: Verify Reading List activation for AR/FR/ZH/ID/VI experiment

Log in and use the browser console to run: mw.xLab.overrideExperimentGroup('we-3-3-4-reading-list-test1', 'treatment')
Refresh the page.
AC3: The Reading List bookmark icon appears.
AC4: Special:ReadingLists is accessible.

Test Case 3: Verify feature remains off for non-enrolled users

Log in as a user not in either experiment group.
Refresh the page.
AC5: The Reading List bookmark icon does not appear.
AC6: Special:ReadingLists is not accessible from the UI.

Rollback plan

Describe the rollback plan in production for this task if something goes wrong.

This task was created by Version 1.0.0 of the Reader Experience team task template using phabulous.

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Check for enwiki experiment name for ReadingLists A/B test	mediawiki/extensions/ReadingLists	master	+8 -5
	Add experiment name for enwiki in readingListAB.js	mediawiki/extensions/WikimediaEvents	master	+4 -2

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	Jdrewniak	T397532 [Hypothesis] FY2025-26 WE 3.3.4 Reading Lists on web
Open	None	T400353 [EPIC] WE 3.3.4 Reading List Experiment infrastructure
Resolved	SToyofuku-WMF	T408438 Reading List Experiment - update code to point to both english and ar/fr/zh/id/vi experiments

Event Timeline

HFan-WMF created this task.Oct 27 2025, 6:33 PM

Restricted Application added subscribers: hubaishan, Gerges, Aklapper. · View Herald TranscriptOct 27 2025, 6:33 PM

HFan-WMF added a parent task: T400353: [EPIC] WE 3.3.4 Reading List Experiment infrastructure.Oct 27 2025, 6:34 PM

Restricted Application added a subscriber: alaa. · View Herald TranscriptOct 27 2025, 6:34 PM

HFan-WMF moved this task from Incoming to Phase 1 (Desktop reading list A/B experiment) on the FY25-26 WE3.3.4 Reading List Experiment & Rollout board.Oct 27 2025, 6:47 PM

HFan-WMF moved this task from Incoming to Needs refinement on the Reader Experience Team board.

I would propose the names be we-3-3-4-reading-list-test1 and we-3-3-4-reading-list-test1-en, but let's give a (short) period of time for eng to bikeshed on names

HFan-WMF triaged this task as High priority.Oct 27 2025, 8:51 PM

Note: We refer to the experiment name in WikimediaEvents and in ReadingLists::HookHandler so would need to update in both places.

SToyofuku-WMF set the point value for this task to 2.Oct 29 2025, 5:37 PM

SToyofuku-WMF moved this task from Needs refinement to Ready for sprint on the Reader Experience Team board.

Shizhao added a project: Chinese-Sites.Oct 31 2025, 3:38 AM

Shizhao moved this task from Backlog to Site configuration on the Chinese-Sites board.

Suggest we add a configuration for the experiment name in mediawiki-config. This way we can more easily vary the experiment name by wiki (enwiki + the others), and have the experiment name defined in one place, to be used by the WikimediaEvents and ReadingLists extensions.

This would require a backport for the config (and later cleanup) but I think this would be the less hacky approach.

Jdrewniak moved this task from Ready for sprint to REx Sprint 9 [Q2 Nov 4 - Nov 17] on the Reader Experience Team board.Nov 4 2025, 6:35 PM

Jdrewniak edited projects, added Reader Experience Team (REx Sprint 9 [Q2 Nov 4 - Nov 17]); removed Reader Experience Team.

aude claimed this task.Nov 4 2025, 6:55 PM

aude moved this task from Ready to In progress on the Reader Experience Team (REx Sprint 9 [Q2 Nov 4 - Nov 17]) board.

Change #1201782 had a related patch set uploaded (by Aude; author: Aude):

[mediawiki/extensions/ReadingLists@master] Check for enwiki experiment name for ReadingLists A/B test

https://gerrit.wikimedia.org/r/1201782

Change #1201783 had a related patch set uploaded (by Aude; author: Aude):

[mediawiki/extensions/WikimediaEvents@master] Add experiment name for enwiki in readingListAB.js

https://gerrit.wikimedia.org/r/1201783

After looking at the possibility of adding a config, it would have to defined in both extensions and in mediawiki-config (with a backport). Perhaps adding a shared config is a bit too much complexity and we can just add a check for 'enwiki' wikiId / wgDBname and then vary the experiment name based on that.

aude removed aude as the assignee of this task.Nov 4 2025, 8:24 PM

aude moved this task from In progress to Needs code review on the Reader Experience Team (REx Sprint 9 [Q2 Nov 4 - Nov 17]) board.

Change #1201783 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Add experiment name for enwiki in readingListAB.js

https://gerrit.wikimedia.org/r/1201783

Change #1201782 merged by jenkins-bot:

[mediawiki/extensions/ReadingLists@master] Check for enwiki experiment name for ReadingLists A/B test

https://gerrit.wikimedia.org/r/1201782

Maintenance_bot removed a project: Patch-For-Review.Nov 4 2025, 9:31 PM

ReleaseTaggerBot added a project: MW-1.46-notes (1.46.0-wmf.2; 2025-11-12).Nov 4 2025, 10:00 PM

aude moved this task from Needs code review to Sign-off on the Reader Experience Team (REx Sprint 9 [Q2 Nov 4 - Nov 17]) board.Nov 5 2025, 2:55 PM

I think this can't really be QA'd so moved this to sign off

It can be QAed!! We should now be able to turn on the reading list feature using either we-3-3-4-reading-list-test1 or we-3-3-4-reading-list-test1-en where previously we could not

(sorry for lurking and sorry to Edward for giving you more work 😭)

HFan-WMF mentioned this in T408619: Placeholder ticket to double check deploys/backport.Nov 6 2025, 7:12 PM

Edtadros mentioned this in T409650: Verify 1.46.0-wmf.2 deployment.Sun, Nov 9, 7:08 AM

From meeting: since we didn't add ourselves to the preference on beta, this will need to be tested on testwiki any time starting tomorrow

Test Result - Prod

Status: ❌ FAIL
Environment: testwiki
OS: macOS Tahoe 26.x
Browser: Chrome Canary (latest as of test date)
Device: MS
Emulated Device: NA

Test Case 1: Verify Reading List activation for English experiment

Log in and use the browser console to run: mw.xLab.overrideExperimentGroup('we-3-3-4-reading-list-test1-en', 'treatment')
Refresh the page.
❌ AC1: The Reading List bookmark icon appears.

✅ AC2: Special:ReadingLists is accessible.

Test Case 2: Verify Reading List activation for AR/FR/ZH/ID/VI experiment

Log in and use the browser console to run: mw.xLab.overrideExperimentGroup('we-3-3-4-reading-list-test1', 'treatment')
Refresh the page.
✅ AC3: The Reading List bookmark icon appears.

✅ AC4: Special:ReadingLists is accessible.

Test Case 3: Verify feature remains off for non-enrolled users

Log in as a user not in either experiment group.
Refresh the page.
✅ AC5: The Reading List bookmark icon does not appear.
✅ AC6: Special:ReadingLists is not accessible from the UI.

bringing this to your attention

Edtadros updated the task description. (Show Details)Wed, Nov 12, 2:12 AM

Edtadros updated the task description. (Show Details)

Discussed in DMs and slack - I mistakenly thought this could be tested as such, but it will only work on english wikipedia (and potentially beta, but we don't have the hidden preference set)

On English (post train rollout tomorrow), only we-3-3-4-reading-list-test1-en should work and we-3-3-4-reading-list-test1 should cease to work
On all non-English wikis, only we-3-3-4-reading-list-test1 should work and we-3-3-4-reading-list-test1-en should not work

I will QA this since Edward is out and it's past Thursday now

Sadly, turns out I can't QA on english wikipedia either since I don't have the preference set 🙃 to feel like I did something I reviewed the code and it looks good to me

@HFan-WMF @Jdrewniak I think we have one of two options - sign this ticket off and then confirm the code path is working correctly on Wednesday when we see english users in the english experiment, or keep this ticket open and in QA until we run the script, then add ourselves to the preference on enwiki and QA then. I have no real preference, so wanted to hear from a product perspective

I don't think there's any harm in waiting until Wed to sign off (the sprint board has to remain unarchived for two more days after the sprint wraps, haha), so let's hold off and sign off next Wed then!

Thanks for digging into this, Steph!

@aude graciously added me to the preference on enwiki, and I was able to confirm that this is working as intended - I was able to see the experiment using we-3-3-4-reading-list-test1-en, and unable to see it using we-3-3-4-reading-list-test1. Signing this off

Stang moved this task from Site configuration to Closed on the Chinese-Sites board.Mon, Nov 17, 3:31 PM

	F70121167: screenshot 18.png
	Wed, Nov 12, 1:04 AM

	F70121125: screenshot 16.png
	Wed, Nov 12, 1:04 AM

	F70121127: screenshot 17.png
	Wed, Nov 12, 1:04 AM

Reading List Experiment - update code to point to both english and ar/fr/zh/id/vi experimentsClosed, ResolvedPublic2 Estimated Story PointsActions