Page MenuHomePhabricator

[Logged in reader retention baseline] Launch A/A experiment
Closed, ResolvedPublic3 Estimated Story Points

Description

Background

We want to run a monthlong cohort tracking through an A/A test in Test Kitchen (similar to one already performed for logged out users) to see the current retention rate of logged-in users. We will run this A/A tracking from April 1, 2026 to May 1, 2026

Instrumentation spec

User story

As the analyst, I want to be have access to a month's worth of data tracking each time a user account loads a page on wikipedia while logged in. This will give me the ability to analyze the retention rate over various timeframes, as well as focus on data from users who are in our target demographic (readers).


Requirements

Technical requirements

  • (See patch.) Create and deploy a new event stream that includes the proper contextual attributes. The web_base stream contains page IDs which we do not need.
  • (See patch.) Create a new loggedInReaderRetention instrument in WikimediaEvents. This will be similar to the loggedOutReaderRetention instrument created last year by the ReaderGrowth team.
    • Ensure that visits by temp accounts are not logged
  • (See test.) Create a new A/B test in the TestKitchen UI. See the instrumentation spec linked above for details. Yes, it's an A/A test, but it's configured as a new A/B test with a second control group.
  • Test locally to confirm the instrument is working
  • Turn on the instrument (at least 24 hours before the start date of 2026-04-01)
  • Confirm that data is being gathered as expected

Acceptance criteria

  • The A/A experiment instrument to track users per sample rate outlined in the spec is set up for the top 50 trafficked wikipedias
  • We begin tracking this cohort on April 2, 2026 and run for 1 month
  • Page-visited data per user ID is recorded (NOT page load history, only that any page is loaded)
  • Temp accounts are not included

BDD

  • For QA engineer to fill out.

Test Steps

  • For QA engineer to fill out.

Communication criteria

Add if this needs an announcement or discussion.

Rollback plan

The experiment can be turned off in the TestKitchen UI if there are any issues.

This task was created by Version 1.0.0 of the Reader Experience team task template using phabulous.

Event Timeline

HFan-WMF renamed this task from [Logged in user retention baseline] A/A Experiment setup in TK to [Logged in reader retention baseline] A/A Experiment setup in TK.Mar 23 2026, 7:32 AM
HFan-WMF updated the task description. (Show Details)

Does this include temp accounts? I see performer_is_temp in the spec as a contextual attributes.

For reference, there is already an A/A test for logged out reader retention, which can perhaps be used as guidance on how to implement this:

https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/WikimediaEvents/+/r[…]xt.wikimediaEvents/testKitchen/loggedOutReaderRetention.js

AnneT renamed this task from [Logged in reader retention baseline] A/A Experiment setup in TK to [Logged in reader retention baseline] Launch A/A experiment.Mar 23 2026, 7:35 PM
AnneT updated the task description. (Show Details)
AnneT updated the task description. (Show Details)
AnneT updated the task description. (Show Details)
AnneT set the point value for this task to 3.

Does this include temp accounts? I see performer_is_temp in the spec as a contextual attributes.

We want to analyze users with logged_in = true and performer_is_temp = false. It’s unclear whether temporary accounts will be bucketed into the experiment. If they are, we will filter them out in the analysis using the performer_is_temp field.

Change #1261546 had a related patch set uploaded (by Anne Tomasevich; author: Anne Tomasevich):

[mediawiki/extensions/WikimediaEvents@master] Add logged-in reader retention instrument

https://gerrit.wikimedia.org/r/1261546

With the patch in gerrit, I can see the events with my local test kitchen setup the same way as Wikimedia's test kitchen:

{
    "instrument_name": "LoggedInReaderRetention",
    "experiment": {
        "enrolled": "logged-in-retention-group1",
        "assigned": "control",
        "subject_id": "5e183e37d760ee0a8c8f5ae84273c94b03150b815bda9e9e3ab645bf22b20ea8",
        "sampling_unit": "mw-user",
        "coordinator": "default"
    },
    "$schema": "/analytics/product_metrics/web/base/2.0.0",
    "meta": {
        "domain": "enwiki.mediawiki.local.wmftest.net",
        "stream": "product_metrics.web_base",
        "id": "9511a6c8-9ada-4f55-a894-5d3d8ba676af",
        "dt": "2026-03-27T14:09:27.170Z",
        "request_id": "90ad2800-29e6-11f1-82ed-f55ba206b6ea"
    },
    "dt": "2026-03-27T14:09:24.875Z",
    "action": "page-visited",
    "agent": {
        "client_platform": "mediawiki_js",
        "client_platform_family": "desktop_browser"
    },
    "mediawiki": {
        "database": "enwiki",
        "skin": "vector-2022"
    },
    "page": {
        "content_language": "en",
        "id": 615,
        "namespace_id": 0,
        "title": "Democratic Republic of the Congo"
    },
    "performer": {
        "is_bot": false,
        "is_logged_in": true,
        "is_temp": false,
        "name": "Admin",
        "session_id": "477cae5ad6cf88f92c4d"
    },
    "http": {
        "request_headers": {
            "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36"
        }
    }
}

do we want to collect the page information for each page visit?

I think the page details come from using the product_metrics.web_base stream locally, but we would setup a new stream (mediawiki.product_metrics.reader_experiments) that has more narrow set of contextual attributes?

Next steps:

  • We need to setup a custom stream in mediawiki-config in ext-EventStreamConfig.php
  • add it to wgTestKitchenExperimentStreamNames in InitialiseSettings.php
  • also add it to wgEventLoggingStreamNames in ext-EventLogging.php

Ensure the that loggedInReaderRetention.js calls setStream. (can use the reading list a/b test as an example)

Change #1264630 had a related patch set uploaded (by Anne Tomasevich; author: Anne Tomasevich):

[operations/mediawiki-config@master] Add event stream for logged-in reader retention experiment

https://gerrit.wikimedia.org/r/1264630

AnneT subscribed.

Thanks @aude for your help in the above comments; I've updated the patches accordingly and am assigning this task to you for code review

AnneT updated the task description. (Show Details)

Change #1264630 merged by jenkins-bot:

[operations/mediawiki-config@master] Add event stream for logged-in reader retention experiment

https://gerrit.wikimedia.org/r/1264630

Mentioned in SAL (#wikimedia-operations) [2026-03-30T20:59:53Z] <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1264630|Add event stream for logged-in reader retention experiment (T420490)]]

Mentioned in SAL (#wikimedia-operations) [2026-03-30T21:02:02Z] <cjming@deploy1003> cjming, annet: Backport for [[gerrit:1264630|Add event stream for logged-in reader retention experiment (T420490)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-03-30T21:06:56Z] <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1264630|Add event stream for logged-in reader retention experiment (T420490)]] (duration: 07m 04s)

Confirming that the experiment is turned on in test kitchen with data collection beginning tomorrow as the related code rides the train to all wikis

Change #1261546 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] Add logged-in reader retention instrument

https://gerrit.wikimedia.org/r/1261546

Change #1267008 had a related patch set uploaded (by Anne Tomasevich; author: Anne Tomasevich):

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.22] Add logged-in reader retention instrument

https://gerrit.wikimedia.org/r/1267008

Change #1267008 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@wmf/1.46.0-wmf.22] Add logged-in reader retention instrument

https://gerrit.wikimedia.org/r/1267008

Mentioned in SAL (#wikimedia-operations) [2026-04-02T20:52:10Z] <kemayo@deploy1003> Started scap sync-world: Backport for [[gerrit:1267008|Add logged-in reader retention instrument (T420490)]]

Mentioned in SAL (#wikimedia-operations) [2026-04-02T20:53:51Z] <kemayo@deploy1003> annet, kemayo: Backport for [[gerrit:1267008|Add logged-in reader retention instrument (T420490)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2026-04-02T21:03:50Z] <kemayo@deploy1003> Finished scap sync-world: Backport for [[gerrit:1267008|Add logged-in reader retention instrument (T420490)]] (duration: 11m 40s)

LMora-WMF claimed this task.