Page MenuHomePhabricator

Donor badge experiment: Experiment instrumentation in TestKitchen
Open, HighPublic3 Estimated Story Points

Description

Background

Configure and validate event logging in TestKitchen for the Donor badge experiment so we can measure the feature's impact and evaluate success criteria.
This covers registering the experiment, confirming bucket assignment works end-to-end, and verifying that exposure events reach the logging pipeline correctly for all three variants.

Spec events would include (not exhaustive at this time): tapping the donor button, multiple taps per user.

Measurement plan: https://docs.google.com/document/d/177uzZs5WyUNYyQcdD6zGQZ9pWBs0hh5HTXR0B6kbHDY/edit?tab=t.0
Instrumentation spec: https://docs.google.com/spreadsheets/d/1NYyinpkV3dswcamBJjLx0AW8P3KzefQyhD85VITIp7U/edit?gid=1635536298#gid=1635536298

A: ControlB: Plain messageC: Delightful experience
Article.png (384×765 px, 127 KB)
Article.png (384×765 px, 136 KB)
Article 1.png (384×765 px, 134 KB)
Article 2.png (384×765 px, 126 KB)

Goal

Experiment is registered, bucketing is verified end-to-end, and exposure events are confirmed in the pipeline before any variant-specific UI ships.

Requirements

  • Register experiment in TestKitchen with variant identifiers A, B, C (correct weights) and initial traffic split and targeting rules (logged-out, mobile web, post-donation only: donor cookie present).
  • Bucket assignment verified as stable and deterministic (persistent at the edge uniques level) across sessions.
  • Exposure event schema
  • Verify exposure events in the logging pipeline for all three variants using test donor sessions.

QA steps

  • Test kitchen will decide if the user is eligible for the experiment and will group the user into control, test A or test B experiment group. On page view we'll fire a page_view event even if they are not a donor
  • Donors can only be detected on the client via a call to require( 'ext.wikimediaCustomizations.donor' ).isDonor()
  • If the user has donated within the last 250 days then we'll fire an exposure event (This long availability is due to enabled anonymous editing on mobile a while ago)

image.png (1,348×710 px, 129 KB)

Depending on the group:

  • If the user is a donor in 'control' group A no change
  • If the user is a donor and in 'treatment-b-simple' group B they see the donor badge instantly and for subsequent page views and it has no delightful animation
  • If the user is a donor in 'treatment-c-delightful' group C they see the donor badge instantly and for subsequent page views and clicking it has delightful animation

Technical information

Experiment Variations (TestKitchen)

  • Control (control)
  • Treatment B: Simple (treatment-b-simple)
  • Treatment C: Delightful (treatment-c-delightful)

Useful JS for testing

$.cookie( 'centralnotice_hide_fundraising', '{"v":1,"created":1779379520,"reason":"donate"}', { path: '/' } );
mw.user.clientPrefs.set( 'minerva-badge', '1' );
mw.testKitchen.overrideExperimentGroup( 'donor-delight-badge', 'control' );
# or
mw.testKitchen.overrideExperimentGroup( 'donor-delight-badge', 'treatment-b-simple' );
# or
mw.testKitchen.overrideExperimentGroup( 'donor-delight-badge', 'treatment-c-delightful' );

Acceptance Criteria for done

Event Timeline

HFan-WMF triaged this task as High priority.May 7 2026, 5:49 PM
HFan-WMF moved this task from Incoming to Needs refinement on the Reader Experience Team board.

Change #1287492 had a related patch set uploaded (by VolkerE; author: VolkerE):

[mediawiki/extensions/WikimediaCustomizations@master] Donor Delight Badge: Add files and hook hander for donor badge experiment

https://gerrit.wikimedia.org/r/1287492

AnneT subscribed.

The setup in test kitchen needs review here. Logging exposure events has been moved to a new task.

AnneT removed AnneT as the assignee of this task.Wed, May 20, 9:23 PM
AnneT added subscribers: bwang, Jdrewniak.

There are a couple of things I'm not sure about:

The wikis don't seem to completely line up:

  • Instrumentation plan lists dawiki, dewiki (Austria), elwiki, mawiki, nowiki (bokmål), and zawiki
  • Test kitchen config lists Danish, Greek, Malay, Norwegian Bokmål, Norwegian Nynorsk, German, and Japanese wikipedias

There are 3 contextual attributes listed in the instrumentation plan that that aren't available in the test kitchen UI:

  • performer.active_browsing_session_token
  • dt
  • meta.dt

I'm not sure if those need to be added, or if they're automatically included (I know that's the case with some of the others, I'm just not familiar with these)

cc @bwang / @Jdrewniak to wrap up once the instrumentation plan is finalized

Change #1287492 merged by jenkins-bot:

[mediawiki/extensions/WikimediaCustomizations@master] Donor Delight Badge: Add files and hook handler for donor badge experiment

https://gerrit.wikimedia.org/r/1287492

These are automatically included with product_metrics.web_base:

  • agent_client_platform
  • agent_client_platform_family

Always included by TestKitchen:

  • dt
  • meta.dt

In the test kitchen UI, we need to add:

  • mediawiki_database
  • mediawiki_skin
  • page_content_language
  • page_namespace_id
  • performer_is_logged_in
  • performer_is_temp
  • performer_session_id (I think only available client-side, but that should work for this experiment)

the product_metrics.web_base schema also defines various fields like page_title but those are not included unless specified when setting up the experiment in test kitchen.

I added these attributes in test kitchen for the donor-delight-badge experiment.

Just "code review"ed this. Instrumentations spec didn't have a machine readable name for the experiment so I added what we have in the code base, which is donor-delight-badge.

I also removed Japanese Wikipedia and added Zulu Wikipedia in the TestKitchen setup, since that reflects the instrumentation spec. I also updated the language code used in the instrumentation spec for Malaysia, it listed mawiki, but the currect code is ms. I was referencing this: https://meta.wikimedia.org/wiki/List_of_Wikipedias.

Volker_E updated the task description. (Show Details)
Volker_E updated the task description. (Show Details)
HFan-WMF subscribed.

bringing this into Sprint 22 because we need to finalize the list of countries in the Test Kitchen config and update the instrumentation spec based on the updated fundraising calendar.

Changes to the target wikis in the spec:

  • Remove mawiki and zuwiki (FR campaigns in Malaysia and South Africa will be in English)
  • Add jawiki (for active Japanese campaign)

After this change, let's update the Test Kitchen config accordingly.

I updated the instrumentation spec to remove mawiki and zuwiki, and added jawiki and enwiki. Thank you Volker for updating the Test Kitchen configuration!