Page MenuHomePhabricator

Data QA for the Donor Badge Delight Experiment
Closed, ResolvedPublic1 Estimated Story Points

Description

Background

We are launching a small experiment showing some donors two versions of a thank you badge on mobile web after their recent donation. (https://phabricator.wikimedia.org/T424069) -- after enabling the experiment on June 18, we need to verify that data is flowing in correctly from the experiment instrumentation (https://phabricator.wikimedia.org/T428800).

Measurement spec

User story

As the REx team, we want to see data flowing in properly for the donor delight badge experiment.


Requirements

Acceptance criteria

  • The contextual attributes outlined in the instrumentation spec are tracked
  • The events to be logged in the spec are tracked

Test Steps

{T429383#12035864}

Communication criteria

Add if this needs an announcement or discussion.

Rollback plan

Describe the rollback plan in production for this task if something goes wrong.

This task was created by Version 1.0.0 of the Reader Experience team task template using phabulous.

Event Timeline

Jdlrobson-WMF set the point value for this task to 1.

I'd also recommend confirming the control and treatment experiences on all target wikis, and checking for translations (although I'm not sure what we could do to speed up translations if they're missing - maybe we just need to get messages into the queue earlier for future experiments)

I did some basic checks [1]

Screenshot 2026-06-18 at 3.58.20 PM.png (619×395 px, 37 KB)

  • We are seeing roughly sized hits for page_visit from the different groups (assigned)
  • We are seeing a small amount of hide_badge events in the treatment groups and none in the control group
  • We are seeing a small amount of heart_badge events showing minimal clicks but evidence that people are clicking it
  • I wasn't expecting exposure events for the control but it seems like the code should be firing exposure events for all users in control. Doesn't seem like a big deal.
  • The exposure events seemed low. These do not fire if the badge was disabled via hide. These should be equivalent to page_visit - hide_badge. So either the 3 hide_badge events account for half of the page_visit events.... when I look at distinct subject IDs [2] all was good in the world.

Screenshot 2026-06-18 at 4.19.47 PM.png (602×440 px, 41 KB)

I can also confirm we are seeing events from both dewiki and enwiki [3]

Screenshot 2026-06-18 at 3.59.57 PM.png (197×143 px, 5 KB)

So this looks like an experiment that is working as expected!

[1]

SELECT
 action, action_context, experiment.assigned, count(*)
FROM
  wmf_experiments.experiment_event_v1
WHERE timestamp >= from_iso8601_timestamp('2026-06-18T14:30:00')
  AND experiment.enrolled = 'donor-delight-badge'
  group by action, action_context, experiment.assigned

[2]

SELECT
 action, action_context, experiment.assigned, count(distinct experiment.subject_id)
FROM
  wmf_experiments.experiment_event_v1
WHERE timestamp >= from_iso8601_timestamp('2026-06-18T14:30:00')
  AND experiment.enrolled = 'donor-delight-badge'
  group by action, action_context, experiment.assigned

[3]

SELECT
 mediawiki.database, count(*)
FROM
  wmf_experiments.experiment_event_v1
WHERE timestamp >= from_iso8601_timestamp('2026-06-18T14:30:00')
  AND experiment.enrolled = 'donor-delight-badge'
  group by mediawiki.database
Jdlrobson-WMF updated the task description. (Show Details)
Jdlrobson-WMF updated the task description. (Show Details)
Jdlrobson-WMF subscribed.

Data is now flowing into the automated Growthbook dashboard, signing off!