Page MenuHomePhabricator

[S] Donation reminder (user story): A/B Test
Closed, ResolvedPublic3 Estimated Story Points

Description

User story

As a product manager, I want to set-up the donation reminders as an A/B test against the current "maybe later" experience, so I can understand the impact on average donor value over time.

Background

The Android team ran a 30-day experiment of the customizable donation in Italy, from August 21 - Sept 20, 2025 (T399601) to learn if behavior-based fundraising requests are a good fundraising mechanism in the mobile apps. We are running a second experiment with the feature. We plan to adapt it to run alongside a banner campaign, as an alternate follow-up action to "Maybe later".

Requirements
  • Donation reminder should be run as an A/B test against the current "Maybe later" experience
    • ✅ Logged-in and logged-out users users into control and variant when they click "Maybe later" on a banner
    • ✅ A users see a subsequent banner impression (control experience) every time they click "Maybe later"
    • ✅ B users receive donation reminder flow
      • Users in the experiment group should continue to see reminders for 3 months after the start of the experiment
      • ✅ Users in the experiment group B should not see any further banner impressions after clicking “Maybe later”
      • [Added Nov 10] Users in group B should only have access to the donation reminder feature for 3 months after release (add end date of March 15).
      • [Added Nov 10] Stop sorting users into control & variant after December 31st (We do not want this test to run in future banner campaigns by default)
  • ✅ Remove the survey from V1
  • ✅ Limit the audience to users with a device country of Great Britain, Australia, or Canada
  • In the edge case that someone sets up a reminder and then switches countries, match the behavior for maybe later banners logic
    • Scenario 1: Someone clicks "Maybe later" on a banner from within a country that's in the experiment and sets up a reminder. Then they change countries during the experiment. The new country is a country that is NOT in the reminder experiment, but it's somewhere we accept donations (I.e. moving from AU to IN during Big English)
      • ✅ Expected behavior: Reminder does not show
    • Scenario 2: Someone clicks "Maybe later" on a banner from within a country that's in the experiment and sets up a reminder. Then they change countries during the experiment. The new country is another country that IS ALSO within in the experiment (I.e. moving from AU to GB)
      • ✅ Expected behavior: Reminder still shows, but with updated currency
Engineering note

This will be an engineering task.

Event Timeline

HNordeenWMF triaged this task as Medium priority.
HNordeenWMF updated the task description. (Show Details)
HNordeenWMF moved this task from Needs Triage to Up Next on the Wikipedia-Android-App-Backlog board.

Initial estimation: M

Question:
The v1 donation reminder was targeted only to logged-in users. For v2, are we going to target both logged-in and non-logged-in users?

Yes - logged-in and logged-out, since that matches our current banner behavior

cooltey renamed this task from Donation reminder (user story): A/B Test to [M] Donation reminder (user story): A/B Test.Nov 3 2025, 5:33 PM
JTannerWMF set the point value for this task to 5.Nov 4 2025, 5:45 PM
JTannerWMF renamed this task from [M] Donation reminder (user story): A/B Test to [S] Donation reminder (user story): A/B Test.Nov 4 2025, 5:56 PM
JTannerWMF changed the point value for this task from 5 to 3.
cooltey updated the task description. (Show Details)
HNordeenWMF updated the task description. (Show Details)

@cooltey two clarifying requirements added. The second one is flexible, as long as we can make sure this test is only happening for the 2025 English campaign.

  • [Added Nov 10] Users in group B should only have access to the donation reminder feature for 3 months after release (add end date of March 15).
  • [Added Nov 10] Stop sorting users into control & variant after December 31st (We do not want this test to run in future banner campaigns by default)
ABorbaWMF subscribed.

Looks good on 2.7.50557-r-2025-11-14, on OnePlus 8 on Android 13, and Pixel 6 on Android 16. Tested in region and out

Hi @cooltey, I'm seeing something and want to check if this is an issue, or just related to me using the "Ignore date" feature. The banner is still appearing to show again 1 day later, even when I'm in the test group.

Steps to reproduce

  • Fresh install
  • Device date setting: automatic
  • Turn on developer settings
    • “Ignore date” toggle
    • donation reminder test group =1
    • GeoIP = AU
  • Open English article, banner appears
  • “Maybe later”
  • Setup reminder for 2 articles, $2 AUD
  • Read 2 articles for 15 seconds each
  • See reminder, ignore it
  • Navigate to new page
  • Change date to be one calendar day later

What happens

  • Banner appears on top of article view with donation reminder

What should happen

  • Banner does not appear because I was in the test group

I was trying to test the change of location requirement, and still saw another banner

Steps to reproduce

  • Fresh install alpha 11-24
  • Device date setting: automatic
  • Turn on developer settings
  • “Ignore date” toggle
  • donation reminder test group =1
  • GeoIP = AU
  • Open English article, banner appears
  • “Maybe later”
  • Setup reminder for 2 articles, $2 AUD
  • Read 2 articles for 5 seconds each
  • See reminder, ignore it
  • Read 2 articles for 5 seconds each
  • Change date to be one calendar day later
  • Change GeoIP = GB
  • Force close app and reopen
  • Navigate to English language page

What happens

  • Banner appears on top of article with donation reminder (in GBP as expected!)

Expected

  • Donation reminder appears with updated currency
  • Banner does not appear

I was trying to test the change of location requirement, and still saw another banner

...I believe this is expected. The banner for AU has a different campaign ID than the one for GB. They are different campaigns, as defined in the campaign configuration.

Fair, it's an edge case so that's fine. I will try to test - theoretically what should happen if they set-up a reminder again from the new banner? Does the new set-up just override the old one?

Yep - it overrides the old one with the new currency one just set-up. That is acceptable