Page MenuHomePhabricator

Create Measurement Plan and Instrument Donor Experience for Apps
Closed, ResolvedPublic



  • Review designs and project goals and ensure validators are satisfactory or recommend improvements
  • Submit schema details to Legal for approval
  • Create or update schema based on designs
  • Instrument feature based on schema doc from Shay
  • Create data validation task

Indicator: 4.0% average CTR across both apps per unique user
Indicator: 20% increase in banner impressions
Indicator: 20% increase in donor acquisition
Guardrail: Abandonment rate is no more than 25%
Guardrail: No further decline in average donation YoY
Guardrail: Reports of a negative experience related to donations do not exceed 25% of user feedback during campaign period

Schema will track Donor Experience feature engagement and banner/promo impression, CTR and donor engagement data for both Android and iOS. Recurring donor, donor retention and donation financial data will be tracked by Fundraising and they will give us access to their outcome data through Superset at end of campaign. Note: Previously the Fundraising team used our internal Feed impression data to calculate impressions/CTR - we will replicate that data in this schema so they are able to maintain access to that data.

Preliminary Docs:
Apps Donor Experience Data Instrumentation Planning deck
Apps Donor Experience Instrumentation Process and Spec

iOS: Figma files
Android: Figma

Donor Experience Deck

Testing notes for iOS

This can be tested starting in Wikipedia TestFlight 7.4.3 (2811) build.

Testing tips:

All new donor experience work (in-app web view, new campaign modals with maybe later, multilingual campaign support, Apple Pay native donation) are set up to become available in the app only once the Netherlands or Italy Oct/Nov 2023 campaigns become valid. To test native Apple Pay, your device needs to be set up like this:

  1. Device iOS Settings > Language & Region > Region needs to be set to either the Netherlands or Italy
  2. Device date & time needs to be set on or after the campaign start date (change device date to 10/17)
  3. After fresh install, turn on logging in onboarding. After landing on the Explore feed you must background and foreground the app, then pull to refresh on the Explore feed to trigger a force refresh of the campaign data.
  4. To be provided the option to donate via Apple Pay (native donate form), you must have a valid card set up in Apple Wallet.
  5. IMPORTANT: Production charges are rate-limited to 4 transactions per 4 hours.

Testing notes on Android

From @cooltey notes on main task :
We have introduced some new developer preferences for the new donor experience for testing purposes.

Screenshot_20231004-155703_Wikipedia Dev.jpg (2×1 px, 80 KB)

Use the test.wikipedia debug URL

If you are going to test with a specific start date or end date, you can access the following page to edit the JSON file if you have the administrator permission. This will help you to test the "Maybe later" button visibility.


This helps you to override the web page link when you tap on "Donate now" in the dialog, and will load the page you specified in the CustomTab.
In general, you will only need to test the "thankyou" page behaviors (click on the link and see how it goes) since it should be opened in a native WebView instead of an external browser.


This relates to the visibility of the "Maybe later" button. The number is the date time in milliseconds. You can use this tool to get the desired date time in milliseconds.

Special note:

We will load the donation page in a CustomTab, which is the one with the URL shown on the toolbar.

Screenshot_20231002-163447_Chrome.jpg (2×1 px, 93 KB)

and we will load the "thankyou" page in a native WebView after the payment is submitted. The thankyou page will only be loaded if the campaign is from the Android app. (with utm_campaign=Android)

Screenshot_20231003-152110_Wikipedia.jpg (2×1 px, 185 KB)

Event Timeline

Seddon triaged this task as High priority.Sep 12 2023, 4:49 PM
Seddon moved this task from Needs Triage to Product Backlog on the Wikipedia-iOS-App-Backlog board.

@JTannerWMF For Guardrail metric 'Reports of a negative experience related to donations do not exceed 25% of user feedback during campaign period' - I do not see a specific problem reporting mechanism in the design docs - however there is a link from the Apple Pay initial screen that goes to an explainer page for payment problems ( From there users are instructed to report issues by emailing As long as we are able to access any messages relevant to app issues* we can use incident reports from that system for this metric - just want to note it's not a report coming directly from within the app payment flow.

That same link is available in Web Pay initiation pages for both iOS and Android.

*I will follow up with FR Tech to make sure reporting data is made visible for us.

Shared Android/iOS schema app_donor_experience is ready to be created in MEP, please create an engineering task and I will create Data Validation tasks. Final version of instrumentation docs are pending some design questions waiting for response that won't affect schema structure.

Apps Donor Experience Data Instrumentation Planning deck
Apps Donor Experience Instrumentation Process and Spec

Request submitted to Legal for approval and data retention extension to 160 days. Asana Task

Tsevener assigned this task to SNowick_WMF.
Tsevener removed SNowick_WMF as the assignee of this task.
Tsevener removed Tsevener as the assignee of this task.

Adding a note that I have requested @Tsevener add value campaign_id to action_data for the active_interface = article_banner , action = impression events. I missed specifying this in the docs. Verifying we are seeing this data as of 2023-10-18.