Page MenuHomePhabricator

Enable A/B test for all parts of the application
Open, HighPublic

Description

As a member of the fundraising team,
I want to check how different appearances of the donation page and its forms (donation, membership) affect the amount and frequency of donations,
so that I can reach the donation target.

Acceptance Criteria:

  • The Fundraising dev team can define "campaigns", i.e. alternate versions of templates that are available for a limited amount of time (similar to the mechanism already in place on the confirmation page).
  • When a donor with no cookies goes to spenden.wikimedia.de, he is sorted in a bucket for each of the currently running campaigns. The buckets are stored in a cookie.
  • When a donor with has a campaign parameter in the URL, the bucket for that campaign will be set according to the parameter (if the parameter is a valid bucket id). In the case of multiple campaigns, the other buckets are not touched or randomly chosen in case the user has no previous assignment.
  • When a donor with a bucket cookie goes to spenden.wikimedia.de, he is shown the correct template (if the bucket id is valid and the campaign is still running)
  • The alternate template name for donations is written into the layout property of DonationTrackingInfo
  • The configuration can be overwritten by a/b test definitions.

Background:

  • Instead of using DonationConfirmationPageSelector there needs to be a campaign-based skin selection, as defined in T172118.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 6 2017, 5:35 PM
This comment was removed by gabriel-wmde.
gabriel-wmde edited projects, added Epic; removed Story.
gabriel-wmde updated the task description. (Show Details)Apr 19 2017, 2:54 PM

Decide on structure for A/B test content - how are messages and templates stored inside the content repo: temporarily (only for the test, e.g. separate file) / permanently (inside the existing files; risk of forgetting it after end of test)

Questions for the fundraising team:

  • Do you need a/b testing with buckets or more of a "theming" where the page design is decided by a parameter from a banner (which does the bucketing)?
  • Do you need a "history" where you can display old themes?
  • Do you need to automatically expire "tests" ? There can only be either theme history OR campaigns.
  • Do you need multiple simultaneous tests on different parts of the page?

TODO rewrite the task description as a story with acceptance criteria (design only, no controller or JS changes).

  • yes, we need buckets and random sampling for people coming from wikimedia.de (to the membership form, but in general also thinkable to the donation form); for tests of the donation laning page we also need the testing possibility via banner links
  • no, we don't need a history
  • tests should have campaign character, hence start and end date
  • yes, we need multiple simultaneous tests; for instance a lading page test for users coming from donation banners and a randomly sampled confirmation page test after the donation
gabriel-wmde triaged this task as High priority.Jun 14 2017, 11:20 AM
gabriel-wmde updated the task description. (Show Details)
gabriel-wmde updated the task description. (Show Details)Jul 31 2017, 2:55 PM

@kai.nissen : I just stumbled upon our "hack" for enabling/disabling Sofortüberweisung with an URL parameter (https://github.com/wmde/FundraisingFrontend/blob/master/src/Presentation/SofortToggleServiceProvider.php), which is kind of an A/B in itself. I think we might need to adjust the acceptance criteria to not only encompass skin changes but also "custom feature toggles" where functionality can be turned on/off.

Or you could put the removal of what I consider technical debt (special handling of a specific URL parameter) into a separate ticket.

kai.nissen updated the task description. (Show Details)Apr 20 2018, 9:46 AM

Thank you, nice thinking! I added a sub-task to this.