Page MenuHomePhabricator

Paypal Express checkout 1 hour test
Closed, ResolvedPublic


this is the container epic for the 1 hour test

Event Timeline

DStrine created this task.Apr 4 2016, 10:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 4 2016, 10:21 PM
Jgreen closed subtask Restricted Task as Resolved.Jun 3 2016, 6:07 PM

@Pcoombe fr-tech would like to schedule a 1 hour test on thursday. @Ejegg and @XenoRyet can help with details.

@DStrine @Ejb1123 @XenoRyet Can you confirm a few things:

  • How do I select the Express Checkout form?
  • Any preference where we test this? It would probably be easiest to do in Spain where we currently have tests up, or we could do US if you're after a lot of traffic
  • Do you want to test mobile as well, or just desktop?

@Pcoombe You can select the Express Checkout form with a link like this:

With the key bit being the 'title=Special:PaypalExpressGateway' and '&gateway=paypal_ec' bits.

I have no real preference on Spain versus US, so I'm cool with Spain if that's easier. I also don't know of any reason not to test mobile, so let's go ahead and do both.

Thanks @XenoRyet! I'm planning to run this in Spain tomorrow 1600-1700 UTC (9-10am Pacific), and sent a calendar event for that time.

Will let you know tomorrow when I have banners.

I made a test donation (in GBP) myself btw, and it went through 😄

Sounds like a plan.

We've got one little thing still left to fix with recurring, but it should go out today. I'll let you know if something goes horribly awry.

Actually the Spain traffic for mobile looks really low now. It's not much more work to run it in the US, so let's do that.

Closing the loop here:

This ran as planned from 1600-1700 UTC in the US. We processed over 1600 successful Express Checkout donations. Compared with the current flow: conversion looks 1-8% higher on desktop, unchanged on mobile, and significantly higher (although with very limited data) on iPad.

Full results

XenoRyet closed this task as Resolved.Apr 26 2017, 10:29 PM

Just to note we ran a second test of this today, from 16:05 to 17:05 UTC in the US. Results look very similar to last time.

  • increase in Paypal conversion somewhere between 0.1% and 7.7% on desktop (p-value 0.045)
  • no significant change on mobile
  • apparent larger increase on iPad but not enough data for significance

Full results