Some of Fundraising spoke with Darwish Gani and Aaron Landman from Optimizely today, and learned that they offer an enterprise mode which might be legal and appropriate for MediaWiki UI experimentation. Their product is a visual editor that generates jQuery to manipulate the interface (or you can use custom js), then delivers variations to viewers and tracks the results.
This task does not belong in Design Research's backlog. However, there is potentially valuable information in the task for you and other Product Analysts to implement A/B testing.
Could you please help assign a more appropriate project to this task?
@ggellerman, thanks for pulling me in! I find this particularly interesting because I've actually used Optimizely in a past job. One of its major strengths is its visual editor, which allows you to do quick-and-dirty tests of interface changes without the manual implementation that Facebook PlanOUT seems to require. Obviously, I'd prefer to use open source software, but this may be one of the cases where proprietary is the lesser of two evils. I'll put this project on my list.
@Neil_P._Quinn_WMF: Can I pass along some meeting minutes? Some Fundraising tech and analytics people went over to their office (a block away!) and chatted about quite a bit. I should probably just publish the notes here, I don't think there's anything private in there. The punchline, however, is that they can offer us an integration in which we control our own traffic and data.
Sorry, I managed to locate the pen and paper meeting minutes last week, but won't be able to enter the contents here until the week of July 22nd, cos I'm currently out of town.
Whew, finally reunited with my scraps of paper.
Notes from our 2014-12-5 meeting with Optimizely, sorry enough time has passed that I only partly understand what I've written here :-/
- Configurable rules to determine whether a person becomes part of a test (audience).
- Each test has a goal: clicks, pageviews, or a custom event. Goals can be split by segment.
- Campaign is an external URL parameter.
- They have an enterprise mode where we can host the data gathering snippet and (I believe) the data collection backend.
- Optimizely staff have published papers we should read, discussing ways to watch the result trends and make a conclusion before hitting rigid significance thresholds. For example, (client name redacted) shifts their p-value over the course of an experiment.
- Custom js can be used to isolate experiment and variations, and do scheduling. start and stop time, or repeated time of day experiments.
- Experiment API does all the UI magic.
- Experiments can be managed via code (was beta quality at the time): http://developers.optimizely.com/rest/introduction/index.html
- We can manage experiments to surface variations which are doing well.
Great question. Adding our project to tag this for further investigation. @MeganHernandez_WMF has been in touch with Optimizely, and a group of Fundraising Tech devs plus @ellery were present at the meeting those notes are from. We're not actively evaluating it, but I'm certainly interested in either pursuing their SaaS or borrowing some of their better ideas.
Also CC'ing @AndyRussG here, he's been doing some thinking about how to isolate the A/B testing component from CentralNotice into a reusable library.