We should now have enough data to do an initial check on the A/B test results, considering we started gathering "good data" on 14 July.
Generate graphs showing how the following metrics compare between the treatment and control test groups:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T255327 [Epic] Evaluate which editing interface should be shown by default | |||
| Open | None | T227338 Test visual editor as the default mobile editor on select wikis | |||
| Resolved | nshahquinn-wmf | T229426 Check on VE as default A/B test results |
Updating the task description to include the metrics we would like to check on. Those metrics are reflected in the task description's "Done" section.
During our 28 August team planning meeting, we discussed the initial results from the A/B test. Below is an unranked list of the questions that surfaced in that conversation.
For now, consider these questions as notes.
@Neil_P._Quinn_WMF @ppelberg is the check-in component done? Are the notes from discussion potential future tasks?
Megan and I have done the initial checks and found the following.
53.4% of the users (both registered and anonymous) who were bucketed ended up in wikitext default bucket. It turns out that it would be incredibly unlikely (p << 10^-15) to get an imbalance this big if our random assignment was actually 50%–50%. So there's clearly a serious issue somewhere that we need to understand.
| bucket | users |
|---|---|
| source default | 1,302,187 |
| visual default | 1,214,917 |
With that said, this is a preliminary look at our key metrics. The table shows the value of each metric for the average user in the bucket (because each user is independent of each other, but different attempts by the same user are not independent).
| bucket | attempts | completed edits | edit completion rate |
|---|---|---|---|
| source default | 1.399 | 0.040 | 0.87% |
| visual default | 1.403 | 0.037 | 0.79% |
In addition, among completed edits in each bucket, the average editing time was:
| bucket | editing time |
|---|---|
| source default | 2 min, 12 s |
| visual default | 2 min, 46 s |
@ppelberg, @MNeisler, and I discussed this yesterday and, yes, at this point we're calling the preliminary check done. Megan will dig into some of the issues above as part of T221198.
@kzimmerman, yep. Although, I've created a task that [hopefully] makes this more explicit: T232175.
Side note: I wonder whether using Phabricator as a drafting space (as I'm doing in T232175) is appropriate. So please, if you see a better way, I'm all ears (or I guess eyes in this context).
@MNeisler + @Neil_P._Quinn_WMF, thanks for pulling these results together. A question [1] related to "Editing interface switching"...
In the following scenario, how would the test make sense of this contributor's editing session? Would the test consider this contributor as having abandoned their edit in default-visual?
Scenario
i. Contributor taps edit
ii. Contributor is bucketed into default-visual
Visual editor loads (read: reaches ready)
iii. Contributor switches to wikitext
iv. Contributor makes some changes
v. Contributor publishes their changes
You're right, I don't think we've discussed this before. Since switching interface on mobile doesn't result in a page reload, it won't cause the creation of a new editing session ID. That means the scenario you described would be treated as a single completed attempt. Deciding which interface to credit for the completion is complex, but deciding which bucket is simple since switching interfaces shouldn't affect the bucket.
Understood, ok. Thanks, Neil.
As a mental note: in the instance described in T229426#5470251, we would count this contributor as having completed their edit in the test bucket they were initially assigned.