This task is about making sure the patch [1] that's intended to address the issue discovered in T232237#5501940 is working as we expect it to.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T255327 [Epic] Evaluate which editing interface should be shown by default | |||
| Open | None | T227338 Test visual editor as the default mobile editor on select wikis | |||
| Resolved | ppelberg | T235101 Rerun mobile VE as default A/B test | |||
| Resolved | Mayakp.wiki | T235104 QA for EventLogging patch |
@Mayakp.wiki, this patch (see below) should be ready for QA as soon as tomorrow.
If there are additional details that would be helpful for you to have, please let me know.
cc @MNeisler
Checked the distribution using Neil/Megan's initial code, for data starting from 10/10 (date when fix was applied). The buckets seem to be balancing now. The data is being collected 50-50 in both buckets.
| bucket | attempts | attempts% | users | users % |
| default-source | 176,263 | 49.96% | 133,092 | 50.17% |
| default-visual | 176,540 | 50.04% | 132,212 | 49.83% |
Excellent – thank you, @Mayakp.wiki. A couple resulting questions...
...the questions above are borrowed from the task description.
Megan has recently shared the list of fields that will be used to answer the research questions (T221195 and T232175#5545364 ) with me. I will be checking the data in those fields this week to answer Q2 and Q3.
In light of T235949, we'll need to come up with a new way to bucket contributors in the A/B test (T236337). We'll then also need to test to make sure this new bucketing method is working properly (T236338).
@DLynch, a question for you: as part of T236338, do you think it will be necessary to re-test questions "2." and "3." below?
...@MNeisler and I assume a change to how contributors are bucketed would not affect the instrumentation needed to answer these questions, tho we thought you would know best.
- 2. Do we now have confidence to say We are logging the data we will need to answer our research questions: T221195?
- 3. Do we now have confidence to say We are logging the data we will need to answer our follow up questions: T232175#5545364?
cc @matmarex
I can answer as well – I don't think it will be necessary to re-check those. We should only re-check that the buckets are balanced 50-50.
Thank you, Bartosz.
@Mayakp.wiki, in light of the above, we can continue testing questions "2." and "3." below; they are not blocked by T236337 and T236338 being completed.
- 2. Do we now have confidence to say We are logging the data we will need to answer our research questions: T221195?
- 3. Do we now have confidence to say We are logging the data we will need to answer our follow up questions: T232175#5545364?
cc @MNeisler
QA for Q2 and Q3 in progress. Completed checks for few of the metrics.
There are a few observations which I will discuss with Megan next week. Update on checks are documented here.
Completed QA for Q2 and Q3. Everything looks good except for when I checked for Wiki field, I found 1 init event recorded on 10-01-2019 for commonswiki which is not part of the experiment as mentioned in the link. This is definitely Not at all an issue since it is just one event but whatever caused this event to be bucketed could potentially cause many other events from this wiki to be bucketed in the experiment maybe in the future and wanted to bring it to the team's attention - @ppelberg @DLynch @MNeisler
Summary of the checks performed posted in this QA Document.
Will post notebook soon.