| mpopov | |
| Apr 14 2016, 5:30 PM |
| F3943115: daily_ctr_2_mini.png | |
| Apr 28 2016, 6:14 PM |
| F3942989: daily_ctr_mini.png | |
| Apr 28 2016, 5:42 PM |
| Unknown Object (File) | |
| Apr 28 2016, 12:02 AM |
Erik managed to make JS send click events that fire between the user clicking a search result and the browser taking them to the page they clicked on. We need to see if the click events allow us to compute the same clickthrough rate as visitPage events do, and how much (if any) browser bias there is.
This validation is required before we try hooking up the TextCat A/B/C test (T121542) to the front-end.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | EBernhardson | T137158 Compile and then resolve issues with TextCat A/B test data | |||
| Declined | mpopov | T134320 Analyse results of TextCat A/B test | |||
| Resolved | EBernhardson | T130321 Disable Schema:Search, since it's outdated and redundant | |||
| Resolved | mpopov | T129564 Switch Desktop data collection for dashboards to use TestSearchSatisfaction2 instead of Search schema | |||
| Resolved | EBernhardson | T134319 Turn off TextCat A/B test on the English Wikipedia on or after May 23 | |||
| Resolved | debt | T134318 Verify data pipeline for TextCat A/B test on English Wikipedia | |||
| Open | None | T118278 [EPIC] Improve Language Identification for use in Cirrus Search | |||
| Resolved | EBernhardson | T121542 Write and deploy an A/B Test on enwiki using TextCat for Language Identification | |||
| Resolved | mpopov | T132706 Validate click events in TestSearchSatisfaction2 |
Started putting together a detailed report at https://github.com/wikimedia-research/Discovery-Search-Adhoc-ClickEventValidation Going to include some of the results here as a comment.
Terminology:
| Sessions from TestSearchSatisfaction2 | Proportion of sessions |
| sessions with valid clicks only | 89.235% |
| sessions with valid visits only | 16.849% |
| sessions with valid clicks AND visits | 14.327% |
| sessions with more valid clicks than valid visits | 78.398% |
| sessions with more valid visits than valid clicks | 3.371% |
| sessions with valid clicks AND visits, AND clicks match visits 100% | 8.971% |
| sessions with valid clicks AND visits, AND clicks don't match visits at all | 0.756% |
| sessions with clicks but not valid clicks | 8.883% |
| sessions with visits but not valid visits | 0.000% |
| sessions with valid clicks that couldn't be matched with valid visits | 4.668% |
| sessions with valid visits that couldn't be matched with valid clicks | 2.752% |
After filtering down to only sessions where the clicks matched the visits 100% and sessions that had 0 clicks/visits, it looks like 52.12% might be our most-valid overall clickthrough rate, with the following daily breakdown:
| Date | Total valid sessions | Abandoned sessions | Clickthrough'd sessions | Clickthrough rate (%) |
| 2016-04-02 | 1631 | 752 | 879 | 53.893 |
| 2016-04-03 | 1946 | 909 | 1037 | 53.289 |
| 2016-04-04 | 2575 | 1255 | 1320 | 51.262 |
| 2016-04-05 | 2586 | 1297 | 1289 | 49.845 |
| 2016-04-06 | 2588 | 1276 | 1312 | 50.696 |
| 2016-04-07 | 2455 | 1187 | 1268 | 51.650 |
| 2016-04-08 | 2322 | 1118 | 1204 | 51.852 |
| 2016-04-09 | 1740 | 841 | 899 | 51.667 |
| 2016-04-10 | 2047 | 976 | 1071 | 52.320 |
| 2016-04-11 | 2798 | 1350 | 1448 | 51.751 |
| 2016-04-12 | 2662 | 1267 | 1395 | 52.404 |
| 2016-04-13 | 2670 | 1293 | 1377 | 51.573 |
| 2016-04-14 | 2555 | 1222 | 1333 | 52.172 |
| 2016-04-15 | 2189 | 1026 | 1163 | 53.129 |
| 2016-04-16 | 1660 | 774 | 886 | 53.373 |
| 2016-04-17 | 1926 | 950 | 976 | 50.675 |
| 2016-04-18 | 2570 | 1201 | 1369 | 53.268 |
| 2016-04-19 | 2544 | 1236 | 1308 | 51.415 |
| 2016-04-20 | 2533 | 1200 | 1333 | 52.625 |
| 2016-04-21 | 2412 | 1158 | 1254 | 51.990 |
| 2016-04-22 | 2184 | 1014 | 1170 | 53.571 |
| 2016-04-23 | 1692 | 765 | 927 | 54.787 |
| 2016-04-24 | 1982 | 960 | 1022 | 51.564 |
How does this method compare with the other estimation methods above? Remarkably close to just looking at "CTR as % of sessions that had at least one click event":
