This would help us to determine how important the ordering of the top few results, currently users click through the second result much more often than the third. Is this because the second result is better, or is this just user behaviour?
| event_position | count | percent | +----------------+--------+---------+ | 1 | 154150 | 64.26% | | 2 | 34214 | 14.26% | | 3 | 16213 | 6.76% |
Current definition of Search SLO is in T335576. We might revisit this as part of a deeper effort.
We actually ran this test some years ago, about 1.5 years after this ticket was filed: https://commons.wikimedia.org/wiki/File:Swap2and3_Search_Test_Analysis.pdf
Short excerpt from the summary;
Overall, we found that user engagement decreased by swapping the second and the third search results. Position 2 had a higher clickthrough rate than position 3 in both groups. However, when comparing the same position between groups the clickthrough rate of third results in the test group was higher and the clickthrough rate of second results in the test group was lower. We also found that test group users were less likely to click on the 2nd result first than the control group, while they are more likely to click on the 3rd result first. Based on these analysis results, we believe that it is not an either/or situation with position (order displayed) and “quality” (as determined by Cirrus). We suspect that both position and quality matter in user behavior, but with different weights.