Full write-up (and it's a lot!) is on MediaWiki.

Summary Results

• ZRR↓: zero-results rate decreased (good)
• Res↑: number of queries with more results increases (good)
• topΔ: top result changes (unclear)
• topΔ+: top result changes, checked (good)
• ZRR↑: zero-results rate increased (possibly bad)
• ZRR↑+: zero-results rate increased due to improved precision (good)
• noΔ: nothing changed (not good)
• net+: net percent of samples with a good outcome

Our targe goal of 75% improvement in either zero-results rate (ZRR↓, preferred) or total results returned (Res↑) held true for all but the introduction of ICU tokenization, where we expected (and found) improvements in both recall and precision, depending on which "foreign" script a query and text on-wiki is in.

For each collection of targeted queries for a given feature, we saw an improvement across more than 90% of query samples.

For the general random sample of Wikipedia queries, we also saw improvement across more than 90% of language samples, mostly in direct improvements to the zero-results rate—indicating that the features we are introducing to all (or almost all) wikis are useful upgrades!