Patch to configure and start the tests was prepped in the earlier patch, shipped the test today. Can turn it off Oct 13.

Patch to start test: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1187108

I mistakenly posted this to the parent ticket, but it belongs here:

In T390858#11274600, @EBernhardson wrote:

Preliminary reports. They might become final, but they haven't been reviewed by anyone else yet:

• All Wikis
• English, French, and German

In both breakdowns default_1 outperforms the other two variations. Top level stats for the change from control to the prefix_len=1 variant:

!! - statistically significant change

ALL WIKIS

display rate: 23.0 -> 24.5 !!
dym selection rate: 0.60 -> 0.61
auto rewrite rate: 9.26 -> 9.99 !!
user selected clickthrough rate: 26.2 -> 26.0
auto rewrite clickthrough rate: 19.9 -> 20.6 !!
all query clickthrough rate: 33.03 -> 33.13

EN, DE, FR

display rate: 28.6 -> 29.8 !!
dym selection rate: 0.52 -> 0.53
auto rewrite rate: 10.23 -> 10.86 !!
user selected clickthrough rate: 31.2 -> 30.9
auto rewrite clickthrough rate: 24.5 -> 25.3 !!
all query clickthrough rate: 37.90 -> 38.03

We reviewed the report in the wed meeting, where a report against major spaceless languages was requested. A quick runthrough of a report restricted to zhwiki and jawiki finds that the default_1v profile is significantly better than the others, suggesting that the variant does potentially have benefits, but it may depend on which language. As such I've run a batch of reports against the top few wikis by size, and a few selected languages that have unique language features:

• arwiki
• dewiki
• enwiki
• eswiki
• frwiki
• hiwiki
• jawiki
• plwiki
• ruwiki
• srwiki
• swwiki
• zhwiki

Will need to ponder on these, it's a large variety of data to consider. There are clearly some languages that benefit more from the variant field than others.

From our review of the initial reports there is also a bit of surprise around the opening_text language model performing worse than the default language model. One plausible explanation is that there are word patterns seen in queries but not the opening text, only in the title fields. As such it would be interesting to run a follow-up test comparing title+redirect.title vs title+redirect.title+opening_text. For that I've created T407432.

Followup test with variant using the combined title+opening_text field, in T407432, surprisingly achieved almost the same results as the opening_text on it's own. As in this first test the default_1 profile, with only titles and prefix_len=1 looks to be the best performing.

The second AB test has been undeployed and reported on. Patches are also up that will stop building the variant field in production and changing cirrus to prefix_len=1.