Change 334491 had a related patch set uploaded (by EBernhardson):
[WIP] Enable Sister project search AB test

https://gerrit.wikimedia.org/r/334491

Change 334673 had a related patch set uploaded (by DCausse):
[WIP] Configure A/B test for CrossProject search results sidebar

https://gerrit.wikimedia.org/r/334673

Change 334685 had a related patch set uploaded (by EBernhardson):
Convert InterwikiSearcher::MAX_RESULTS into variable

https://gerrit.wikimedia.org/r/334685

In T149806#2974790, @EBernhardson wrote:

Another clarification, in the spec we state:

Eventlogging needs to capture:
* if results were shown and from which wiki projects
...

I think we decided that this information would be obtained by joining the EventLogging data against the CirrusSearchRequestSet data. The link between the two is already collected, and the CirrusSearchRequestSet logs already contain the list of wiki's queried and the # of results returned for each query.

Yup, that's the plan.

In T149806#2975109, @EBernhardson wrote:

Also, more clarifications. For if the user clicked on an individual result and what wiki project that result came from along with if the user clicked on the 'more from' on any wiki project result that was displayed we agreed to record the URL of the link the user is clicking.

For a click through to an article this would be, for example:

/wiki/s:History_of_Iowa_From_the_Earliest_Times_to_the_Beginning_of_the_Twentieth_Century/3/Counties/Clayton

That would indicate they clicked an individual results, and that result was for wikisource.

Since that doesn't follow the usual format we're used to dealing with (e.g 'enwiki' => 'english'+'wikipedia'), is there a table somewhere for us to see how '/wiki/s' => '[language of project the search was made on]'+'wikisource'?

For a click through to the 'more from' button this would be:

https://en.wikibooks.org/wiki/Special:Search?search=guttenburg&fulltext=1

It looks like that will consistently be 'Special:Search' regardless of the wiki it's run on, but we should re-validate that after deployment. The 'ssclick' events for these will have the same position argument, indicating the position of the selected wiki in the ordering of the results.

Thanks for pointing that out! Yeah, that should definitely be one of the things we look for in the validation step.

Change 334721 had a related patch set uploaded (by EBernhardson):
Setup sister search prefix display types

https://gerrit.wikimedia.org/r/334721

Change 334685 merged by jenkins-bot:
Convert InterwikiSearcher::MAX_RESULTS into variable

https://gerrit.wikimedia.org/r/334685

I should add, it looks like adding the click types will require updating the search satisfaction schema revision number. As long as we are doing this is there anything else we want to change about the recorded event?

In T149806#2977257, @mpopov wrote:

In T149806#2974790, @EBernhardson wrote:

Another clarification, in the spec we state:

Eventlogging needs to capture:
* if results were shown and from which wiki projects
...

I think we decided that this information would be obtained by joining the EventLogging data against the CirrusSearchRequestSet data. The link between the two is already collected, and the CirrusSearchRequestSet logs already contain the list of wiki's queried and the # of results returned for each query.

Yup, that's the plan.

In T149806#2975109, @EBernhardson wrote:

Also, more clarifications. For if the user clicked on an individual result and what wiki project that result came from along with if the user clicked on the 'more from' on any wiki project result that was displayed we agreed to record the URL of the link the user is clicking.

For a click through to an article this would be, for example:

/wiki/s:History_of_Iowa_From_the_Earliest_Times_to_the_Beginning_of_the_Twentieth_Century/3/Counties/Clayton

That would indicate they clicked an individual results, and that result was for wikisource.

Since that doesn't follow the usual format we're used to dealing with (e.g 'enwiki' => 'english'+'wikipedia'), is there a table somewhere for us to see how '/wiki/s' => '[language of project the search was made on]'+'wikisource'?

Sadly there is not a single definitive answer here, which is quite annoying :( The sister project prefixes are the same for all sites at least though:

wikt: wiktionary
b: wikibooks
n: wikinews
q: wikiquote
s: wikisource
v: wikiversity
voy: wikivoyage.

Generally it's safe to assume es.wikipedia.org/wiki/n:Something will send the user to es.wikinews.org/wiki/Something. I'm not aware of any exceptions here but I wouldn't be surprised if there are a few. Translating that into a db name is a little harder, as there are exceptions (but i can't think of any off the top of my head). generally es.wikinews.org should be eswikinews, and es.wikipedia.org should be eswiki, but there isn't a strong guarantee there just that's how it's typically done.

I think for the purposes of analysis though, since we can't collect metrics from the wiki on the other end anyways, just having the name (via the map above) should be good enough?

For a click through to the 'more from' button this would be:

https://en.wikibooks.org/wiki/Special:Search?search=guttenburg&fulltext=1

It looks like that will consistently be 'Special:Search' regardless of the wiki it's run on, but we should re-validate that after deployment. The 'ssclick' events for these will have the same position argument, indicating the position of the selected wiki in the ordering of the results.

Thanks for pointing that out! Yeah, that should definitely be one of the things we look for in the validation step.

The sistersearch site has been updated will all the latest code (including un-merged patches). The number of live hacks is now down to 1, and that won't be needed in production. So basically this should now be pretty close to what we expect to ship.

EventLogging debugging is enabled, so if you are added to one of the satisfaction tests a notification will popup on the top right every time a new event is sent. Additionally events are logged to the javascript console. The best way to see the events triggered by clicks is to open the javascript console and enable 'preserve logs' (for chrome, other browsers likely have something similar). This will keep the events logged to console from disapearing between page loads.

Example urls to join a particular test. After visiting one of these urls your browser will stay in the test until

• no_sidebar: http://sistersearch.wmflabs.org/w/index.php?forceSearchSatisfaction=no_sidebar&cirrusUserTesting=no_sidebar&title=Special:Search&fulltext=1&search=test
• recall_sidebar_results: http://sistersearch.wmflabs.org/w/index.php?forceSearchSatisfaction=recall_sidebar_results&cirrusUserTesting=recall_sidebar_results&title=Special:Search&fulltext=1&search=test
• random_sidebar_results: http://sistersearch.wmflabs.org/w/index.php?forceSearchSatisfaction=random_sidebar_results&cirrusUserTesting=random_sidebar_results&title=Special:Search&fulltext=1&search=test

Note that random still gives the same user the same order on every search, so for a single user it is consistent (to not confuse them), but different users will get sidebar results in a different order.

@mpopov with chelsy out, do you want to review the data collection here and ensure we are collecting everything necessary? Links above will opt you into the test on sistersearch and show eventlogging as described.

Also note that because we are in the progress of updating relforge instances from trusty -> jessie it's running on half capacity right now and results will be slow to come back (unless the necessary data just happens to be in disk cache already).

In T149806#2994810, @EBernhardson wrote:

@mpopov with chelsy out, do you want to review the data collection here and ensure we are collecting everything necessary? Links above will opt you into the test on sistersearch and show eventlogging as described.

yup! I'm actually so tired of software engineering with the reportupdate thing that I actually asked her for dibs on data verification and stuff :)

Just checked and everything looks good! I'm guessing the Multimedia box at the top is in place of Commons result?

Change 334673 merged by jenkins-bot:
Configure A/B test for CrossProject search results sidebar

https://gerrit.wikimedia.org/r/334673

I've pushed the configuration part of this live. It can be seen on beta and once the train rolls forward this will be available on the production wikis to verify everything is working before shipping the javascript that turns on the test. Note that because this is only the trigger, and not the the javascript, future searches will not be in the test. You have to manually add the 'cirrusUserTesting=recall_sidebar_results' query string to searches to get the new rendering.

https://en.wikipedia.beta.wmflabs.org/wiki/?search=~something&cirrusUserTesting=recall_sidebar_results
https://en.wikipedia.beta.wmflabs.org/wiki/?search=~something&cirrusUserTesting=random_sidebar_results

It looks like there is an i18n message that might need to be switched from ->escaped() to ->parsed(). Small patch will be up shortly.

Change 336328 had a related patch set uploaded (by EBernhardson):
Switch search-interwiki-caption i18n to parsed

https://gerrit.wikimedia.org/r/336328

Change 336328 merged by jenkins-bot:
Switch search-interwiki-caption i18n to parsed

https://gerrit.wikimedia.org/r/336328

I'm not seeing in the task description, exactly which wiki's are we enabling the test on at first? Only enwiki?

In T149806#3006573, @EBernhardson wrote:

I'm not seeing in the task description, exactly which wiki's are we enabling the test on at first? Only enwiki?

list of languages that have all or nearly all projects that we might want to also test on (in no particular order):

• Arabic
• Czech
• English
• Finnish
• French
• German
• Greek
• Hebrew
• Russian
• Portuguese
• Swedish
• Chinese
• Ukrainian

@EBernhardson enwiki for sure. How about ruwiki, arwiki, frwiki, dewiki, zhwiki, itwiki? I have no idea how trivial or non-trivial it is to enable this test on those and other wikis.

It's pretty easy to set a list of wikis to test on, i just need the list. I'll update the patch with your list. If we want different sampling on different wikis things get a bit more complicated, but mostly just in deciding the samplings to use.

I'm OK with the same sampling on those wikis and OK with the current sampling rates we have in the description.

This test is expected to go live on February 9, 2017.

@EBernhardson: nevermind what I wrote earlier :) here's the actual answer, from the task description: "This test is expected to last at least a week and will be run on Persian, Italian, Catalan and Polish Wikipedias (their selection was based on community input)."

Yes, please - just those four because we have not yet communicated to other wikipedias that a test might be run on them. :) We want to just run on Persian, Italian, Catalan and Polish Wikipedias at this time. Thanks!

Ok, test updated to run on fawiki, itwiki, cawiki and plwiki

cawiki is part of group1, so wmf.11 went live there today and is testable:

https://ca.wikipedia.org/wiki/?fulltext=1&search=cossolar&cirrusUserTesting=recall_sidebar_results

There is a problem with their pre-existing sidebar, we will probably want to remove it somehow during the test but I'm not sure (yet) where it comes from. Will investigate.

Oh nice - but, we definitely need to fix not showing the original sidebar. I thought we had already put code in for this case - maybe it's specific only to itwiki? @Jdrewniak can you help?

that particular sidebar is specific to cawiki (catalan), but i might have seen it on other wiki's. I'm having a bit of trouble tracking down where it comes from though ...

I found it, but i'm not sure what we can do about it that isn't a complete hack. The sidebar itself it is a hack, they've embedded it into the i18n message displayed when your search string doesn't exist as a page on the wiki:

https://ca.wikipedia.org/wiki/MediaWiki:Searchmenu-new

About the only thing we can do is disappear it with javacsript, giving a bit of a FOUC. We can't suppress this from being output, unless we hack up something in core to kill the message entirely on cawiki.

Change 336843 had a related patch set uploaded (by EBernhardson):
Temporary hax to hide cawiki's hacked in search sidebar

https://gerrit.wikimedia.org/r/336843

Change 336843 merged by jenkins-bot:
Temporary hax to hide cawiki's hacked in search sidebar

https://gerrit.wikimedia.org/r/336843

Change 336855 had a related patch set uploaded (by EBernhardson):
Temporary hax to hide cawiki's hacked in search sidebar

https://gerrit.wikimedia.org/r/336855

Change 334491 merged by jenkins-bot:
Enable Sister project search AB test

https://gerrit.wikimedia.org/r/334491

Change 336855 merged by jenkins-bot:
Temporary hax to hide cawiki's hacked in search sidebar

https://gerrit.wikimedia.org/r/336855

Mentioned in SAL (#wikimedia-operations) [2017-02-09T20:05:13Z] <thcipriani@tin> Synchronized php-1.29.0-wmf.11/resources/src/mediawiki.special/mediawiki.special.search.interwikiwidget.styles.less: SWAT: [[gerrit:336855|Temporary hax to hide cawiki hacked in search sidebar]] T149806 (duration: 00m 40s)

Change 336896 had a related patch set uploaded (by EBernhardson):
Enable Sister project search AB test

https://gerrit.wikimedia.org/r/336896

cawiki looks good now. group2 hasn't rolled forward yet so unable to test, but the patch is cherry picked for deploy in ~3 hours. If group2 hasn't rolled forward by then will need to delay the test until monday (but not expecting any problem).

Polish: https://pl.wikipedia.org/wiki/Special:Search?search=test&fulltext=1&cirrusUserTesting=random_sidebar_results
Arabic: https://fa.wikipedia.org/wiki/Special:Search?search=test&fulltext=1&cirrusUserTesting=random_sidebar_results
Catalan: https://ca.wikipedia.org/wiki/Special:Search?search=test&fulltext=1&cirrusUserTesting=recall_sidebar_results
Italian: https://it.wikipedia.org/wiki/Special:Search?search=test&fulltext=1&cirrusUserTesting=recall_sidebar_results

Everything looks good to go, will ship out the patch to enable the test in an hour with SWAT.

Change 336896 merged by jenkins-bot:
Enable Sister project search AB test

https://gerrit.wikimedia.org/r/336896

Mentioned in SAL (#wikimedia-operations) [2017-02-10T00:32:11Z] <thcipriani@tin> Synchronized php-1.29.0-wmf.11/extensions/WikimediaEvents: SWAT: [[gerrit:336896|Enable Sister project search AB test]] T149806 (duration: 00m 45s)

Change 334721 merged by jenkins-bot:
Setup sister search prefix display types

https://gerrit.wikimedia.org/r/334721

Mentioned in SAL (#wikimedia-operations) [2017-02-10T01:43:11Z] <ebernhardson@tin> Synchronized wmf-config/CirrusSearch-common.php: Setup sister search prefix display types T149806 (duration: 00m 48s)

Mentioned in SAL (#wikimedia-operations) [2017-02-10T01:48:11Z] <ebernhardson@tin> Synchronized wmf-config/CirrusSearch-common.php: Setup sister search prefix display types T149806 (duration: 00m 40s)

Test is running, only minor glitch is the search-interwiki-caption message has never been translated in catalan. Someone will need to do that to get simple names instead of the "Resultats de ca.wikisource.org:" messages

Thanks so much for your work on this, @EBernhardson!

I did some looking around other wikis using the generalized link and found that several other languages that don't have the project names translated.

Is there a better way to display the project name when it hasn't been translated?

@Jdrewniak - we should also refine the text to be as similar as possible - Wikitionary has: Word definitions from Wiktionary but Wikibooks has In Wikibooks and Wikiversity has From Wikiversity

The other available option is the project-localized-name-cawikiquote, which in english would be ''Catalan Wikiquote". The annoying part is I don't know that the renderer here has access to a map from interwiki prefix to the dbname used in these message names. There doesn't seem to be individual messages that just translate the project names without the language as well.

Love it.
1 emphatic request: Please make the main links use the normal Blue color, as per almost all standard links on our sites.

• Context: I first did a mouse-over of (more) because it was the only blue link (but then I saw it was just another local search). After I realized that "Irlanda" was the direct link, I tried clicking on ALL the bold black text (and icon), because I was suddenly unsure which parts were links, and which were just bold text...

Looks nice so far! However the multimedia results seem to be frequently irrelevant, perhaps a result of searching based primarily on title... Titles of images are often not descriptive, are in other languages, or contain irrelevant details that are sorted/scored higher than relevant images that appear in articles.

It's also quite odd that the links look like non-linked section headers; recommend using standard link colors here.

There's also a "flashing" behavior where the sidebar moves down a fraction of a second after loading, when the multimedia bits are loaded; this is a bit distracting.

The display of the metadata from Wikisource is kindof ugly in that preview: its doing both the title and the file name, and some other stuff.

The three surfaced images appear to be shown backwards from the top 3 results, resulting in the 3rd result being shown largest and the top result being shown smallest:

In T149806#3013680, @gerritbot wrote:

Change 336843 had a related patch set uploaded (by EBernhardson):
Temporary hax to hide cawiki's hacked in search sidebar

https://gerrit.wikimedia.org/r/336843

 /* Evil temporary hax for cawiki */
 #sisterproject {
	display: none;
 }

For once, I suppose this was fine. But its quite far out of scope for MediaWiki core master (WikimediaEvents or a wmf-only patch would be better).

Assuming there will likely be more conflicts with future roll outs, better use a class on a common ancestor. Then this could placed in local wiki's Common.css, e.g.:

.mw-searchresults-has-iw #sisterproject {
  display: none;
}

I suppose such class would be useful in general as well to make other minor layout adjustments where needed, to accommodate the sidebar. It seems interwikiwidget.styles.less already makes such adjustment, but without a scope class - under the assumption the styles won't be loaded unless the mode is activated, which is a pattern better avoided.

Change 339333 had a related patch set uploaded (by Krinkle):
Revert "Temporary hax to hide cawiki's hacked in search sidebar"

https://gerrit.wikimedia.org/r/339333

Change 339333 merged by jenkins-bot:
Revert "Temporary hax to hide cawiki's hacked in search sidebar"

https://gerrit.wikimedia.org/r/339333

Change 340111 had a related patch set uploaded (by Jdrewniak):
Fixing search results percentage width

https://gerrit.wikimedia.org/r/340111