Page MenuHomePhabricator

Change search satisfaction schema sub test from common terms to language search
Closed, DeclinedPublic

Description

The existing common terms test is interesting, but not part of our quarterly goals. To get things moving forward we need to change the sub test over to using the new language search. For the most part this should just be changing the form parameters it is adding to have the hidden field 'cirrusAltLanguage' set to 'yes'. To determine if the result page had an interwiki response should be able to check for p.mw-search-interwiki-header

Event Timeline

EBernhardson raised the priority of this task from to Needs Triage.
EBernhardson updated the task description. (Show Details)
EBernhardson added a subscriber: EBernhardson.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 2 2015, 5:25 PM
Restricted Application added a project: Discovery. · View Herald TranscriptNov 2 2015, 5:25 PM

@mpopov what do we need to collect for the control group here? The language detection feature only runs on queries with zero results, so flagging the queries that could have been effected but wern't isn't particularly useful.

Perhaps the control group doesn't need to be specified, and you just take a random sampling of people within the satisfaction schema?

Change 250571 had a related patch set uploaded (by EBernhardson):
[WIP] Change search satisfaction subtest to language search

https://gerrit.wikimedia.org/r/250571

mpopov added a comment.Nov 3 2015, 5:02 PM

I think what we're interested in is whether our "satisfaction" metrics for the language search folks are consistent with what we've been seeing. Like, we shouldn't expect their clickthroughs and time-on-page to be greater than controls'. What we should hope for is that those things aren't much lower than controls.

Yeah, I'll just take a random sample for the control group.

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 3 2015, 5:02 PM

Change 250571 abandoned by EBernhardson:
Change search satisfaction subtest to language search

Reason:
the generated links go to another domain, so the existing method of using localStorage in the satisfaction schema will not work for the inter-wiki use case. Will work up another patch to measure click through.

https://gerrit.wikimedia.org/r/250571

EBernhardson closed this task as Declined.Nov 16 2015, 4:49 PM

After reviewing the previous test we ran, we basically already have this information just not in the event logging table. Within hive wmf.webrequest table all interwiki clicks are recorded with x_analytics_map['wprov'] = 'iwsw1'. This was already going to be a poor substitute for the default satisfaction schema, due to the inability to pass localstorage data across sub-domains. As such marking this 'declined'. We will source the appropriate data about clickthroughs from hive.