Page MenuHomePhabricator

Write A/B test code for test with phrase rescore boost of 1
Closed, ResolvedPublic

Description

  • Control: Has default phrase rescore boost of 10
  • Test: Has phrase rescore boost of 1
  • Bucketing/sampling: Satisfaction schema sampling increse from 0.5% to 1%. 50% of the 1% go into the Test bucket.
  • Measuring:
    • number of searches per session (Should we assume ↓ is good? – MP)
    • clickthrough rate (↑ is good) (P.S. this is inverse of session abandonment)
    • position of first clicked result (I don't expect this to change since vast majority of clicks are already on the first result – MP)
    • time to first clickthrough (↓ is good)
    • search session duration (we should investigate the impact, but not assign a pos/neg judgement to either direction; can elaborate if needed – MP)
    • number of results visited (as with search session duration, we should see how this is impacted, but conflicting goals make make it hard to assign a pos/neg judgement – MP)
    • time spent on visited pages ("hazard" in survival analysis, similar concern as with search session and number of results visited)
  • Duration: one week

More context for this task is available in T129593.

Event Timeline

Deskana created this task.Mar 11 2016, 12:57 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 11 2016, 12:57 AM
Deskana updated the task description. (Show Details)Mar 11 2016, 12:58 AM
EBernhardson updated the task description. (Show Details)Mar 11 2016, 1:20 AM

Change 276685 had a related patch set uploaded (by EBernhardson):
Run AB test with phrase boost of 1

https://gerrit.wikimedia.org/r/276685

mpopov updated the task description. (Show Details)Mar 11 2016, 8:25 PM
mpopov added a subscriber: mpopov.Mar 11 2016, 10:05 PM

Change 276685 merged by jenkins-bot:
Run AB test with phrase boost of 1

https://gerrit.wikimedia.org/r/276685

Change 277337 had a related patch set uploaded (by EBernhardson):
Run AB test with phrase boost of 1

https://gerrit.wikimedia.org/r/277337

Change 277337 merged by jenkins-bot:
Run AB test with phrase boost of 1

https://gerrit.wikimedia.org/r/277337

Deskana triaged this task as Normal priority.Mar 15 2016, 3:41 AM
Deskana closed this task as Resolved.