Page MenuHomePhabricator

EPIC: run an A/B test to evaluate new features involved by enabling BM25
Closed, ResolvedPublic

Description

This task is about running an A/B test to confirm the benefits of the new features we implemented tahnks to BM25.
Since BM25 requires a reindex we cannot run an A/B easily.
The solution would be to use a second cluster that remains configured with the ClassicSimilarity. We could use the UserTesting backend framework to switch between both clusters.
Tasks (not yet created)

  • Configure the backend UserTesting to switch between eqiad and codfw
  • Add config flag that simulates extra latency
  • Use the search satisfaction schema to switch between groups like we did for the Textcat A/B test
  • All usual tasks that we create when enabling A/B tests.