Page MenuHomePhabricator

Ensure that CirrusSearch instrumentation isn't broken accidentally by other changes
Open, HighPublic

Description

As a relevance engineer, I want to automate testing of instrumentation so that this instrumentation isn't broken accidentally.

Instrumentation is a critical part of any search / relevance work. It is easy to break it accidentally and silently since it has no direct functional impact on the user experience. The migration to vue.js is one example of such a breakage. This is then detected only when we need this instrumentation, for example to run an A/B test. At that point, instrumentation needs to be fixed before we can run such a test. Early detection of breakage would simplify fixing it and would remove delays at critical phases of projects.

Detection of breakage could be done in multiple ways:

  • automated browser test / contract test / regression test
  • alerting based on metrics related to instrumentation

AC:

  • Search Platform team is notified when instrumentation is broken on a wiki