Search Quality Monitoring Metrics Dashboard
As a member of the Search team, I want to quickly detect unexpected changes in search result quality, so that we can identify regressions early and investigate potential retrieval or ranking issues before they significantly impact users.
Metrics
The following metrics should be available and explorable per wiki:
- Zero results rate
- Percentage of search sessions/queries returning no results
- Broken down by wiki
- Click position
- Distribution and/or average position of clicked results
- Lower click positions indicate users find relevant results near the top
- Broken down by wiki
Acceptance Criteria
- Both metrics are available in Grafana and/or Superset
- Definitions for both metrics are documented
- Alerting creates Phabricator tasks
Notes / Considerations
- Clarify whether metrics are computed from:
- raw search requests,
- search sessions,
- or unique users
- Define expected handling for:
- bots (?),
- desktop/mobile
- autocomplete/search-as-you-type,
- and anonymous vs logged-in traffic
- For click position:
- consider median and percentile views in addition to averages
- consider separating desktop/mobile if behavior differs significantly
- For zero results rate:
- consider alert thresholds for significant regressions