Background

This task evaluates all instrumentation requirements described in the Semantic Search MVP Measurement Plan and rewrites placeholder (green) events/properties so they align with the Android Apps analytics system.

Scope of Work

1. Review Measurement Requirements (Pages 1–7)

Identify instrumentation required to evaluate:

• Search session initiation and exit
• Query submission and reformulation
• Result impressions and clicks
• Exploration and reformulation labeling
• Good abandonment
• In-app satisfaction micro-surveys
• Seven-day search return behaviors
• Latency and performance guardrails
• Per-result metadata (rank, provenance, snippet type, retrieval mode)

2. Map PDF “Placeholder” Events to Apps System

Review the Measurement Plan’s core events:

• search_session_start
• query_submitted
• search_results_impression
• search_result_click
• search_query_reformulated
• search_exit
• search_good_abandonment_candidate

For each:

• Determine whether an equivalent Apps event already exists
• Identify naming updates needed
• Identify if it must be emitted client-side, server-side, or both

3. Rewrite Placeholder Event Properties to Apps Schema

Review all Required Event Properties (Pages 4–5) and rewrite them to match Apps logging conventions.

Property groups to rewrite:

• '''Session metadata''' (session_id, user_id_hashed, entry_point, etc.)
• '''Query metadata''' (query_string_hashed, tokens, language, similarity metrics)
• '''Retrieval/API metadata''' (retrieval_mode, latency_ms, fallback_triggered)
• '''Results list metadata''' (snippet_type, provenance_label_shown, rank, snippet_length_chars)
• '''Click metadata''' (clicked_result_id, scroll_depth_at_click, dwell_time_on_article_ms)
• '''Exploration vs reformulation''' signals
• '''Exit metadata''' (exit_reason, queries_in_session_count, session_length_ms)
• '''Performance/guardrail metadata''' (api_timeout_occurred, network_type, cost_bucket)

For each property:

• Indicate whether it is an existing field, renamed field, new addition, or not feasible
• Propose final property names consistent with Android logging standards

4. Evaluate Instrumentation Needed for all Metrics

Map the Measurement Plan’s metrics to required events and properties, including:

• Search sessions initiated per unique user
• Avg. session length
• Time-to-click
• Reformulation rate
• Good abandonment rate
• Top-results CTR
• Satisfaction and perceived relevance
• Seven-day search return rate
• Latency, fallback, and error tracking

6. Identify Missing Instrumentation

Highlight missing fields needed for:

• Ranking mode (lexical / semantic / hybrid)
• Provenance indicator exposure
• Snippet-length variants
• Semantic → lexical fallback detection
• Per-result metadata arrays
• Scroll depth tracking
• Observation window compliance for “good abandonment”
• Other identified metrics

7. Produce Final Instrumentation Specification (Deliverable)

The final document should include:

• Revised event and property names
• Event triggers and firing conditions
• Required timestamps
• Cross-join schema
• Open questions for Engineering
• A final logging schema proposal for the MVP
• Recommendations for placeholder goals (e.g. X%)

Deliverables

• Annotated review of Measurement Plan instrumentation requirements
• Full event & property rewrite table (placeholder → App final name)
• Gap analysis (missing fields vs. available fields)
• Proposed Android-side logging schema
• Proposed server-side logging schema updates
• Metric-to-event/property mapping table
• Recommended goals for placeholder metrics based on our existing growth rates