key components
- March deployment
- brief, Product Planning doc: https://docs.google.com/document/d/1SPrfaeX35sqWfUMwRctBLa3clZSX66-2gLWdPtnIqhk/edit#heading=h.wa6t6efaafc4
- irene/mike notes doc with discussion: https://docs.google.com/document/d/16Bi24LIV9R4HMAs4kcyio457fHFQYOSRxWvLo6ykomg/edit#heading=h.bq8pms66tw9a
There appear to be four different questions here...
- MVP questions to answer
- MVP data to collect
- mid-term questions to answer and data collected which will answer those questions
- Where can Irene support
MVP priority is to make sure the extension is functional and being installed: i.e. we want to be able to track how many users are installing it to reach our minimal goals for enough experimental data. We also want to know that they are able to use the extension successfully: e.g. it's not timing out, they are not abandoning it due to latency, they are getting results to their queries.
Ideally these are all metrics we can try to directly influence to get more people using the extension: e.g. try to reduce latency; increase search results, etc
Elias is also working on logging some operational metrics, so we should touch base with him.
After MVP, we will want to be able to look at logged historical queries and their results in order to do an internal quality audit of how well the extension is performing.
We will also want to make sure we're logging the right metrics in order to have some signals to help answer the major experimental questions. This can be in conjunction with qualitative surveys
Components:
- Understand and prioritize Fact Checker questions to answer
- Discuss questions with PM to understand the crux of the questions’ intention, how important is that question, can we answer that question with the information available…and what is the cost of answer that question
- Help PM define which questions are best answered with product quantitative vs qualitative data
- Review sample logging data to make intro rough level logging requests/recommendations
- Review plugin key components to ensure key analytics tracking details are carried over to the Fact Checker code base
- Identify MVP data to collect from OpenAI logging
- Identify which data sources can answer which questions (whether with full conviction or as a signal or proxy)
- Define where Irene support in February/March (note product launches in March…what date?)
- Note tiers of work involved (short term vs mid term)