@Ramsey-WMF @CBogen @Abit Can you clarify Structured Data's priority & timing for this request?

• Are the metrics needed prior to a launch? If so, what is your planned launch date?
• Are their other planned actions or decisions that depend on the metrics? Will not having the metrics block work or launches?

@kzimmerman

• Baseline metrics for the old search are what is time sensitive. We are hoping to have those by early August. That is approximately when we will be ready to launch an A/B test comparing the old results to the new results.
• Our OKRs depend heavily on the metrics for MediaSearch, and will help us make decisions going forward about the launch. We have a bit more time for that, as we're planning to launch in Q2.

Hi @nettrom_WMF - position is listed in the schema, but not listed in this ticket - can you confirm that we're planning to measure position for MediaSearch and that we'll have a baseline for position in the current Commons search? Thanks!

In T258229#6357127, @CBogen wrote:

Hi @nettrom_WMF - position is listed in the schema, but not listed in this ticket - can you confirm that we're planning to measure position for MediaSearch and that we'll have a baseline for position in the current Commons search? Thanks!

Yes, I want to have position in the schema for Media Search as I expect that to be something we'll be investigating down the line. I've added "average position of clicked result in successful searches" to the list of metrics above, as I suspect that'll be one of the metrics you're interested in for comparing backends in legacy search. While I'm not certain how we'd go about making comparisons of positions in a grid layout as the research wasn't clear on that (there are lots of complicating factors), I'm thinking that we can consult with Miriam on that when the time comes.

I've made T258183 a subtask of this task, since we can't build dashboards until the instrumentation is in place.

@CBogen @Ramsey-WMF since this depends on instrumentation that is not yet done (T258183) and we're coming upon the end of the quarter, I'm moving it to Q3.

We now have a draft dashboard on Superset that covers almost all the metrics specified in this task. Statistics are gathered daily and are driven by two notebooks that can be found on GitHub: this one aggregates the legacy and MediaSearch statistics, and this one aggregates the MediaSearch-specific funnel metrics. The dashboard then pulls in the data and calculates any necessary percentages on the fly.

I propose that we hold off on calculating how many filters and what filters are used until we have a few weeks of usage data, rather than aggregating that on a daily basis.

Given that MediaSearch was first made default for non-logged-in users and will become default for logged-in users later, I suspect we'll be wanting to be able to split statistics by whether the user was logged in or not. That's currently not possible as the mediasearch_interaction schema doesn't capture that. If we need that as well, we'll need two additional tasks, one to update the schema and instrumentation of Special:MediaSearch, and one to update the aggregation.

I'll discuss all of this with @CBogen in our next sync meeting and create tasks depending on what the needs are.

In T258229#6981007, @nettrom_WMF wrote:

We now have a draft dashboard on Superset that covers almost all the metrics specified in this task. Statistics are gathered daily and are driven by two notebooks that can be found on GitHub: this one aggregates the legacy and MediaSearch statistics, and this one aggregates the MediaSearch-specific funnel metrics. The dashboard then pulls in the data and calculates any necessary percentages on the fly.

I propose that we hold off on calculating how many filters and what filters are used until we have a few weeks of usage data, rather than aggregating that on a daily basis.

Given that MediaSearch was first made default for non-logged-in users and will become default for logged-in users later, I suspect we'll be wanting to be able to split statistics by whether the user was logged in or not. That's currently not possible as the mediasearch_interaction schema doesn't capture that. If we need that as well, we'll need two additional tasks, one to update the schema and instrumentation of Special:MediaSearch, and one to update the aggregation.

I'll discuss all of this with @CBogen in our next sync meeting and create tasks depending on what the needs are.

@nettrom_WMF and I discussed this and decided not to pursue differentiating non-logged-in users from logged-in users for now, because we have higher priority work to focus on. We think that if logged-in users experience specific issues, they'll let us know in on-wiki discussions.

Remaining on this task:

1. Add captions/clarify what each chart measures on the dashboard
2. Add a chart showing monthly numbers for each filter