Page MenuHomePhabricator

Search metrics dashboard should (somehow) show queries that returned 0 results
Closed, DeclinedPublic

Description

As Deskana originally said in T101774: I want us to figure out what users are searching for when they get zero results, so we can know whether they're actually searching for something that we can't give them meaningful results for (e.g. "isfnaifiabfiabuabig") or something where we could do better (e.g. "borick obama" which gives two mostly irrelevant results)

That task was narrowed to some easy/safe work. This remaining part is trickier, because search queries might contain private data. We can't just display them on a publicly-accessible dashboard. Maybe we can MD5 them, or maybe we can restrict access to that part of the dashboard, or maybe we can do some kind of human redaction. TBD.

Event Timeline

ksmith raised the priority of this task from to Low.
ksmith updated the task description. (Show Details)
ksmith subscribed.

Aggregate data (X searches returned 0 results today) is fine. Specific search phrases should not leave the cluster, as a rule. If that's needed, let's talk and fine a safe way to do that.

@ksmith and I chatted to @csteipp, and he said that we're okay from his perspective for security/privacy (for now) if we publish the "top n queries with 0 results" on the dashboards, provided there's also some metric in there to really ensure that we're not publishing low traffic queries, e.g. top n queries && query_count > 1000

Moving forwards he'd like us to write a brief set of guidelines (e.g. about a paragraph) that explains our general principles around privacy and how we're publishing this data, which sounds fine to me. That can live on the dashboard index at searchdata.wmflabs.org.

Hi!

I wanted to post a recent investigation into this issue by @TJones - here's a snippet of it and the full analysis is here:

"I think the problem with all of these strategies is that so many high-frequency queries would be eliminated by any of them that any useful mining would be down to slogging through the low-impact long tail.

I don’t think there’s a lot here worth extracting, though others may disagree. The privacy concerns expressed earlier are genuine, and simple attempts to filter PII (using patterns, minimum IP counts, etc) are not guaranteed to be effective."

I'd like to close this ticket out - but wanted to get thoughts on it first.

Related: T8373

Per @debt's comment above, this isn't going to happen any time soon. Marking as declined.