Page MenuHomePhabricator

Switch Desktop data collection for dashboards to use TestSearchSatisfaction2 instead of Search schema
Closed, ResolvedPublic2 Story Points

Description

Note that the format of the data in TSS2 is very different from Search so the switch is non-trivial. This is also an entirely backend change and nothing on the dashboards needs to be changed, except this switch will result in different values for the Desktop data, and should be annotated appropriately.

Details

Related Gerrit Patches:
wikimedia/discovery/rainbow : masterAnnotate Desktop schema switch
wikimedia/discovery/golden : masterSwitch Desktop data from Schema:Search to Schema:TSS2

Related Objects

Event Timeline

Deskana created this task.Mar 10 2016, 9:19 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 10 2016, 9:19 PM
Deskana triaged this task as Normal priority.Mar 10 2016, 9:19 PM
Deskana moved this task from Needs triage to UX on the Discovery board.
Deskana moved this task from UX to On Sprint Board on the Discovery board.
Deskana added subscribers: mpopov, EBernhardson.

@mpopov wanted to ask @EBernhardson whether Schema:Search was turned off, so we know what we're doing here.

It has not been turned off yet, but i'm imagining that it can be once we evaluate the new data coming in.

I don't think this is a sufficiently defined task. Which metrics precisely are we meant to surface that we are currently not?

As it is described, I think this is actually a task asking us to switch our Desktop-related metrics from Schema:Search to Schema:AutocompleteSatisfaction? Also it seems like we would benefit from having a second zero results rate on the dashboard that is derived from AutocompleteSatisfaction, rather than Cirrus request logs.

I don't think this is a sufficiently defined task.

Well, if you think it is not, then by definition it is not! :-) Let's figure out these details in our analysis meeting which is (fortunately) taking place in 25 minutes.

As it is described, I think this is actually a task asking us to switch our Desktop-related metrics from Schema:Search to Schema:AutocompleteSatisfaction?

Yes, having thought about it more I guess that is what I'm asking. Let's make this task for that.

Also it seems like we would benefit from having a second zero results rate on the dashboard that is derived from AutocompleteSatisfaction, rather than Cirrus request logs.

That sounds worthwhile, but should be in a separate task for that.

Deskana renamed this task from Create dashboard for AutocompleteSatisfaction schema to show how well the completion suggester is doing at meeting user needs to Switch over "Desktop" tabs on Search dashboards to AutocompleteSatisfaction schema, to show how well the completion suggester is doing at meeting user needs.Mar 17 2016, 8:20 PM
mpopov renamed this task from Switch over "Desktop" tabs on Search dashboards to AutocompleteSatisfaction schema, to show how well the completion suggester is doing at meeting user needs to Switch Desktop data collection to use TestSearchSatisfaction2 instead of Search schema.Mar 17 2016, 10:15 PM
mpopov claimed this task.
mpopov updated the task description. (Show Details)
mpopov set the point value for this task to 2.
Deskana renamed this task from Switch Desktop data collection to use TestSearchSatisfaction2 instead of Search schema to Switch Desktop data collection for dashboards to use TestSearchSatisfaction2 instead of Search schema.Mar 22 2016, 8:14 PM
mpopov added a comment.Apr 1 2016, 6:40 PM

I just realized that there's a component that Schema:Search has that Schema:TestSearchSatisfaction2 doesn't: time to display results, which means we can't switch the data scripts to the new schema because that'd stop the data used for http://discovery.wmflabs.org/metrics/#desktop_load

@EBernhardson: do you think we can add a "time it took to display SERP" field for the action=searchResultPage events? How doable is that?

Doesn't look like it's too hard, should be a couple hours work

Doesn't look like it's too hard, should be a couple hours work

I need to file a task for that and put it in the backlog for search. I'll do that after this meeting.

Erik said the patch in T132516 should be out by next Thursday.

Note to self that this will require us to update the reference to the TestSearchSatisfaction2 table in the data retrieval codebase for dashboards.

SELECT min(event_msToDisplayResults) FROM TestSearchSatisfaction2_15700292; gives -1473820578422

So I think we're still stalled on this because there appears to be a bug in calculating time to display results.

hmm, basically at this point i am trusting users browsers + clocks to return sane values ... will look closer to see what can be done.

debt added a comment.Jul 6 2016, 7:49 PM

Hey @EBernhardson - how're you doing on this? :)

patch has been merged from T132516, will be deployed everywhere tomorrow as part of the train rollout. There is an open question from Krinkle about if the particular timing chosen is the right one.

Change 298891 had a related patch set uploaded (by Bearloga):
Switch Desktop data from Schema:Search to Schema:TSS2

https://gerrit.wikimedia.org/r/298891

Change 298891 merged by Bearloga:
Switch Desktop data from Schema:Search to Schema:TSS2

https://gerrit.wikimedia.org/r/298891

Change 298905 had a related patch set uploaded (by Bearloga):
Annotate Desktop schema switch

https://gerrit.wikimedia.org/r/298905

Change 298905 merged by Bearloga:
Annotate Desktop schema switch

https://gerrit.wikimedia.org/r/298905

Switched to using TestSearchSatisfaction2 instead of Search schema for Desktop event counts and load times data. Let's keep Search up for another week or so just in case something isn't working as it should. Annotations have been deployed to the beta version, but not production: http://discovery-beta.wmflabs.org/metrics/

it's been about a week, how are we feeling about turning off Schema:Search?

it's been about a week, how are we feeling about turning off Schema:Search?

@mpopov How are things looking?

debt closed this task as Resolved.Aug 4 2016, 11:43 PM