Page MenuHomePhabricator

click-result events underreported in desktop search event logging
Closed, ResolvedPublic

Description

It seems like the search event logging is underreporting click events. The data shows that only 4.3% of users who start a search end up navigating to a result. This seems unlikely to me.

We need to investigate whether this logging is working properly. @bmansurov was the person who implemented the logging originally, and may be able to assist.

Extra notes:

There were 24,610 sessions started [1], and only 1,217 clicks on results [2]. This gives a conversion rate of 4.3%.

There were 24,594 unique session tokens [3], which implies that the session-start events are being sent correctly [3]. There were 11,763 result sets generated and shown [4], giving a ratio of start events to result events of 2:1, which is odd considering the same ratio on the mobile apps is closer to 1:4.

We need to investigate whether this logging is working properly.

[1]: SELECT count(*) FROM Search_11670541 WHERE timestamp >= 20150426000000 AND timestamp <= 20150427000000 AND event_action = 'session-start' AND wiki = 'enwiki'
[2]: SELECT count(*) FROM Search_11670541 WHERE timestamp >= 20150426000000 AND timestamp <= 20150427000000 AND event_action = 'click-result' AND wiki = 'enwiki'
[3]: SELECT count(DISTINCT event_searchSessionToken) FROM Search_11670541 WHERE timestamp >= 20150426000000 AND timestamp <= 20150427000000 AND wiki = 'enwiki'
[4]: SELECT count(*) FROM Search_11670541 WHERE timestamp >= 20150426000000 AND timestamp <= 20150427000000 AND event_action = 'impression-results' AND wiki = 'enwiki'

Event Timeline

Deskana created this task.Apr 27 2015, 5:27 PM
Deskana raised the priority of this task from to Needs Triage.
Deskana updated the task description. (Show Details)
Deskana added a project: Wikimedia-Search.
Deskana added a subscriber: Deskana.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2015, 5:27 PM
Deskana triaged this task as High priority.Apr 27 2015, 5:30 PM
Deskana updated the task description. (Show Details)
Deskana set Security to None.
Deskana updated the task description. (Show Details)Apr 27 2015, 5:46 PM
Deskana updated the task description. (Show Details)Apr 27 2015, 5:48 PM
Deskana removed projects: CirrusSearch, Discovery.
Deskana updated the task description. (Show Details)
Deskana added a subscriber: bmansurov.

I have a suspicion that it maybe related to sendBeacon. The analysis should only take the browsers that support this feature into account.

@bmansurov I re-ran all the above queries with userAgent LIKE '%Chrome/42%', and the ratios are all the same. I think there's something else at play here.

session-start: 9981
click-result: 523
distinct search tokens: 9983
impression-results: 4334

@bmansurov Just so I'm sure, does hitting enter in the search box count as a 'click-result' event?

@Deskana, if you're getting the similar results for Chrome, then this needs more investigation. As far as I know, hitting enter does trigger click-result. Browsing the search suggestions code, I noticed that there is also another callback used when the enter is hit. I'll try to find some time tomorrow to look into this.

Deskana edited projects, added Discovery; removed Wikimedia-Search.Apr 27 2015, 8:09 PM

@Deskana, I think the divisor needs to be the event_userSessionToken, not event_searchSessionToken:

> SELECT count(DISTINCT event_userSessionToken) FROM Search_11670541 WHERE timestamp >= 20150426000000 AND timestamp <= 20150427000000 AND wiki = 'enwiki' and userAgent LIKE '%Chrome/42%';
+----------------------------------------+
| count(DISTINCT event_userSessionToken) |
+----------------------------------------+
|                                   1169 |
+----------------------------------------+
> SELECT count(*) FROM Search_11670541 WHERE timestamp >= 20150426000000 AND timestamp <= 20150427000000 AND event_action = 'click-result' AND wiki = 'enwiki' and userAgent LIKE '%Chrome/42%';
+----------+
| count(*) |
+----------+
|      523 |
+----------+

Nonetheless, while approximately 44% clickthrough rate alone seems conceivable, the "do-something" ratio seems like it should be higher.

@bmansurov, my recommendation would be to add an onFormSubmit type action that captures clicks/taps on the magnifying glass icon or entry of the ENTER/RETURN key. It may make sense to try capturing both types of events if possible, and then assign new and different event_action values to each (failing that, just create another type of event_action of something like 'form-submitted')...this will require a schema rev and corresponding code file update. @Deskana, one new event type or two? Or just bundle them into the "click-result" event_action?

@Deskana, I think the divisor needs to be the event_userSessionToken, not event_searchSessionToken:

For now that's fine, but it is indicative that there's either something wrong in the implementation or that the rows in the schema are named incorrectly. I need to get this running in a test instance and audit whether the collection matches my expectations as the data analyst.

For now that's fine, but it is indicative that there's either something wrong in the implementation or that the rows in the schema are named incorrectly. I need to get this running in a test instance and audit whether the collection matches my expectations as the data analyst.

Agreed.

Change 207262 had a related patch set uploaded (by Bmansurov):
Track the skin provided search box form submission

https://gerrit.wikimedia.org/r/207262

Change 207263 had a related patch set uploaded (by Bmansurov):
Track the 'form-submit' event

https://gerrit.wikimedia.org/r/207263

Above are two patches that track the form submission. I didn't differentiate between hitting the Enter key and clicking on the magnifying glass. Let me know if I should.

Deskana moved this task from Needs triage to Needs review on the Discovery board.May 4 2015, 5:50 PM

Change 207262 merged by jenkins-bot:
mediawiki.searchSuggest: Add track event on search form submission

https://gerrit.wikimedia.org/r/207262

@Krinkle merged:

mediawiki.searchSuggest: Add track event on search form submission
https://gerrit.wikimedia.org/r/207262

Note that form submission still leaves ambiguity over two things:

  • The user may or may not have selected a suggestion from the list. (The input field may either contain the user's input, or a a selected page title).
  • The user may or may not end up on the search result page after form submission (if the user entered or selected a title, form submission will redirect to that page title directly instead of showing search results).

Change 207263 merged by jenkins-bot:
Track the 'submit-form' event

https://gerrit.wikimedia.org/r/207263

Deskana closed this task as Resolved.Sep 12 2015, 2:58 AM
Deskana assigned this task to bmansurov.
Deskana moved this task from Done to Resolved on the Discovery-Search (Current work) board.